Skip to contentSkip to conversation

Multi-size THP for anonymous memory

PATCH Applied Ryan Roberts 7 participants 39 messages Dec 7, 2023 – Jan 24, 2024

Subsystems (12)
lore
Share & tools
lore

Ryan Roberts

author reviewer
#0 lore raw

v9 cover Multi-size THP for anonymous memory

Hi All,

This is v9 (and hopefully the last) of a series to implement multi-size THP (mTHP) for anonymous memory (previously called "small-sized THP" and "large anonymous folios").

The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults:

1) Since SW (the kernel) is dealing with larger chunks of memory than base pages, there are efficiency savings to be had; fewer page faults, batched PTE and RMAP manipulation, reduced lru list, etc. In short, we reduce kernel overhead. This should benefit all architectures.
2) Since we are now mapping physically contiguous chunks of memory, we can take advantage of HW TLB compression techniques. A reduction in TLB pressure speeds up kernel and user space. arm64 systems have 2 mechanisms to coalesce TLB entries; "the contiguous bit" (architectural) and HPA (uarch).

This version incorporates David's feedback on the core patches (#3, #4) and adds some RB and TB tags (see change log for details).

By default, the existing behaviour (and performance) is maintained. The user must explicitly enable multi-size THP to see the performance benefit. This is done via a new sysfs interface (as recommended by David Hildenbrand - thanks to David for the suggestion)! This interface is inspired by the existing per-hugepage-size sysfs interface used by hugetlb, provides full backwards compatibility with the existing PMD-size THP interface, and provides a base for future extensibility. See [9] for detailed discussion of the interface.

This series is based on mm-unstable (715b67adf4c8).

Prerequisites
=============

I'm removing this section on the basis that I don't believe what we were previously calling prerequisites are really prerequisites anymore. We originally defined them when mTHP was a compile-time feature. There is now a runtime control to opt-in to mTHP; when disabled, correctness and performance are as before. When enabled, the code is still correct/robust, but in the absence of the one remaining item (compaction) there may be a performance impact in some corners. See the old list in the v8 cover letter at [8]. And a longer explanation of my thinking here [10].

SUMMARY: I don't think we should hold this series up, waiting for the items on the prerequisites list. I believe this series should be ready now so hopefully can be added to mm-unstable for some testing, then fingers crossed for v6.8.

Testing
=======

The series includes patches for mm selftests to enlighten the cow and khugepaged tests to explicitly test with multi-size THP, in the same way that PMD-sized THP is tested. The new tests all pass, and no regressions are observed in the mm selftest suite. I've also run my usual kernel compilation and java script benchmarks without any issues.

Refer to my performance numbers posted with v6 [6]. (These are for multi-size THP only - they do not include the arm64 contpte follow-on series).

John Hubbard at Nvidia has indicated dramatic 10x performance improvements for some workloads at [11]. (Observed using v6 of this series as well as the arm64 contpte series).

Kefeng Wang at Huawei has also indicated he sees improvements at [12] although there are some latency regressions also.

I've also checked that there is no regression in the write fault path when mTHP is disabled using a microbenchmark. I ran it for a baseline kernel, as well as v8 and v9. I repeated on Ampere Altra (bare metal) and Apple M2 (VM):

|              |        m2 vm        |        altra        |
|--------------|---------------------|---------------------|
| kernel       |     mean |  std_rel |     mean |  std_rel |
|--------------|----------|----------|----------|----------|
| baseline     |   0.000% |   0.341% |   0.000% |   3.581% |
| anonfolio-v8 |   0.005% |   0.272% |   5.068% |   1.128% |
| anonfolio-v9 |  -0.013% |   0.442% |   0.107% |   1.788% |

There is no measurable difference on M2, but altra has a slow down in v8 which is fixed in v9 by moving the THP order check to be inline within thp_vma_allowable_orders(), as suggested by David.

Changes since v8 [8]
====================

- Added various Reviewed-by/Tested-by tags (Barry, David, Kefeng, John)
- Patch 3:
- Renamed first_order() -> highest_order() (David)
- Made helpers for thp_vma_suitable_orders() thp_vma_allowable_orders() that take a single unencoded order parameter: thp_vma_suitable_order() and thp_vma_allowable_order(), and use them to aid readability (David)
- Split thp_vma_allowable_orders() into an order-0 fast-path inline and slow-path __thp_vma_allowable_orders() part (David)
- Added spin lock to serialize changes to huge_anon_orders_* fields to prevent possibility of clearing all bits when threads are racing (David)
- Patch 4:
- Pass address of faulting page (not start of folio) to clear_huge_page()
- Reverse xmas tree for variable lists (David)
- Added unlikely() for uffd check (David)
- tidied up a local variable in alloc_anon_folio() (David)
- Separated update_mmu_tlb() handling for nr_pages == 1 vs > 1 (David)

Changes since v7 [7]
====================

- Renamed "small-sized THP" -> "multi-size THP" in commit logs
- Added various Reviewed-by/Tested-by tags (Barry, David, Alistair)
- Patch 3:
- Fine-tuned transhuge documentation multi-size THP (JohnH)
- Converted hugepage_global_enabled() and hugepage_global_always() macros to static inline functions (JohnH)
- Renamed hugepage_vma_check() to thp_vma_allowable_orders() (JohnH)
- Renamed transhuge_vma_suitable() to thp_vma_suitable_orders() (JohnH)
- Renamed "global" enabled sysfs file option to "inherit" (JohnH)
- Patch 9:
- cow selftest: Renamed param size -> thpsize (David)
- cow selftest: Changed test fail to assert() (David)
- cow selftest: Log PMD size separately from all the supported THP sizes (David)
- Patch 10:
- cow selftest: No longer special case pmdsize; keep all THP sizes in thpsizes[]

Changes since v6 [6]
====================

- Refactored vmf_pte_range_changed() to remove uffd special-case (suggested by JohnH)
- Dropped accounting patch (#3 in v6) (suggested by DavidH)
- Continue to account *PMD-sized* THP only for now
- Can add more counters in future if needed
- Page cache large folios haven't needed any new counters yet
- Pivot to sysfs ABI proposed by DavidH
- per-size directories in a similar shape to that used by hugetlb
- Dropped "recommend" keyword patch (#6 in v6) (suggested by DavidH, Yu Zhou)
- For now, users need to understand implicitly which sizes are beneficial to their HW/SW
- Dropped arch_wants_pte_order() patch (#7 in v6)
- No longer needed due to dropping patch "recommend" keyword patch
- Enlightened khugepaged mm selftest to explicitly test with small-size THP
- Scrubbed commit logs to use "small-sized THP" consistently (suggested by DavidH)

Changes since v5 [5]
====================

- Added accounting for PTE-mapped THPs (patch 3)
- Added runtime control mechanism via sysfs as extension to THP (patch 4)
- Minor refactoring of alloc_anon_folio() to integrate with runtime controls
- Stripped out hardcoded policy for allocation order; its now all user space controlled (although user space can request "recommend" which will configure the HW-preferred order)

Changes since v4 [4]
====================

- Removed "arm64: mm: Override arch_wants_pte_order()" patch; arm64 now uses the default order-3 size. I have moved this patch over to the contpte series.
- Added "mm: Allow deferred splitting of arbitrary large anon folios" back into series. I originally removed this at v2 to add to a separate series, but that series has transformed significantly and it no longer fits, so bringing it back here.
- Reintroduced dependency on set_ptes(); Originally dropped this at v2, but set_ptes() is in mm-unstable now.
- Updated policy for when to allocate LAF; only fallback to order-0 if MADV_NOHUGEPAGE is present or if THP disabled via prctl; no longer rely on sysfs's never/madvise/always knob.
- Fallback to order-0 whenever uffd is armed for the vma, not just when uffd-wp is set on the pte.
- alloc_anon_folio() now returns struct folio *, where errors are encoded

    with ERR_PTR().

The last 3 changes were proposed by Yu Zhao - thanks!

Changes since v3 [3]
====================

- Renamed feature from FLEXIBLE_THP to LARGE_ANON_FOLIO.
- Removed flexthp_unhinted_max boot parameter. Discussion concluded that a sysctl is preferable but we will wait until real workload needs it.
- Fixed uninitialized addr on read fault path in do_anonymous_page().
- Added mm selftests for large anon folios in cow test suite.

Changes since v2 [2]
====================

- Dropped commit "Allow deferred splitting of arbitrary large anon folios"
- Huang, Ying suggested the "batch zap" work (which I dropped from this series after v1) is a prerequisite for merging FLXEIBLE_THP, so I've moved the deferred split patch to a separate series along with the batch zap changes. I plan to submit this series early next week.
- Changed folio order fallback policy
- We no longer iterate from preferred to 0 looking for acceptable policy
- Instead we iterate through preferred, PAGE_ALLOC_COSTLY_ORDER and 0 only
- Removed vma parameter from arch_wants_pte_order()
- Added command line parameter flexthp_unhinted_max
- clamps preferred order when vma hasn't explicitly opted-in to THP
- Never allocate large folio for MADV_NOHUGEPAGE vma (or when THP is disabled for process or system).
- Simplified implementation and integration with do_anonymous_page()
- Removed dependency on set_ptes()

Changes since v1 [1]
====================

- removed changes to arch-dependent vma_alloc_zeroed_movable_folio()
- replaced with arch-independent alloc_anon_folio()
- follows THP allocation approach
- no longer retry with intermediate orders if allocation fails
- fallback directly to order-0
- remove folio_add_new_anon_rmap_range() patch
- instead add its new functionality to folio_add_new_anon_rmap()
- remove batch-zap pte mappings optimization patch
- remove enabler folio_remove_rmap_range() patch too
- These offer real perf improvement so will submit separately
- simplify Kconfig
- single FLEXIBLE_THP option, which is independent of arch
- depends on TRANSPARENT_HUGEPAGE
- when enabled default to max anon folio size of 64K unless arch explicitly overrides
- simplify changes to do_anonymous_page():
- no more retry loop

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
[4] https://lore.kernel.org/linux-mm/[email protected]/
[5] https://lore.kernel.org/linux-mm/[email protected]/
[6] https://lore.kernel.org/linux-mm/[email protected]/
[7] https://lore.kernel.org/linux-mm/[email protected]/
[8] https://lore.kernel.org/linux-mm/[email protected]/
[9] https://lore.kernel.org/linux-mm/[email protected]/
[10] https://lore.kernel.org/linux-mm/[email protected]/
[11] https://lore.kernel.org/linux-mm/[email protected]/
[12] https://lore.kernel.org/linux-mm/[email protected]/

Thanks,
Ryan

Ryan Roberts (10):
mm: Allow deferred splitting of arbitrary anon large folios
mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()
mm: thp: Introduce multi-size THP sysfs interface
mm: thp: Support allocation of anonymous multi-size THP selftests/mm/kugepaged: Restore thp settings at exit selftests/mm: Factor out thp settings management selftests/mm: Support multi-size THP interface in thp_settings selftests/mm/khugepaged: Enlighten for multi-size THP selftests/mm/cow: Generalize do_run_with_thp() helper selftests/mm/cow: Add tests for anonymous multi-size THP

 Documentation/admin-guide/mm/transhuge.rst |  97 ++++-
 Documentation/filesystems/proc.rst         |   6 +-
 fs/proc/task_mmu.c                         |   3 +-
 include/linux/huge_mm.h                    | 183 +++++++--
 mm/huge_memory.c                           | 231 ++++++++++--
 mm/khugepaged.c                            |  20 +-
 mm/memory.c                                | 117 +++++-
 mm/page_vma_mapped.c                       |   3 +-
 mm/rmap.c                                  |  32 +-
 tools/testing/selftests/mm/Makefile        |   4 +-
 tools/testing/selftests/mm/cow.c           | 185 +++++++---
 tools/testing/selftests/mm/khugepaged.c    | 410 ++++-----------------
 tools/testing/selftests/mm/run_vmtests.sh  |   2 +
 tools/testing/selftests/mm/thp_settings.c  | 349 ++++++++++++++++++
 tools/testing/selftests/mm/thp_settings.h  |  80 ++++
 15 files changed, 1218 insertions(+), 504 deletions(-)
 create mode 100644 tools/testing/selftests/mm/thp_settings.c
 create mode 100644 tools/testing/selftests/mm/thp_settings.h

--
2.25.1

author reviewer
#1 lore raw

PATCH v9 1/10 mm: Allow deferred splitting of arbitrary anon large folios

In preparation for the introduction of anonymous multi-size THP, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.

---
 mm/rmap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/rmap.c +2 -2
index de9426ad0f1b..2a1e45e6419f 100644
@@ -1518,11 +1518,11 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,
 		__lruvec_stat_mod_folio(folio, idx, -nr);
 
 		/*
-		 * Queue anon THP for deferred split if at least one
+		 * Queue anon large folio for deferred split if at least one
 		 * page of the folio is unmapped and at least one page
 		 * is still mapped.
 		 */
-		if (folio_test_pmd_mappable(folio) && folio_test_anon(folio))
+		if (folio_test_large(folio) && folio_test_anon(folio))
 			if (!compound || nr < nr_pmdmapped)
 				deferred_split_folio(folio);
 	}
Signature · 1 line
2.25.1
author reviewer
#2 lore raw

PATCH v9 2/10 mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

In reply to Ryan Roberts

In preparation for supporting anonymous multi-size THP, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme.

---
 mm/rmap.c | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)
mm/rmap.c +20 -8
index 2a1e45e6419f..846fc79f3ca9 100644
@@ -1335,32 +1335,44 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
  * This means the inc-and-test can be bypassed.
  * The folio does not have to be locked.
  *
- * If the folio is large, it is accounted as a THP.  As the folio
+ * If the folio is pmd-mappable, it is accounted as a THP.  As the folio
  * is new, it's assumed to be mapped exclusively by a single process.
  */
 void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
 		unsigned long address)
 {
-	int nr;
+	int nr = folio_nr_pages(folio);
 
-	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+	VM_BUG_ON_VMA(address < vma->vm_start ||
+			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);
 	__folio_set_swapbacked(folio);
+	__folio_set_anon(folio, vma, address, true);
 
-	if (likely(!folio_test_pmd_mappable(folio))) {
+	if (likely(!folio_test_large(folio))) {
 		/* increment count (starts at -1) */
 		atomic_set(&folio->_mapcount, 0);
-		nr = 1;
+		SetPageAnonExclusive(&folio->page);
+	} else if (!folio_test_pmd_mappable(folio)) {
+		int i;
+
+		for (i = 0; i < nr; i++) {
+			struct page *page = folio_page(folio, i);
+
+			/* increment count (starts at -1) */
+			atomic_set(&page->_mapcount, 0);
+			SetPageAnonExclusive(page);
+		}
+
+		atomic_set(&folio->_nr_pages_mapped, nr);
 	} else {
 		/* increment count (starts at -1) */
 		atomic_set(&folio->_entire_mapcount, 0);
 		atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED);
-		nr = folio_nr_pages(folio);
+		SetPageAnonExclusive(&folio->page);
 		__lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr);
 	}
 
 	__lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr);
-	__folio_set_anon(folio, vma, address, true);
-	SetPageAnonExclusive(&folio->page);
 }
 
 /**
Signature · 1 line
2.25.1
#3 lore raw

Re: [PATCH v9 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

Quoted 27 lines · Ryan Roberts

In preparation for supporting anonymous multi-size THP, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme.

---
 mm/rmap.c | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)
mm/rmap.c +1 -1
index 2a1e45e6419f..846fc79f3ca9 100644
@@ -1335,32 +1335,44 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
  * This means the inc-and-test can be bypassed.
  * The folio does not have to be locked.
  *
- * If the folio is large, it is accounted as a THP.  As the folio
+ * If the folio is pmd-mappable, it is accounted as a THP.  As the folio
  * is new, it's assumed to be mapped exclusively by a single process.

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1347

Quoted 10 lines
mm/rmap.c +3 -2
  */
 void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
 		unsigned long address)
 {
-	int nr;
+	int nr = folio_nr_pages(folio);
 
-	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+	VM_BUG_ON_VMA(address < vma->vm_start ||
+			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

hi,
I'm hitting this bug (console output below) with adding uprobe on simple program like:

  $ cat up.c 
  int main(void)
  { 
     return 0;
  }
  # bpftrace -e 'uprobe:/home/jolsa/up:_start {}'
  $ ./up

it's on top of current linus tree master:
052d534373b7 Merge tag 'exfat-for-6.8-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/exfat

before this patch it seems to work, I can send my .config if needed

thanks,
jirka

---

[  147.562264][  T719] vma ffff888166134e68 start 0000000000401000 end 0000000000402000 mm ffff88817cf2a840
[  147.562264][  T719] prot 25 anon_vma ffff88817b6818e0 vm_ops ffffffff83475ec0
[  147.562264][  T719] pgoff 1 file ffff888168d01240 private_data 0000000000000000
[  147.562264][  T719] flags: 0x75(read|exec|mayread|maywrite|mayexec)
[  147.571660][  T719] ------------[ cut here ]------------
[  147.572319][  T719] kernel BUG at mm/rmap.c:1412!
[  147.572825][  T719] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC KASAN NOPTI
[  147.573792][  T719] CPU: 3 PID: 719 Comm: up Not tainted 6.7.0+ #273 faf755a6fc44b54f4ff1c207411fbd9df5a3968d
[  147.574831][  T719] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc38 04/01/2014
[  147.575652][  T719] RIP: 0010:folio_add_new_anon_rmap+0x2cc/0x8f0
[  147.576164][  T719] Code: c7 c6 20 d2 38 83 48 89 df e8 c0 4a fb ff 0f 0b 48 89 ef e8 16 ab 08 00 4c 3b 65 00 0f 83 cd fd ff ff 48 89 ef e8 34 44 fb ff  f7 c3 ff 0f 00 00 0f 85 de fe ff ff be 08 00 00 00 48 89 df
[  147.577609][  T719] RSP: 0018:ffff88815759f568 EFLAGS: 00010286
[  147.578140][  T719] RAX: 00000000000000fa RBX: ffffea00053eef40 RCX: 0000000000000000
[  147.578825][  T719] RDX: 0000000000000000 RSI: ffffffff81289b44 RDI: ffffffff872ff1a0
[  147.579513][  T719] RBP: ffff888166134e68 R08: 0000000000000001 R09: ffffed102aeb3e5f
[  147.580198][  T719] R10: ffff88815759f2ff R11: 0000000000000000 R12: 0000000000401020
[  147.580886][  T719] R13: 0000000000000001 R14: ffffea00053eef40 R15: ffffea00053eef40
[  147.581566][  T719] FS:  0000000000000000(0000) GS:ffff88842ce00000(0000) knlGS:0000000000000000
[  147.582263][  T719] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  147.582724][  T719] CR2: 00005634f0ffe880 CR3: 000000010c0f8002 CR4: 0000000000770ef0
[  147.583304][  T719] PKRU: 55555554
[  147.583586][  T719] Call Trace:
[  147.583869][  T719]  <TASK>
[  147.584122][  T719]  ? die+0x32/0x80
[  147.584422][  T719]  ? do_trap+0x12f/0x220
[  147.584800][  T719]  ? folio_add_new_anon_rmap+0x2cc/0x8f0
[  147.585411][  T719]  ? folio_add_new_anon_rmap+0x2cc/0x8f0
[  147.585891][  T719]  ? do_error_trap+0xa7/0x160
[  147.586349][  T719]  ? folio_add_new_anon_rmap+0x2cc/0x8f0
[  147.586879][  T719]  ? handle_invalid_op+0x2c/0x40
[  147.587354][  T719]  ? folio_add_new_anon_rmap+0x2cc/0x8f0
[  147.587892][  T719]  ? exc_invalid_op+0x29/0x40
[  147.588352][  T719]  ? asm_exc_invalid_op+0x16/0x20
[  147.588847][  T719]  ? preempt_count_sub+0x14/0xc0
[  147.589335][  T719]  ? folio_add_new_anon_rmap+0x2cc/0x8f0
[  147.589899][  T719]  ? folio_add_new_anon_rmap+0x2cc/0x8f0
[  147.590437][  T719]  __replace_page+0x364/0xb40
[  147.590918][  T719]  ? __pfx___replace_page+0x10/0x10
[  147.591412][  T719]  ? __pfx_lock_release+0x10/0x10
[  147.591910][  T719]  ? do_raw_spin_trylock+0xcd/0x120
[  147.592555][  T719]  ? __pfx_vma_alloc_folio+0x10/0x10
[  147.593095][  T719]  ? preempt_count_add+0x6e/0xc0
[  147.593612][  T719]  ? preempt_count_sub+0x14/0xc0
[  147.594143][  T719]  uprobe_write_opcode+0x3f6/0x820
[  147.594616][  T719]  ? __pfx_uprobe_write_opcode+0x10/0x10
[  147.595125][  T719]  ? preempt_count_sub+0x14/0xc0
[  147.595551][  T719]  ? up_write+0x125/0x2f0
[  147.596014][  T719]  install_breakpoint.isra.0+0xe5/0x470
[  147.596635][  T719]  uprobe_mmap+0x37b/0x8d0
[  147.598111][  T719]  ? __pfx_uprobe_mmap+0x10/0x10
[  147.598561][  T719]  mmap_region+0xa02/0x1220
[  147.599013][  T719]  ? rcu_is_watching+0x34/0x60
[  147.599602][  T719]  ? lock_acquired+0xbf/0x670
[  147.600024][  T719]  ? __pfx_mmap_region+0x10/0x10
[  147.600458][  T719]  ? security_mmap_addr+0x20/0x60
[  147.600909][  T719]  ? get_unmapped_area+0x169/0x1f0
[  147.601353][  T719]  do_mmap+0x425/0x660
[  147.601739][  T719]  vm_mmap_pgoff+0x15e/0x2b0
[  147.602156][  T719]  ? __pfx_vm_mmap_pgoff+0x10/0x10
[  147.602597][  T719]  ? __pfx_get_random_u64+0x10/0x10
[  147.603059][  T719]  elf_load+0xdc/0x3a0
[  147.603433][  T719]  load_elf_binary+0x6f6/0x22b0
[  147.603889][  T719]  ? __pfx_load_elf_binary+0x10/0x10
[  147.604385][  T719]  ? __pfx_lock_acquired+0x10/0x10
[  147.604952][  T719]  bprm_execve+0x494/0xc80
[  147.605379][  T719]  ? __pfx_bprm_execve+0x10/0x10
[  147.605843][  T719]  do_execveat_common.isra.0+0x24f/0x330
[  147.606358][  T719]  __x64_sys_execve+0x52/0x60
[  147.606797][  T719]  do_syscall_64+0x87/0x1b0
[  147.607148][  T719]  entry_SYSCALL_64_after_hwframe+0x6e/0x76
[  147.607630][  T719] RIP: 0033:0x7faa9b0bdb4b
[  147.608732][  T719] Code: Unable to access opcode bytes at 0x7faa9b0bdb21.
[  147.609318][  T719] RSP: 002b:00007ffff9921708 EFLAGS: 00000246 ORIG_RAX: 000000000000003b
[  147.610046][  T719] RAX: ffffffffffffffda RBX: 00005634f1964990 RCX: 00007faa9b0bdb4b
[  147.610727][  T719] RDX: 00005634f1966d20 RSI: 00005634f19612c0 RDI: 00005634f1964990
[  147.611528][  T719] RBP: 00007ffff9921800 R08: 0000000000000001 R09: 0000000000000001
[  147.612192][  T719] R10: 0000000000000008 R11: 0000000000000246 R12: 00000000ffffffff
[  147.612829][  T719] R13: 00005634f1964990 R14: 00005634f19612c0 R15: 00005634f1966d20
[  147.613479][  T719]  </TASK>
[  147.613787][  T719] Modules linked in: intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel kvm_intel rapl iTiTCO_vendor_support i2c_i801 i2c_smbus lpc_ich drm loop drm_panel_orientation_quirks zram
[  147.615630][  T719] ---[ end trace 0000000000000000 ]---
[  147.616253][  T719] RIP: 0010:folio_add_new_anon_rmap+0x2cc/0x8f0
[  147.616714][  T719] Code: c7 c6 20 d2 38 83 48 89 df e8 c0 4a fb ff 0f 0b 48 89 ef e8 16 ab 08 00 4c 3b 65 00 0f 83 cd fd ff ff 48 89 ef e8 34 44 fb ff  f7 c3 ff 0f 00 00 0f 85 de fe ff ff be 08 00 00 00 48 89 df
[  147.618160][  T719] RSP: 0018:ffff88815759f568 EFLAGS: 00010286
[  147.618594][  T719] RAX: 00000000000000fa RBX: ffffea00053eef40 RCX: 0000000000000000
[  147.619318][  T719] RDX: 0000000000000000 RSI: ffffffff81289b44 RDI: ffffffff872ff1a0
[  147.619930][  T719] RBP: ffff888166134e68 R08: 0000000000000001 R09: ffffed102aeb3e5f
[  147.620577][  T719] R10: ffff88815759f2ff R11: 0000000000000000 R12: 0000000000401020
[  147.621236][  T719] R13: 0000000000000001 R14: ffffea00053eef40 R15: ffffea00053eef40
[  147.621894][  T719] FS:  0000000000000000(0000) GS:ffff88842ce00000(0000) knlGS:0000000000000000
[  147.622596][  T719] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  147.623186][  T719] CR2: 00007faa9b0bdb21 CR3: 000000010c0f8002 CR4: 0000000000770ef0
[  147.623960][  T719] PKRU: 55555554
[  147.624331][  T719] note: up[719] exited with preempt_count 1
[  147.624953][  T719] BUG: sleeping function called from invalid context at include/linux/percpu-rwsem.h:49
[  147.625898][  T719] in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid: 719, name: up
[  147.626672][  T719] preempt_count: 0, expected: 0
[  147.627945][  T719] RCU nest depth: 1, expected: 0
[  147.628410][  T719] INFO: lockdep is turned off.
[  147.628898][  T719] CPU: 3 PID: 719 Comm: up Tainted: G      D            6.7.0+ #273 faf755a6fc44b54f4ff1c207411fbd9df5a3968d
[  147.629954][  T719] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc38 04/01/2014
[  147.630838][  T719] Call Trace:
[  147.631185][  T719]  <TASK>
[  147.631514][  T719]  dump_stack_lvl+0x15d/0x180
[  147.631973][  T719]  __might_resched+0x270/0x3b0
[  147.636533][  T719]  exit_signals+0x1d/0x460
[  147.636947][  T719]  do_exit+0x27f/0x13b0
[  147.637368][  T719]  ? __pfx__printk+0x10/0x10
[  147.637827][  T719]  ? __pfx_do_exit+0x10/0x10
[  147.638238][  T719]  make_task_dead+0xd9/0x240
[  147.638610][  T719]  rewind_stack_and_make_dead+0x17/0x20
[  147.639064][  T719] RIP: 0033:0x7faa9b0bdb4b
[  147.639445][  T719] Code: Unable to access opcode bytes at 0x7faa9b0bdb21.
[  147.640015][  T719] RSP: 002b:00007ffff9921708 EFLAGS: 00000246 ORIG_RAX: 000000000000003b
[  147.640694][  T719] RAX: ffffffffffffffda RBX: 00005634f1964990 RCX: 00007faa9b0bdb4b
[  147.641407][  T719] RDX: 00005634f1966d20 RSI: 00005634f19612c0 RDI: 00005634f1964990
[  147.642133][  T719] RBP: 00007ffff9921800 R08: 0000000000000001 R09: 0000000000000001
[  147.642911][  T719] R10: 0000000000000008 R11: 0000000000000246 R12: 00000000ffffffff
[  147.643685][  T719] R13: 00005634f1964990 R14: 00005634f19612c0 R15: 00005634f1966d20
[  147.644454][  T719]  </TASK>
[  147.644819][  T719] ------------[ cut here ]------------
#4 lore raw

Re: [PATCH v9 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

Quoted 49 lines · Jiri Olsa
Quoted 27 lines · Ryan Roberts

In preparation for supporting anonymous multi-size THP, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme.

---
  mm/rmap.c | 28 ++++++++++++++++++++--------
  1 file changed, 20 insertions(+), 8 deletions(-)
mm/rmap.c +1 -1
index 2a1e45e6419f..846fc79f3ca9 100644
@@ -1335,32 +1335,44 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
   * This means the inc-and-test can be bypassed.
   * The folio does not have to be locked.
   *
- * If the folio is large, it is accounted as a THP.  As the folio
+ * If the folio is pmd-mappable, it is accounted as a THP.  As the folio
   * is new, it's assumed to be mapped exclusively by a single process.

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1347

Quoted 10 lines
mm/rmap.c +3 -2
   */
  void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
  		unsigned long address)
  {
-	int nr;
+	int nr = folio_nr_pages(folio);
 
-	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+	VM_BUG_ON_VMA(address < vma->vm_start ||
+			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

hi,
I'm hitting this bug (console output below) with adding uprobe on simple program like:

   $ cat up.c
   int main(void)
   {
      return 0;
   }
Quoted 8 lines
   # bpftrace -e 'uprobe:/home/jolsa/up:_start {}'
   $ ./up

it's on top of current linus tree master:

   052d534373b7 Merge tag 'exfat-for-6.8-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/exfat

before this patch it seems to work, I can send my .config if needed

bpf only inserts a small folio, so no magic there.

It was:

	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
And now it is
	VM_BUG_ON_VMA(address < vma->vm_start || address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

I think this change is sane. As long as the address is aligned to full pages (which it better should be)

Staring at uprobe_write_opcode, likely vaddr isn't aligned ...

Likely (hopefully) that is not an issue for __folio_set_anon(), because linear_page_index() will mask these bits off.

Would the following change fix it for you?

From c640a8363e47bc96965a35115a040b5f876c4320 Mon Sep 17 00:00:00 2001
From: David Hildenbrand <[email protected]>
Date: Sun, 14 Jan 2024 18:32:57 +0100
Subject: [PATCH] tmp

---
 kernel/events/uprobes.c | 2 +-
 mm/rmap.c               | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
kernel/events/uprobes.c +1 -1
index 485bb0389b488..929e98c629652 100644
@@ -537,7 +537,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm,
 		}
 	}
 
-	ret = __replace_page(vma, vaddr, old_page, new_page);
+	ret = __replace_page(vma, vaddr & PAGE_MASK, old_page, new_page);
 	if (new_page)
 		put_page(new_page);
 put_old:
mm/rmap.c +1 -0
index f5d43edad529a..a903db4df6b97 100644
@@ -1408,6 +1408,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
 {
 	int nr = folio_nr_pages(folio);
 
+	VM_WARN_ON_FOLIO(!IS_ALIGNED(address, PAGE_SIZE), folio);
 	VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio);
 	VM_BUG_ON_VMA(address < vma->vm_start ||
 			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);
Signature · 8 lines
2.43.0



-- 
Cheers,

David / dhildenb
Tested-by
#5 lore raw

Re: [PATCH v9 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

Quoted 68 lines · David Hildenbrand
Quoted 48 lines · Jiri Olsa
Quoted 26 lines · Ryan Roberts

In preparation for supporting anonymous multi-size THP, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme.

---
  mm/rmap.c | 28 ++++++++++++++++++++--------
  1 file changed, 20 insertions(+), 8 deletions(-)
mm/rmap.c +1 -1
index 2a1e45e6419f..846fc79f3ca9 100644
@@ -1335,32 +1335,44 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
   * This means the inc-and-test can be bypassed.
   * The folio does not have to be locked.
   *
- * If the folio is large, it is accounted as a THP.  As the folio
+ * If the folio is pmd-mappable, it is accounted as a THP.  As the folio

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1346

Quoted 10 lines
mm/rmap.c +3 -2
   * is new, it's assumed to be mapped exclusively by a single process.
   */
  void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
  		unsigned long address)
  {
-	int nr;
+	int nr = folio_nr_pages(folio);
-	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+	VM_BUG_ON_VMA(address < vma->vm_start ||
+			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

hi,
I'm hitting this bug (console output below) with adding uprobe on simple program like:

   $ cat up.c
   int main(void)
   {
      return 0;
   }
Quoted 8 lines
   # bpftrace -e 'uprobe:/home/jolsa/up:_start {}'
   $ ./up

it's on top of current linus tree master:

   052d534373b7 Merge tag 'exfat-for-6.8-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/exfat

before this patch it seems to work, I can send my .config if needed

bpf only inserts a small folio, so no magic there.

It was:

	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
And now it is
	VM_BUG_ON_VMA(address < vma->vm_start || address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

I think this change is sane. As long as the address is aligned to full pages (which it better should be)

Quoted 7 lines

Staring at uprobe_write_opcode, likely vaddr isn't aligned ...

Likely (hopefully) that is not an issue for __folio_set_anon(), because linear_page_index() will mask these bits off.

Would the following change fix it for you?

great, that fixes it for me, you can add my

thanks,
jirka

Quoted 45 lines

From c640a8363e47bc96965a35115a040b5f876c4320 Mon Sep 17 00:00:00 2001
From: David Hildenbrand <[email protected]>
Date: Sun, 14 Jan 2024 18:32:57 +0100
Subject: [PATCH] tmp

---
 kernel/events/uprobes.c | 2 +-
 mm/rmap.c               | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
kernel/events/uprobes.c +1 -1
index 485bb0389b488..929e98c629652 100644
@@ -537,7 +537,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm,
 		}
 	}
-	ret = __replace_page(vma, vaddr, old_page, new_page);
+	ret = __replace_page(vma, vaddr & PAGE_MASK, old_page, new_page);
 	if (new_page)
 		put_page(new_page);
 put_old:
mm/rmap.c +1 -0
index f5d43edad529a..a903db4df6b97 100644
@@ -1408,6 +1408,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
 {
 	int nr = folio_nr_pages(folio);
+	VM_WARN_ON_FOLIO(!IS_ALIGNED(address, PAGE_SIZE), folio);
 	VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio);
 	VM_BUG_ON_VMA(address < vma->vm_start ||
 			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);
Signature · 8 lines
2.43.0



-- 
Cheers,

David / dhildenb
author reviewer Reviewed-by
#6 lore raw

Re: [PATCH v9 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

Quoted 50 lines · Jiri Olsa
Quoted 49 lines · David Hildenbrand
Quoted 48 lines · Jiri Olsa
Quoted 26 lines · Ryan Roberts

In preparation for supporting anonymous multi-size THP, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme.

---
  mm/rmap.c | 28 ++++++++++++++++++++--------
  1 file changed, 20 insertions(+), 8 deletions(-)
mm/rmap.c +1 -1
index 2a1e45e6419f..846fc79f3ca9 100644
@@ -1335,32 +1335,44 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
   * This means the inc-and-test can be bypassed.
   * The folio does not have to be locked.
   *
- * If the folio is large, it is accounted as a THP.  As the folio
+ * If the folio is pmd-mappable, it is accounted as a THP.  As the folio

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1346

Quoted 10 lines
mm/rmap.c +3 -2
   * is new, it's assumed to be mapped exclusively by a single process.
   */
  void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
  		unsigned long address)
  {
-	int nr;
+	int nr = folio_nr_pages(folio);
-	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+	VM_BUG_ON_VMA(address < vma->vm_start ||
+			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

hi,
I'm hitting this bug (console output below) with adding uprobe on simple program like:

   $ cat up.c
   int main(void)
   {
      return 0;
   }
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
   # bpftrace -e 'uprobe:/home/jolsa/up:_start {}'
   $ ./up

it's on top of current linus tree master:

   052d534373b7 Merge tag 'exfat-for-6.8-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/exfat

before this patch it seems to work, I can send my .config if needed

Thanks for the bug report!

Quoted 11 lines
Quoted 11 lines

bpf only inserts a small folio, so no magic there.

It was:

	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
And now it is
	VM_BUG_ON_VMA(address < vma->vm_start || address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

I think this change is sane. As long as the address is aligned to full pages (which it better should be)

Quoted 7 lines
Quoted 7 lines

Staring at uprobe_write_opcode, likely vaddr isn't aligned ...

Likely (hopefully) that is not an issue for __folio_set_anon(), because linear_page_index() will mask these bits off.

Would the following change fix it for you?

And thanks for fixing my mess so quickly, David.

FWIW, I agree with your diagnosis. One small suggestion below.

Quoted 32 lines

great, that fixes it for me, you can add my

thanks,
jirka

Quoted 24 lines

From c640a8363e47bc96965a35115a040b5f876c4320 Mon Sep 17 00:00:00 2001
From: David Hildenbrand <[email protected]>
Date: Sun, 14 Jan 2024 18:32:57 +0100
Subject: [PATCH] tmp

---
 kernel/events/uprobes.c | 2 +-
 mm/rmap.c               | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
kernel/events/uprobes.c +1 -1
index 485bb0389b488..929e98c629652 100644
@@ -537,7 +537,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm,
 		}
 	}
-	ret = __replace_page(vma, vaddr, old_page, new_page);
+	ret = __replace_page(vma, vaddr & PAGE_MASK, old_page, new_page);
 	if (new_page)
 		put_page(new_page);
 put_old:
Quoted 8 lines

mm/rmap.c void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, · line 1410

Quoted 8 lines
mm/rmap.c +1 -0
index f5d43edad529a..a903db4df6b97 100644
@@ -1408,6 +1408,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
 {
 	int nr = folio_nr_pages(folio);
+	VM_WARN_ON_FOLIO(!IS_ALIGNED(address, PAGE_SIZE), folio);

nit: Is it worth also adding this to __folio_add_anon_rmap() so that folio_add_anon_rmap_ptes() and folio_add_anon_rmap_pmd() also benefit?

Regardless:

Quoted 13 lines
Quoted 13 lines
mm/rmap.c +0 -0
 	VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio);
 	VM_BUG_ON_VMA(address < vma->vm_start ||
 			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);
Signature · 8 lines
2.43.0



-- 
Cheers,

David / dhildenb
#7 lore raw

Re: [PATCH v9 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
kernel/events/uprobes.c +1 -1
index 485bb0389b488..929e98c629652 100644
@@ -537,7 +537,7 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm,
  		}
  	}
-	ret = __replace_page(vma, vaddr, old_page, new_page);
+	ret = __replace_page(vma, vaddr & PAGE_MASK, old_page, new_page);
  	if (new_page)
  		put_page(new_page);
  put_old:
mm/rmap.c +0 -1
index f5d43edad529a..a903db4df6b97 100644
--- a/mm/rmap.c
Quoted 9 lines
Quoted 5 lines

mm/rmap.c void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, · line 1410

Quoted 5 lines

+++ b/mm/rmap.c

mm/rmap.c +1 -0
@@ -1408,6 +1408,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
  {
  	int nr = folio_nr_pages(folio);
+	VM_WARN_ON_FOLIO(!IS_ALIGNED(address, PAGE_SIZE), folio);

nit: Is it worth also adding this to __folio_add_anon_rmap() so that folio_add_anon_rmap_ptes() and folio_add_anon_rmap_pmd() also benefit?

Yes, same thoughts. Just included it so we would catch if still something goes wrong here.

I'll split that change out either way.

Quoted 3 lines

Regardless:

Thanks!

Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#8 lore raw

Re: [PATCH v9 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

In reply to Ryan Roberts
Quoted 53 lines · Ryan Roberts
Quoted 52 lines · Jiri Olsa
Quoted 51 lines · David Hildenbrand
Quoted 50 lines · Jiri Olsa
Quoted 26 lines · Ryan Roberts

In preparation for supporting anonymous multi-size THP, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme.

---
  mm/rmap.c | 28 ++++++++++++++++++++--------
  1 file changed, 20 insertions(+), 8 deletions(-)
mm/rmap.c +1 -1
index 2a1e45e6419f..846fc79f3ca9 100644
@@ -1335,32 +1335,44 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
   * This means the inc-and-test can be bypassed.
   * The folio does not have to be locked.
   *
- * If the folio is large, it is accounted as a THP.  As the folio
+ * If the folio is pmd-mappable, it is accounted as a THP.  As the folio

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1346

Quoted 10 lines
mm/rmap.c +3 -2
   * is new, it's assumed to be mapped exclusively by a single process.
   */
  void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
  		unsigned long address)
  {
-	int nr;
+	int nr = folio_nr_pages(folio);
-	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+	VM_BUG_ON_VMA(address < vma->vm_start ||
+			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

hi,
I'm hitting this bug (console output below) with adding uprobe on simple program like:

   $ cat up.c
   int main(void)
   {
      return 0;
   }
   # bpftrace -e 'uprobe:/home/jolsa/up:_start {}'
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
   $ ./up

it's on top of current linus tree master:

   052d534373b7 Merge tag 'exfat-for-6.8-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/exfat

before this patch it seems to work, I can send my .config if needed

Thanks for the bug report!

I just hit the same bug in our CI, but can't find the fix in -next. Is this in the queue somewhere?

Thanks
Sven

#9 lore raw

Re: [PATCH v9 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

Quoted 60 lines · Sven Schnelle
Quoted 58 lines · Ryan Roberts
Quoted 57 lines · Jiri Olsa
Quoted 56 lines · David Hildenbrand
Quoted 55 lines · Jiri Olsa
Quoted 26 lines · Ryan Roberts

In preparation for supporting anonymous multi-size THP, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme.

---
  mm/rmap.c | 28 ++++++++++++++++++++--------
  1 file changed, 20 insertions(+), 8 deletions(-)
mm/rmap.c +1 -1
index 2a1e45e6419f..846fc79f3ca9 100644
@@ -1335,32 +1335,44 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
   * This means the inc-and-test can be bypassed.
   * The folio does not have to be locked.
   *
- * If the folio is large, it is accounted as a THP.  As the folio
+ * If the folio is pmd-mappable, it is accounted as a THP.  As the folio

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1346

Quoted 10 lines
mm/rmap.c +3 -2
   * is new, it's assumed to be mapped exclusively by a single process.
   */
  void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
  		unsigned long address)
  {
-	int nr;
+	int nr = folio_nr_pages(folio);
-	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+	VM_BUG_ON_VMA(address < vma->vm_start ||
+			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

hi,
I'm hitting this bug (console output below) with adding uprobe on simple program like:

   $ cat up.c
   int main(void)
   {
      return 0;
   }
   # bpftrace -e 'uprobe:/home/jolsa/up:_start {}'
   $ ./up

it's on top of current linus tree master:

   052d534373b7 Merge tag 'exfat-for-6.8-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/exfat
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

before this patch it seems to work, I can send my .config if needed

Thanks for the bug report!

I just hit the same bug in our CI, but can't find the fix in -next. Is this in the queue somewhere?

we hit it as well, but I can see the fix in linux-next/master

4c137bc28064 uprobes: use pagesize-aligned virtual address when replacing pages

jirka

author reviewer
#10 lore raw

Re: [PATCH v9 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

Quoted 63 lines · Jiri Olsa
Quoted 62 lines · Sven Schnelle
Quoted 60 lines · Ryan Roberts
Quoted 58 lines · Jiri Olsa
Quoted 57 lines · David Hildenbrand
Quoted 56 lines · Jiri Olsa
Quoted 26 lines · Ryan Roberts

In preparation for supporting anonymous multi-size THP, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme.

---
  mm/rmap.c | 28 ++++++++++++++++++++--------
  1 file changed, 20 insertions(+), 8 deletions(-)
mm/rmap.c +1 -1
index 2a1e45e6419f..846fc79f3ca9 100644
@@ -1335,32 +1335,44 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
   * This means the inc-and-test can be bypassed.
   * The folio does not have to be locked.
   *
- * If the folio is large, it is accounted as a THP.  As the folio
+ * If the folio is pmd-mappable, it is accounted as a THP.  As the folio

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1346

Quoted 10 lines
mm/rmap.c +3 -2
   * is new, it's assumed to be mapped exclusively by a single process.
   */
  void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
  		unsigned long address)
  {
-	int nr;
+	int nr = folio_nr_pages(folio);
-	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+	VM_BUG_ON_VMA(address < vma->vm_start ||
+			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

hi,
I'm hitting this bug (console output below) with adding uprobe on simple program like:

   $ cat up.c
   int main(void)
   {
      return 0;
   }
   # bpftrace -e 'uprobe:/home/jolsa/up:_start {}'
   $ ./up

it's on top of current linus tree master:

   052d534373b7 Merge tag 'exfat-for-6.8-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/exfat

before this patch it seems to work, I can send my .config if needed

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Thanks for the bug report!

I just hit the same bug in our CI, but can't find the fix in -next. Is this in the queue somewhere?

we hit it as well, but I can see the fix in linux-next/master

4c137bc28064 uprobes: use pagesize-aligned virtual address when replacing pages

Yes that's the one. Just to confirm: you are still hitting the VM_BUG_ON despite having this change in your kernel? Could you please send over the full bug log?

Quoted 2 lines

jirka

#11 lore raw

Re: [PATCH v9 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

Quoted 69 lines · Ryan Roberts
Quoted 68 lines · Jiri Olsa
Quoted 66 lines · Sven Schnelle
Quoted 53 lines · Ryan Roberts
Quoted 52 lines · Jiri Olsa
Quoted 51 lines · David Hildenbrand
Quoted 50 lines · Jiri Olsa
Quoted 26 lines · Ryan Roberts

In preparation for supporting anonymous multi-size THP, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme.

---
  mm/rmap.c | 28 ++++++++++++++++++++--------
  1 file changed, 20 insertions(+), 8 deletions(-)
mm/rmap.c +1 -1
index 2a1e45e6419f..846fc79f3ca9 100644
@@ -1335,32 +1335,44 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
   * This means the inc-and-test can be bypassed.
   * The folio does not have to be locked.
   *
- * If the folio is large, it is accounted as a THP.  As the folio
+ * If the folio is pmd-mappable, it is accounted as a THP.  As the folio

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1346

Quoted 10 lines
mm/rmap.c +3 -2
   * is new, it's assumed to be mapped exclusively by a single process.
   */
  void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
  		unsigned long address)
  {
-	int nr;
+	int nr = folio_nr_pages(folio);
-	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+	VM_BUG_ON_VMA(address < vma->vm_start ||
+			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

hi,
I'm hitting this bug (console output below) with adding uprobe on simple program like:

   $ cat up.c
   int main(void)
   {
      return 0;
   }
   # bpftrace -e 'uprobe:/home/jolsa/up:_start {}'
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
   $ ./up

it's on top of current linus tree master:

   052d534373b7 Merge tag 'exfat-for-6.8-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/exfat

before this patch it seems to work, I can send my .config if needed

Thanks for the bug report!

I just hit the same bug in our CI, but can't find the fix in -next. Is this in the queue somewhere?

Quoted 6 lines
Quoted 3 lines

we hit it as well, but I can see the fix in linux-next/master

4c137bc28064 uprobes: use pagesize-aligned virtual address when replacing pages

Yes that's the one. Just to confirm: you are still hitting the VM_BUG_ON despite having this change in your kernel? Could you please send over the full bug log?

ah sorry.. I meant the change fixes the problem for us, it just did not yet propagate through the merge cycle into bpf trees.. but I can see it in linux-next tree, so it's probably just matter of time

jirka

author reviewer
#12 lore raw

Re: [PATCH v9 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

Quoted 72 lines · Jiri Olsa
Quoted 71 lines · Ryan Roberts
Quoted 70 lines · Jiri Olsa
Quoted 60 lines · Sven Schnelle
Quoted 58 lines · Ryan Roberts
Quoted 57 lines · Jiri Olsa
Quoted 56 lines · David Hildenbrand
Quoted 55 lines · Jiri Olsa
Quoted 26 lines · Ryan Roberts

In preparation for supporting anonymous multi-size THP, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme.

---
  mm/rmap.c | 28 ++++++++++++++++++++--------
  1 file changed, 20 insertions(+), 8 deletions(-)
mm/rmap.c +1 -1
index 2a1e45e6419f..846fc79f3ca9 100644
@@ -1335,32 +1335,44 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
   * This means the inc-and-test can be bypassed.
   * The folio does not have to be locked.
   *
- * If the folio is large, it is accounted as a THP.  As the folio
+ * If the folio is pmd-mappable, it is accounted as a THP.  As the folio

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1346

Quoted 10 lines
mm/rmap.c +3 -2
   * is new, it's assumed to be mapped exclusively by a single process.
   */
  void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
  		unsigned long address)
  {
-	int nr;
+	int nr = folio_nr_pages(folio);
-	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+	VM_BUG_ON_VMA(address < vma->vm_start ||
+			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

hi,
I'm hitting this bug (console output below) with adding uprobe on simple program like:

   $ cat up.c
   int main(void)
   {
      return 0;
   }
   # bpftrace -e 'uprobe:/home/jolsa/up:_start {}'
   $ ./up

it's on top of current linus tree master:

   052d534373b7 Merge tag 'exfat-for-6.8-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/exfat
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

before this patch it seems to work, I can send my .config if needed

Thanks for the bug report!

I just hit the same bug in our CI, but can't find the fix in -next. Is this in the queue somewhere?

we hit it as well, but I can see the fix in linux-next/master

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

4c137bc28064 uprobes: use pagesize-aligned virtual address when replacing pages

Yes that's the one. Just to confirm: you are still hitting the VM_BUG_ON despite having this change in your kernel? Could you please send over the full bug log?

ah sorry.. I meant the change fixes the problem for us, it just did not yet propagate through the merge cycle into bpf trees.. but I can see it in linux-next tree, so it's probably just matter of time

OK great! How about you, Sven? Do you have this change in your kernel? Hopefully it should fix your problem.

Quoted 2 lines

jirka

reviewer
#13 lore raw

Re: [PATCH v9 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

Hi Ryan,

Quoted 30 lines · Ryan Roberts
Quoted 30 lines
Quoted 29 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines

I'm hitting this bug (console output below) with adding uprobe on simple program like:

   $ cat up.c
   int main(void)
   {
      return 0;
   }
   # bpftrace -e 'uprobe:/home/jolsa/up:_start {}'
   $ ./up

it's on top of current linus tree master:

   052d534373b7 Merge tag 'exfat-for-6.8-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/exfat

before this patch it seems to work, I can send my .config if needed

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Thanks for the bug report!

I just hit the same bug in our CI, but can't find the fix in -next. Is this in the queue somewhere?

we hit it as well, but I can see the fix in linux-next/master

4c137bc28064 uprobes: use pagesize-aligned virtual address when replacing pages

Yes that's the one. Just to confirm: you are still hitting the VM_BUG_ON despite having this change in your kernel? Could you please send over the full bug log?

Quoted 6 lines
Quoted 3 lines

ah sorry.. I meant the change fixes the problem for us, it just did not yet propagate through the merge cycle into bpf trees.. but I can see it in linux-next tree, so it's probably just matter of time

OK great! How about you, Sven? Do you have this change in your kernel? Hopefully it should fix your problem.

Same here - the fix makes uprobes work again, i just didn't see it in torvalds-master and neither in todays linux-next. But Jiri is right, it's in linux-next/master. I just missed to find it there. So everything should be ok.

author reviewer
#14 lore raw

Re: [PATCH v9 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

Quoted 38 lines · Sven Schnelle

Hi Ryan,

Quoted 34 lines · Ryan Roberts
Quoted 33 lines
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

I'm hitting this bug (console output below) with adding uprobe on simple program like:

   $ cat up.c
   int main(void)
   {
      return 0;
   }
   # bpftrace -e 'uprobe:/home/jolsa/up:_start {}'
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
   $ ./up

it's on top of current linus tree master:

   052d534373b7 Merge tag 'exfat-for-6.8-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/exfat

before this patch it seems to work, I can send my .config if needed

Thanks for the bug report!

I just hit the same bug in our CI, but can't find the fix in -next. Is this in the queue somewhere?

Quoted 6 lines
Quoted 3 lines

we hit it as well, but I can see the fix in linux-next/master

4c137bc28064 uprobes: use pagesize-aligned virtual address when replacing pages

Yes that's the one. Just to confirm: you are still hitting the VM_BUG_ON despite having this change in your kernel? Could you please send over the full bug log?

ah sorry.. I meant the change fixes the problem for us, it just did not yet propagate through the merge cycle into bpf trees.. but I can see it in linux-next tree, so it's probably just matter of time

Quoted 7 lines
Quoted 2 lines

OK great! How about you, Sven? Do you have this change in your kernel? Hopefully it should fix your problem.

Same here - the fix makes uprobes work again, i just didn't see it in torvalds-master and neither in todays linux-next. But Jiri is right, it's in linux-next/master. I just missed to find it there. So everything should be ok.

Great!

author reviewer
#15 lore raw

PATCH v9 3/10 mm: thp: Introduce multi-size THP sysfs interface

In reply to Ryan Roberts

In preparation for adding support for anonymous multi-size THP, introduce new sysfs structure that will be used to control the new behaviours. A new directory is added under transparent_hugepage for each supported THP size, and contains an enabled file, which can be set to "inherit" (to inherit the global setting), "always", "madvise" or "never". For now, the kernel still only supports PMD-sized anonymous THP, so only 1 directory is populated.

The first half of the change converts transhuge_vma_suitable() and hugepage_vma_check() so that they take a bitfield of orders for which the user wants to determine support, and the functions filter out all the orders that can't be supported, given the current sysfs configuration and the VMA dimensions. The resulting functions are renamed to thp_vma_suitable_orders() and thp_vma_allowable_orders() respectively. Convenience functions that take a single, unencoded order and return a boolean are also defined as thp_vma_suitable_order() and thp_vma_allowable_order().

The second half of the change implements the new sysfs interface. It has been done so that each supported THP size has a struct thpsize, which describes the relevant metadata and is itself a kobject. This is pretty minimal for now, but should make it easy to add new per-thpsize files to the interface if needed in future (e.g. per-size defrag). Rather than keep the enabled state directly in the struct thpsize, I've elected to directly encode it into huge_anon_orders_[always|madvise|inherit] bitfields since this reduces the amount of work required in thp_vma_allowable_orders() which is called for every page fault.

See Documentation/admin-guide/mm/transhuge.rst, as modified by this commit, for details of how the new sysfs interface works.

---
 Documentation/admin-guide/mm/transhuge.rst |  97 +++++++--
 Documentation/filesystems/proc.rst         |   6 +-
 fs/proc/task_mmu.c                         |   3 +-
 include/linux/huge_mm.h                    | 181 +++++++++++++---
 mm/huge_memory.c                           | 231 ++++++++++++++++++---
 mm/khugepaged.c                            |  20 +-
 mm/memory.c                                |   6 +-
 mm/page_vma_mapped.c                       |   3 +-
 8 files changed, 459 insertions(+), 88 deletions(-)
Documentation/admin-guide/mm/transhuge.rst +78 -19
index b0cc8243e093..04eb45a2f940 100644
@@ -45,10 +45,25 @@ components:
    the two is using hugepages just because of the fact the TLB miss is
    going to run faster.
 
+Modern kernels support "multi-size THP" (mTHP), which introduces the
+ability to allocate memory in blocks that are bigger than a base page
+but smaller than traditional PMD-size (as described above), in
+increments of a power-of-2 number of pages. mTHP can back anonymous
+memory (for example 16K, 32K, 64K, etc). These THPs continue to be
+PTE-mapped, but in many cases can still provide similar benefits to
+those outlined above: Page faults are significantly reduced (by a
+factor of e.g. 4, 8, 16, etc), but latency spikes are much less
+prominent because the size of each page isn't as huge as the PMD-sized
+variant and there is less memory to clear in each page fault. Some
+architectures also employ TLB compression mechanisms to squeeze more
+entries in when a set of PTEs are virtually and physically contiguous
+and approporiately aligned. In this case, TLB misses will occur less
+often.
+
 THP can be enabled system wide or restricted to certain tasks or even
 memory ranges inside task's address space. Unless THP is completely
 disabled, there is ``khugepaged`` daemon that scans memory and
-collapses sequences of basic pages into huge pages.
+collapses sequences of basic pages into PMD-sized huge pages.
 
 The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>`
 interface and using madvise(2) and prctl(2) system calls.
@@ -95,12 +110,40 @@ Global THP controls
 Transparent Hugepage Support for anonymous memory can be entirely disabled
 (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE
 regions (to avoid the risk of consuming more memory resources) or enabled
-system wide. This can be achieved with one of::
+system wide. This can be achieved per-supported-THP-size with one of::
+
+	echo always >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+	echo madvise >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+	echo never >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+
+where <size> is the hugepage size being addressed, the available sizes
+for which vary by system.
+
+For example::
+
+	echo always >/sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled
+
+Alternatively it is possible to specify that a given hugepage size
+will inherit the top-level "enabled" value::
+
+	echo inherit >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+
+For example::
+
+	echo inherit >/sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled
+
+The top-level setting (for use with "inherit") can be set by issuing
+one of the following commands::
 
 	echo always >/sys/kernel/mm/transparent_hugepage/enabled
 	echo madvise >/sys/kernel/mm/transparent_hugepage/enabled
 	echo never >/sys/kernel/mm/transparent_hugepage/enabled
 
+By default, PMD-sized hugepages have enabled="inherit" and all other
+hugepage sizes have enabled="never". If enabling multiple hugepage
+sizes, the kernel will select the most appropriate enabled size for a
+given allocation.
+
 It's also possible to limit defrag efforts in the VM to generate
 anonymous hugepages in case they're not immediately free to madvise
 regions or to never try to defrag memory and simply fallback to regular
@@ -146,25 +189,34 @@ madvise
 never
 	should be self-explanatory.
 
-By default kernel tries to use huge zero page on read page fault to
-anonymous mapping. It's possible to disable huge zero page by writing 0
-or enable it back by writing 1::
+By default kernel tries to use huge, PMD-mappable zero page on read
+page fault to anonymous mapping. It's possible to disable huge zero
+page by writing 0 or enable it back by writing 1::
 
 	echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page
 	echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page
 
-Some userspace (such as a test program, or an optimized memory allocation
-library) may want to know the size (in bytes) of a transparent hugepage::
+Some userspace (such as a test program, or an optimized memory
+allocation library) may want to know the size (in bytes) of a
+PMD-mappable transparent hugepage::
 
 	cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
 
-khugepaged will be automatically started when
-transparent_hugepage/enabled is set to "always" or "madvise, and it'll
-be automatically shutdown if it's set to "never".
+khugepaged will be automatically started when one or more hugepage
+sizes are enabled (either by directly setting "always" or "madvise",
+or by setting "inherit" while the top-level enabled is set to "always"
+or "madvise"), and it'll be automatically shutdown when the last
+hugepage size is disabled (either by directly setting "never", or by
+setting "inherit" while the top-level enabled is set to "never").
 
 Khugepaged controls
 -------------------
 
+.. note::
+   khugepaged currently only searches for opportunities to collapse to
+   PMD-sized THP and no attempt is made to collapse to other THP
+   sizes.
+
 khugepaged runs usually at low frequency so while one may not want to
 invoke defrag algorithms synchronously during the page faults, it
 should be worth invoking defrag at least in khugepaged. However it's
@@ -282,19 +334,26 @@ force
 Need of application restart
 ===========================
 
-The transparent_hugepage/enabled values and tmpfs mount option only affect
-future behavior. So to make them effective you need to restart any
-application that could have been using hugepages. This also applies to the
-regions registered in khugepaged.
+The transparent_hugepage/enabled and
+transparent_hugepage/hugepages-<size>kB/enabled values and tmpfs mount
+option only affect future behavior. So to make them effective you need
+to restart any application that could have been using hugepages. This
+also applies to the regions registered in khugepaged.
 
 Monitoring usage
 ================
 
-The number of anonymous transparent huge pages currently used by the
+.. note::
+   Currently the below counters only record events relating to
+   PMD-sized THP. Events relating to other THP sizes are not included.
+
+The number of PMD-sized anonymous transparent huge pages currently used by the
 system is available by reading the AnonHugePages field in ``/proc/meminfo``.
-To identify what applications are using anonymous transparent huge pages,
-it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages fields
-for each mapping.
+To identify what applications are using PMD-sized anonymous transparent huge
+pages, it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages
+fields for each mapping. (Note that AnonHugePages only applies to traditional
+PMD-sized THP for historical reasons and should have been called
+AnonHugePmdMapped).
 
 The number of file transparent huge pages mapped to userspace is available
 by reading ShmemPmdMapped and ShmemHugePages fields in ``/proc/meminfo``.
@@ -413,7 +472,7 @@ for huge pages.
 Optimizing the applications
 ===========================
 
-To be guaranteed that the kernel will map a 2M page immediately in any
+To be guaranteed that the kernel will map a THP immediately in any
 memory region, the mmap region has to be hugepage naturally
 aligned. posix_memalign() can provide that guarantee.
 
Documentation/filesystems/proc.rst +3 -3
index 49ef12df631b..104c6d047d9b 100644
@@ -528,9 +528,9 @@ replaced by copy-on-write) part of the underlying shmem object out on swap.
 does not take into account swapped out page of underlying shmem objects.
 "Locked" indicates whether the mapping is locked in memory or not.
 
-"THPeligible" indicates whether the mapping is eligible for allocating THP
-pages as well as the THP is PMD mappable or not - 1 if true, 0 otherwise.
-It just shows the current status.
+"THPeligible" indicates whether the mapping is eligible for allocating
+naturally aligned THP pages of any currently enabled size. 1 if true, 0
+otherwise.
 
 "VmFlags" field deserves a separate description. This member represents the
 kernel flags associated with the particular virtual memory area in two letter
fs/proc/task_mmu.c +2 -1
index d19924bf0a39..79855e1c5b57 100644
@@ -865,7 +865,8 @@ static int show_smap(struct seq_file *m, void *v)
 	__show_smap(m, &mss, false);
 
 	seq_printf(m, "THPeligible:    %8u\n",
-		   hugepage_vma_check(vma, vma->vm_flags, true, false, true));
+		   !!thp_vma_allowable_orders(vma, vma->vm_flags, true, false,
+					      true, THP_ORDERS_ALL));
 
 	if (arch_pkeys_enabled())
 		seq_printf(m, "ProtectionKey:  %8u\n", vma_pkey(vma));
include/linux/huge_mm.h +155 -26

231 lines not on this page · raw message

mm/huge_memory.c +204 -27

317 lines not on this page · raw message

mm/khugepaged.c +12 -8
index 0da6937572cf..de174d049e71 100644
@@ -446,7 +446,8 @@ void khugepaged_enter_vma(struct vm_area_struct *vma,
 {
 	if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) &&
 	    hugepage_flags_enabled()) {
-		if (hugepage_vma_check(vma, vm_flags, false, false, true))
+		if (thp_vma_allowable_order(vma, vm_flags, false, false, true,
+					    PMD_ORDER))
 			__khugepaged_enter(vma->vm_mm);
 	}
 }
@@ -922,16 +923,16 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,
 	if (!vma)
 		return SCAN_VMA_NULL;
 
-	if (!transhuge_vma_suitable(vma, address))
+	if (!thp_vma_suitable_order(vma, address, PMD_ORDER))
 		return SCAN_ADDRESS_RANGE;
-	if (!hugepage_vma_check(vma, vma->vm_flags, false, false,
-				cc->is_khugepaged))
+	if (!thp_vma_allowable_order(vma, vma->vm_flags, false, false,
+				     cc->is_khugepaged, PMD_ORDER))
 		return SCAN_VMA_CHECK;
 	/*
 	 * Anon VMA expected, the address may be unmapped then
 	 * remapped to file after khugepaged reaquired the mmap_lock.
 	 *
-	 * hugepage_vma_check may return true for qualified file
+	 * thp_vma_allowable_order may return true for qualified file
 	 * vmas.
 	 */
 	if (expect_anon && (!(*vmap)->anon_vma || !vma_is_anonymous(*vmap)))
@@ -1506,7 +1507,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
 	 * and map it by a PMD, regardless of sysfs THP settings. As such, let's
 	 * analogously elide sysfs THP settings here.
 	 */
-	if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false))
+	if (!thp_vma_allowable_order(vma, vma->vm_flags, false, false, false,
+				     PMD_ORDER))
 		return SCAN_VMA_CHECK;
 
 	/* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */
@@ -2371,7 +2373,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,
 			progress++;
 			break;
 		}
-		if (!hugepage_vma_check(vma, vma->vm_flags, false, false, true)) {
+		if (!thp_vma_allowable_order(vma, vma->vm_flags, false, false,
+					     true, PMD_ORDER)) {
 skip:
 			progress++;
 			continue;
@@ -2708,7 +2711,8 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev,
 
 	*prev = vma;
 
-	if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false))
+	if (!thp_vma_allowable_order(vma, vma->vm_flags, false, false, false,
+				     PMD_ORDER))
 		return -EINVAL;
 
 	cc = kmalloc(sizeof(*cc), GFP_KERNEL);
mm/memory.c +3 -3
index 99582b188ed2..8ab2d994d997 100644
@@ -4322,7 +4322,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page)
 	pmd_t entry;
 	vm_fault_t ret = VM_FAULT_FALLBACK;
 
-	if (!transhuge_vma_suitable(vma, haddr))
+	if (!thp_vma_suitable_order(vma, haddr, PMD_ORDER))
 		return ret;
 
 	page = compound_head(page);
@@ -5116,7 +5116,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
 		return VM_FAULT_OOM;
 retry_pud:
 	if (pud_none(*vmf.pud) &&
-	    hugepage_vma_check(vma, vm_flags, false, true, true)) {
+	    thp_vma_allowable_order(vma, vm_flags, false, true, true, PUD_ORDER)) {
 		ret = create_huge_pud(&vmf);
 		if (!(ret & VM_FAULT_FALLBACK))
 			return ret;
@@ -5150,7 +5150,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
 		goto retry_pud;
 
 	if (pmd_none(*vmf.pmd) &&
-	    hugepage_vma_check(vma, vm_flags, false, true, true)) {
+	    thp_vma_allowable_order(vma, vm_flags, false, true, true, PMD_ORDER)) {
 		ret = create_huge_pmd(&vmf);
 		if (!(ret & VM_FAULT_FALLBACK))
 			return ret;
mm/page_vma_mapped.c +2 -1
index e0b368e545ed..74d2de15fb5e 100644
@@ -268,7 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)
 			 * cleared *pmd but not decremented compound_mapcount().
 			 */
 			if ((pvmw->flags & PVMW_SYNC) &&
-			    transhuge_vma_suitable(vma, pvmw->address) &&
+			    thp_vma_suitable_order(vma, pvmw->address,
+						   PMD_ORDER) &&
 			    (pvmw->nr_pages >= HPAGE_PMD_NR)) {
 				spinlock_t *ptl = pmd_lock(mm, pvmw->pmd);
 
Signature · 1 line
2.25.1
Acked-by on v9 3/10
#16 lore raw

Re: [PATCH v9 03/10] mm: thp: Introduce multi-size THP sysfs interface

Quoted 36 lines · Ryan Roberts

In preparation for adding support for anonymous multi-size THP, introduce new sysfs structure that will be used to control the new behaviours. A new directory is added under transparent_hugepage for each supported THP size, and contains an enabled file, which can be set to "inherit" (to inherit the global setting), "always", "madvise" or "never". For now, the kernel still only supports PMD-sized anonymous THP, so only 1 directory is populated.

The first half of the change converts transhuge_vma_suitable() and hugepage_vma_check() so that they take a bitfield of orders for which the user wants to determine support, and the functions filter out all the orders that can't be supported, given the current sysfs configuration and the VMA dimensions. The resulting functions are renamed to thp_vma_suitable_orders() and thp_vma_allowable_orders() respectively. Convenience functions that take a single, unencoded order and return a boolean are also defined as thp_vma_suitable_order() and thp_vma_allowable_order().

The second half of the change implements the new sysfs interface. It has been done so that each supported THP size has a struct thpsize, which describes the relevant metadata and is itself a kobject. This is pretty minimal for now, but should make it easy to add new per-thpsize files to the interface if needed in future (e.g. per-size defrag). Rather than keep the enabled state directly in the struct thpsize, I've elected to directly encode it into huge_anon_orders_[always|madvise|inherit] bitfields since this reduces the amount of work required in thp_vma_allowable_orders() which is called for every page fault.

See Documentation/admin-guide/mm/transhuge.rst, as modified by this commit, for details of how the new sysfs interface works.

---

[...]

Quoted 22 lines
mm/huge_memory.c +22 -0
+
+static ssize_t thpsize_enabled_store(struct kobject *kobj,
+				     struct kobj_attribute *attr,
+				     const char *buf, size_t count)
+{
+	int order = to_thpsize(kobj)->order;
+	ssize_t ret = count;
+
+	if (sysfs_streq(buf, "always")) {
+		spin_lock(&huge_anon_orders_lock);
+		clear_bit(order, &huge_anon_orders_inherit);
+		clear_bit(order, &huge_anon_orders_madvise);
+		set_bit(order, &huge_anon_orders_always);
+		spin_unlock(&huge_anon_orders_lock);
+	} else if (sysfs_streq(buf, "inherit")) {
+		spin_lock(&huge_anon_orders_lock);
+		clear_bit(order, &huge_anon_orders_always);
+		clear_bit(order, &huge_anon_orders_madvise);
+		set_bit(order, &huge_anon_orders_inherit);
+		spin_unlock(&huge_anon_orders_lock);
+	} else if (sysfs_streq(buf, "madvise")) {
+		spin_lock(&huge_anon_orders_lock);

mm/huge_memory.c static const struct attribute_group hugepage_attr_group = { · line 503

Quoted 10 lines
mm/huge_memory.c +10 -0
+		clear_bit(order, &huge_anon_orders_always);
+		clear_bit(order, &huge_anon_orders_inherit);
+		set_bit(order, &huge_anon_orders_madvise);
+		spin_unlock(&huge_anon_orders_lock);
+	} else if (sysfs_streq(buf, "never")) {
+		spin_lock(&huge_anon_orders_lock);
+		clear_bit(order, &huge_anon_orders_always);
+		clear_bit(order, &huge_anon_orders_inherit);
+		clear_bit(order, &huge_anon_orders_madvise);
+		spin_unlock(&huge_anon_orders_lock);

Why not perform lock/unlock only once in surrounding code? :)

Much better

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#17 lore raw

Re: [PATCH v9 03/10] mm: thp: Introduce multi-size THP sysfs interface

Quoted 66 lines · David Hildenbrand
Quoted 36 lines · Ryan Roberts

In preparation for adding support for anonymous multi-size THP, introduce new sysfs structure that will be used to control the new behaviours. A new directory is added under transparent_hugepage for each supported THP size, and contains an enabled file, which can be set to "inherit" (to inherit the global setting), "always", "madvise" or "never". For now, the kernel still only supports PMD-sized anonymous THP, so only 1 directory is populated.

The first half of the change converts transhuge_vma_suitable() and hugepage_vma_check() so that they take a bitfield of orders for which the user wants to determine support, and the functions filter out all the orders that can't be supported, given the current sysfs configuration and the VMA dimensions. The resulting functions are renamed to thp_vma_suitable_orders() and thp_vma_allowable_orders() respectively. Convenience functions that take a single, unencoded order and return a boolean are also defined as thp_vma_suitable_order() and thp_vma_allowable_order().

The second half of the change implements the new sysfs interface. It has been done so that each supported THP size has a struct thpsize, which describes the relevant metadata and is itself a kobject. This is pretty minimal for now, but should make it easy to add new per-thpsize files to the interface if needed in future (e.g. per-size defrag). Rather than keep the enabled state directly in the struct thpsize, I've elected to directly encode it into huge_anon_orders_[always|madvise|inherit] bitfields since this reduces the amount of work required in thp_vma_allowable_orders() which is called for every page fault.

See Documentation/admin-guide/mm/transhuge.rst, as modified by this commit, for details of how the new sysfs interface works.

---

[...]

Quoted 26 lines
mm/huge_memory.c +26 -0
+
+static ssize_t thpsize_enabled_store(struct kobject *kobj,
+                     struct kobj_attribute *attr,
+                     const char *buf, size_t count)
+{
+    int order = to_thpsize(kobj)->order;
+    ssize_t ret = count;
+
+    if (sysfs_streq(buf, "always")) {
+        spin_lock(&huge_anon_orders_lock);
+        clear_bit(order, &huge_anon_orders_inherit);
+        clear_bit(order, &huge_anon_orders_madvise);
+        set_bit(order, &huge_anon_orders_always);
+        spin_unlock(&huge_anon_orders_lock);
+    } else if (sysfs_streq(buf, "inherit")) {
+        spin_lock(&huge_anon_orders_lock);
+        clear_bit(order, &huge_anon_orders_always);
+        clear_bit(order, &huge_anon_orders_madvise);
+        set_bit(order, &huge_anon_orders_inherit);
+        spin_unlock(&huge_anon_orders_lock);
+    } else if (sysfs_streq(buf, "madvise")) {
+        spin_lock(&huge_anon_orders_lock);
+        clear_bit(order, &huge_anon_orders_always);
+        clear_bit(order, &huge_anon_orders_inherit);
+        set_bit(order, &huge_anon_orders_madvise);
+        spin_unlock(&huge_anon_orders_lock);
Quoted 8 lines
Quoted 6 lines

+    } else if (sysfs_streq(buf, "never")) { +        spin_lock(&huge_anon_orders_lock);
+        clear_bit(order, &huge_anon_orders_always); +        clear_bit(order, &huge_anon_orders_inherit); +        clear_bit(order, &huge_anon_orders_madvise); +        spin_unlock(&huge_anon_orders_lock);

Why not perform lock/unlock only once in surrounding code? :)

I was nervous that sysfs_streq() may be unhappy in atomic context... Unfounded?

Quoted 6 lines

Much better

maintainer
#18 lore raw

Re: [PATCH v9 03/10] mm: thp: Introduce multi-size THP sysfs interface

Quoted 4 lines · On Tue, 12 Dec 2023 15:32:29 +0000 Ryan Roberts <[email protected]> wrote:
Quoted 1 line

Why not perform lock/unlock only once in surrounding code? :)

I was nervous that sysfs_streq() may be unhappy in atomic context... Unfounded?

Yes, unfounded.

author reviewer
#19 lore raw

PATCH v9 4/10 mm: thp: Support allocation of anonymous multi-size THP

In reply to Ryan Roberts

Introduce the logic to allow THP to be configured (through the new sysfs interface we just added) to allocate large folios to back anonymous memory, which are larger than the base page size but smaller than PMD-size. We call this new THP extension "multi-size THP" (mTHP).

mTHP continues to be PTE-mapped, but in many cases can still provide similar benefits to traditional PMD-sized THP: Page faults are significantly reduced (by a factor of e.g. 4, 8, 16, etc. depending on the configured order), but latency spikes are much less prominent because the size of each page isn't as huge as the PMD-sized variant and there is less memory to clear in each page fault. The number of per-page operations (e.g. ref counting, rmap management, lru list management) are also significantly reduced since those ops now become per-folio.

Some architectures also employ TLB compression mechanisms to squeeze more entries in when a set of PTEs are virtually and physically contiguous and approporiately aligned. In this case, TLB misses will occur less often.

The new behaviour is disabled by default, but can be enabled at runtime by writing to /sys/kernel/mm/transparent_hugepage/hugepage-XXkb/enabled (see documentation in previous commit). The long term aim is to change the default to include suitable lower orders, but there are some risks around internal fragmentation that need to be better understood first.

---
 include/linux/huge_mm.h |   6 ++-
 mm/memory.c             | 111 ++++++++++++++++++++++++++++++++++++----
 2 files changed, 106 insertions(+), 11 deletions(-)
include/linux/huge_mm.h +4 -2
index 609c153bae57..fa7a38a30fc6 100644
@@ -68,9 +68,11 @@ extern struct kobj_attribute shmem_enabled_attr;
 #define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER)
 
 /*
- * Mask of all large folio orders supported for anonymous THP.
+ * Mask of all large folio orders supported for anonymous THP; all orders up to
+ * and including PMD_ORDER, except order-0 (which is not "huge") and order-1
+ * (which is a limitation of the THP implementation).
  */
-#define THP_ORDERS_ALL_ANON	BIT(PMD_ORDER)
+#define THP_ORDERS_ALL_ANON	((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
 
 /*
  * Mask of all large folio orders supported for file THP.
mm/memory.c +102 -9

155 lines not on this page · raw message

Signature · 1 line
2.25.1
Acked-by on v9 4/10
#20 lore raw

Re: [PATCH v9 04/10] mm: thp: Support allocation of anonymous multi-size THP

Quoted 5 lines · Ryan Roberts

Introduce the logic to allow THP to be configured (through the new sysfs interface we just added) to allocate large folios to back anonymous memory, which are larger than the base page size but smaller than PMD-size. We call this new THP extension "multi-size THP" (mTHP).

Quoted 8 lines

mTHP continues to be PTE-mapped, but in many cases can still provide similar benefits to traditional PMD-sized THP: Page faults are significantly reduced (by a factor of e.g. 4, 8, 16, etc. depending on the configured order), but latency spikes are much less prominent because the size of each page isn't as huge as the PMD-sized variant and there is less memory to clear in each page fault. The number of per-page operations (e.g. ref counting, rmap management, lru list management) are also significantly reduced since those ops now become per-folio.

I'll note that with always-pte-mapped-thp it will be much easier to support incremental page clearing (e.g., zero only parts of the folio and map the remainder in a pro-non-like fashion whereby we'll zero on the next page fault). With a PMD-sized thp, you have to eventually place/rip out page tables to achieve that.

Quoted 26 lines

Some architectures also employ TLB compression mechanisms to squeeze more entries in when a set of PTEs are virtually and physically contiguous and approporiately aligned. In this case, TLB misses will occur less often.

The new behaviour is disabled by default, but can be enabled at runtime by writing to /sys/kernel/mm/transparent_hugepage/hugepage-XXkb/enabled (see documentation in previous commit). The long term aim is to change the default to include suitable lower orders, but there are some risks around internal fragmentation that need to be better understood first.

---
  include/linux/huge_mm.h |   6 ++-
  mm/memory.c             | 111 ++++++++++++++++++++++++++++++++++++----
  2 files changed, 106 insertions(+), 11 deletions(-)
include/linux/huge_mm.h +0 -0
index 609c153bae57..fa7a38a30fc6 100644
@@ -68,9 +68,11 @@ extern struct kobj_attribute shmem_enabled_attr;
  #define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER)

[...]

Quoted 59 lines
include/linux/huge_mm.h +59 -0
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static struct folio *alloc_anon_folio(struct vm_fault *vmf)
+{
+	struct vm_area_struct *vma = vmf->vma;
+	unsigned long orders;
+	struct folio *folio;
+	unsigned long addr;
+	pte_t *pte;
+	gfp_t gfp;
+	int order;
+
+	/*
+	 * If uffd is active for the vma we need per-page fault fidelity to
+	 * maintain the uffd semantics.
+	 */
+	if (unlikely(userfaultfd_armed(vma)))
+		goto fallback;
+
+	/*
+	 * Get a list of all the (large) orders below PMD_ORDER that are enabled
+	 * for this vma. Then filter out the orders that can't be allocated over
+	 * the faulting address and still be fully contained in the vma.
+	 */
+	orders = thp_vma_allowable_orders(vma, vma->vm_flags, false, true, true,
+					  BIT(PMD_ORDER) - 1);
+	orders = thp_vma_suitable_orders(vma, vmf->address, orders);
+
+	if (!orders)
+		goto fallback;
+
+	pte = pte_offset_map(vmf->pmd, vmf->address & PMD_MASK);
+	if (!pte)
+		return ERR_PTR(-EAGAIN);
+
+	/*
+	 * Find the highest order where the aligned range is completely
+	 * pte_none(). Note that all remaining orders will be completely
+	 * pte_none().
+	 */
+	order = highest_order(orders);
+	while (orders) {
+		addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+		if (pte_range_none(pte + pte_index(addr), 1 << order))
+			break;
+		order = next_order(&orders, order);
+	}
+
+	pte_unmap(pte);
+
+	/* Try allocating the highest of the remaining orders. */
+	gfp = vma_thp_gfp_mask(vma);
+	while (orders) {
+		addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+		folio = vma_alloc_folio(gfp, order, vma, addr, true);
+		if (folio) {
+			clear_huge_page(&folio->page, vmf->address, 1 << order);
+			return folio;
+		}

include/linux/huge_mm.h

Quoted 10 lines
include/linux/huge_mm.h +10 -0
+		order = next_order(&orders, order);
+	}
+
+fallback:
+	return vma_alloc_zeroed_movable_folio(vma, vmf->address);
+}
+#else
+#define alloc_anon_folio(vmf) \
+		vma_alloc_zeroed_movable_folio((vmf)->vma, (vmf)->address)
+#endif

A neater alternative might be

static struct folio *alloc_anon_folio(struct vm_fault *vmf) {

#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	/* magic */
fallback:
#endif
	return vma_alloc_zeroed_movable_folio((vmf)->vma, (vmf)->address):
}

[...]

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#21 lore raw

Re: [PATCH v9 04/10] mm: thp: Support allocation of anonymous multi-size THP

Quoted 15 lines · David Hildenbrand
Quoted 5 lines · Ryan Roberts

Introduce the logic to allow THP to be configured (through the new sysfs interface we just added) to allocate large folios to back anonymous memory, which are larger than the base page size but smaller than PMD-size. We call this new THP extension "multi-size THP" (mTHP).

Quoted 8 lines

mTHP continues to be PTE-mapped, but in many cases can still provide similar benefits to traditional PMD-sized THP: Page faults are significantly reduced (by a factor of e.g. 4, 8, 16, etc. depending on the configured order), but latency spikes are much less prominent because the size of each page isn't as huge as the PMD-sized variant and there is less memory to clear in each page fault. The number of per-page operations (e.g. ref counting, rmap management, lru list management) are also significantly reduced since those ops now become per-folio.

Quoted 5 lines

I'll note that with always-pte-mapped-thp it will be much easier to support incremental page clearing (e.g., zero only parts of the folio and map the remainder in a pro-non-like fashion whereby we'll zero on the next page fault). With a PMD-sized thp, you have to eventually place/rip out page tables to achieve that.

But then you lose the benefits of reduced number of page faults; reducing page faults gives a big speed up for workloads with lots of short lived processes like compiling.

But yes, I agree this could be an interesting future optimization for some workloads.

Quoted 102 lines
Quoted 27 lines

Some architectures also employ TLB compression mechanisms to squeeze more entries in when a set of PTEs are virtually and physically contiguous and approporiately aligned. In this case, TLB misses will occur less often.

The new behaviour is disabled by default, but can be enabled at runtime by writing to /sys/kernel/mm/transparent_hugepage/hugepage-XXkb/enabled (see documentation in previous commit). The long term aim is to change the default to include suitable lower orders, but there are some risks around internal fragmentation that need to be better understood first.

---
  include/linux/huge_mm.h |   6 ++-
  mm/memory.c             | 111 ++++++++++++++++++++++++++++++++++++----

  2 files changed, 106 insertions(+), 11 deletions(-)
include/linux/huge_mm.h +0 -0
index 609c153bae57..fa7a38a30fc6 100644
@@ -68,9 +68,11 @@ extern struct kobj_attribute shmem_enabled_attr;
   #define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER)

[...]

Quoted 59 lines
include/linux/huge_mm.h +59 -0
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static struct folio *alloc_anon_folio(struct vm_fault *vmf)
+{
+    struct vm_area_struct *vma = vmf->vma;
+    unsigned long orders;
+    struct folio *folio;
+    unsigned long addr;
+    pte_t *pte;
+    gfp_t gfp;
+    int order;
+
+    /*
+     * If uffd is active for the vma we need per-page fault fidelity to
+     * maintain the uffd semantics.
+     */
+    if (unlikely(userfaultfd_armed(vma)))
+        goto fallback;
+
+    /*
+     * Get a list of all the (large) orders below PMD_ORDER that are enabled
+     * for this vma. Then filter out the orders that can't be allocated over
+     * the faulting address and still be fully contained in the vma.
+     */
+    orders = thp_vma_allowable_orders(vma, vma->vm_flags, false, true, true,
+                      BIT(PMD_ORDER) - 1);
+    orders = thp_vma_suitable_orders(vma, vmf->address, orders);
+
+    if (!orders)
+        goto fallback;
+
+    pte = pte_offset_map(vmf->pmd, vmf->address & PMD_MASK);
+    if (!pte)
+        return ERR_PTR(-EAGAIN);
+
+    /*
+     * Find the highest order where the aligned range is completely
+     * pte_none(). Note that all remaining orders will be completely
+     * pte_none().
+     */
+    order = highest_order(orders);
+    while (orders) {
+        addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+        if (pte_range_none(pte + pte_index(addr), 1 << order))
+            break;
+        order = next_order(&orders, order);
+    }
+
+    pte_unmap(pte);
+
+    /* Try allocating the highest of the remaining orders. */
+    gfp = vma_thp_gfp_mask(vma);
+    while (orders) {
+        addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+        folio = vma_alloc_folio(gfp, order, vma, addr, true);
+        if (folio) {
+            clear_huge_page(&folio->page, vmf->address, 1 << order);
+            return folio;
+        }

include/linux/huge_mm.h

Quoted 10 lines
include/linux/huge_mm.h +10 -0
+        order = next_order(&orders, order);
+    }
+
+fallback:
+    return vma_alloc_zeroed_movable_folio(vma, vmf->address);
+}
+#else
+#define alloc_anon_folio(vmf) \
+        vma_alloc_zeroed_movable_folio((vmf)->vma, (vmf)->address)
+#endif

A neater alternative might be

Quoted 8 lines

static struct folio *alloc_anon_folio(struct vm_fault *vmf) {
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
    /* magic */
fallback:
#endif
    return vma_alloc_zeroed_movable_folio((vmf)->vma, (vmf)->address):
}

I guess beauty lies in the eye of the beholder... I don't find it much neater personally :). But happy to make the change if you insist; what's the process now that its in mm-unstable? Just send a patch to Andrew for squashing?

Quoted 5 lines

[...]

#22 lore raw

Re: [PATCH v9 04/10] mm: thp: Support allocation of anonymous multi-size THP

Quoted 22 lines · Ryan Roberts
Quoted 15 lines · David Hildenbrand
Quoted 5 lines · Ryan Roberts

Introduce the logic to allow THP to be configured (through the new sysfs interface we just added) to allocate large folios to back anonymous memory, which are larger than the base page size but smaller than PMD-size. We call this new THP extension "multi-size THP" (mTHP).

Quoted 8 lines

mTHP continues to be PTE-mapped, but in many cases can still provide similar benefits to traditional PMD-sized THP: Page faults are significantly reduced (by a factor of e.g. 4, 8, 16, etc. depending on the configured order), but latency spikes are much less prominent because the size of each page isn't as huge as the PMD-sized variant and there is less memory to clear in each page fault. The number of per-page operations (e.g. ref counting, rmap management, lru list management) are also significantly reduced since those ops now become per-folio.

Quoted 5 lines

I'll note that with always-pte-mapped-thp it will be much easier to support incremental page clearing (e.g., zero only parts of the folio and map the remainder in a pro-non-like fashion whereby we'll zero on the next page fault). With a PMD-sized thp, you have to eventually place/rip out page tables to achieve that.

Quoted 3 lines

But then you lose the benefits of reduced number of page faults; reducing page faults gives a big speed up for workloads with lots of short lived processes like compiling.

Well, you can do interesting things like "allocate order-5", but zero in order-3 chunks. You get less page faults and pay for alloc/rmap only once.

But yes, all has pros and cons.

[...]

Quoted 111 lines
Quoted 102 lines
Quoted 27 lines

Some architectures also employ TLB compression mechanisms to squeeze more entries in when a set of PTEs are virtually and physically contiguous and approporiately aligned. In this case, TLB misses will occur less often.

The new behaviour is disabled by default, but can be enabled at runtime by writing to /sys/kernel/mm/transparent_hugepage/hugepage-XXkb/enabled (see documentation in previous commit). The long term aim is to change the default to include suitable lower orders, but there are some risks around internal fragmentation that need to be better understood first.

---
   include/linux/huge_mm.h |   6 ++-
   mm/memory.c             | 111 ++++++++++++++++++++++++++++++++++++----
   2 files changed, 106 insertions(+), 11 deletions(-)
include/linux/huge_mm.h +0 -0
index 609c153bae57..fa7a38a30fc6 100644
@@ -68,9 +68,11 @@ extern struct kobj_attribute shmem_enabled_attr;
   #define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER)

[...]

Quoted 59 lines
include/linux/huge_mm.h +59 -0
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static struct folio *alloc_anon_folio(struct vm_fault *vmf)
+{
+    struct vm_area_struct *vma = vmf->vma;
+    unsigned long orders;
+    struct folio *folio;
+    unsigned long addr;
+    pte_t *pte;
+    gfp_t gfp;
+    int order;
+
+    /*
+     * If uffd is active for the vma we need per-page fault fidelity to
+     * maintain the uffd semantics.
+     */
+    if (unlikely(userfaultfd_armed(vma)))
+        goto fallback;
+
+    /*
+     * Get a list of all the (large) orders below PMD_ORDER that are enabled
+     * for this vma. Then filter out the orders that can't be allocated over
+     * the faulting address and still be fully contained in the vma.
+     */
+    orders = thp_vma_allowable_orders(vma, vma->vm_flags, false, true, true,
+                      BIT(PMD_ORDER) - 1);
+    orders = thp_vma_suitable_orders(vma, vmf->address, orders);
+
+    if (!orders)
+        goto fallback;
+
+    pte = pte_offset_map(vmf->pmd, vmf->address & PMD_MASK);
+    if (!pte)
+        return ERR_PTR(-EAGAIN);
+
+    /*
+     * Find the highest order where the aligned range is completely
+     * pte_none(). Note that all remaining orders will be completely
+     * pte_none().
+     */
+    order = highest_order(orders);
+    while (orders) {
+        addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+        if (pte_range_none(pte + pte_index(addr), 1 << order))
+            break;
+        order = next_order(&orders, order);
+    }
+
+    pte_unmap(pte);
+
+    /* Try allocating the highest of the remaining orders. */
+    gfp = vma_thp_gfp_mask(vma);
+    while (orders) {
+        addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+        folio = vma_alloc_folio(gfp, order, vma, addr, true);
+        if (folio) {
+            clear_huge_page(&folio->page, vmf->address, 1 << order);
+            return folio;
+        }

include/linux/huge_mm.h

Quoted 10 lines
include/linux/huge_mm.h +10 -0
+        order = next_order(&orders, order);
+    }
+
+fallback:
+    return vma_alloc_zeroed_movable_folio(vma, vmf->address);
+}
+#else
+#define alloc_anon_folio(vmf) \
+        vma_alloc_zeroed_movable_folio((vmf)->vma, (vmf)->address)
+#endif

A neater alternative might be

Quoted 8 lines

static struct folio *alloc_anon_folio(struct vm_fault *vmf) {
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
    /* magic */
fallback:
#endif
    return vma_alloc_zeroed_movable_folio((vmf)->vma, (vmf)->address):
}

Quoted 3 lines

I guess beauty lies in the eye of the beholder... I don't find it much neater personally :). But happy to make the change if you insist; what's the process now that its in mm-unstable? Just send a patch to Andrew for squashing?

That way it is clear that the fallback for thp is just what !thp does.

But either is fine for me; no need to change if you disagree.

Signature · 3 lines
Cheers,

David / dhildenb
#23 lore raw

Re: [PATCH v9 04/10] mm: thp: Support allocation of anonymous multi-size THP

In reply to Ryan Roberts

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4267

Quoted 10 lines · Ryan Roberts
mm/memory.c +3 -1
@@ -4176,10 +4260,15 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	/* Allocate our own private page. */
 	if (unlikely(anon_vma_prepare(vma)))
 		goto oom;
-	folio = vma_alloc_zeroed_movable_folio(vma, vmf->address);
+	folio = alloc_anon_folio(vmf);
+	if (IS_ERR(folio))
+		return 0;
 	if (!folio)
 		goto oom;

Returning zero is weird. I think it should be a vm_fault_t code.

This mixing of error pointers and NULL is going to cause problems. Normally when we have a mix of error pointers and NULL then the NULL is not an error but instead means that the feature has been deliberately turned off. I'm unable to figure out what the meaning is here.

It should return one or the other, or if it's a mix then add a giant comment explaining what they mean.

regards,
dan carpenter

Quoted 7 lines
mm/memory.c +3 -0
+	nr_pages = folio_nr_pages(folio);
+	addr = ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE);
+
 	if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL))
 		goto oom_free_page;
 	folio_throttle_swaprate(folio, GFP_KERNEL);
author reviewer
#24 lore raw

Re: [PATCH v9 04/10] mm: thp: Support allocation of anonymous multi-size THP

Quoted 13 lines · Dan Carpenter

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4267

Quoted 10 lines · Ryan Roberts
mm/memory.c +3 -1
@@ -4176,10 +4260,15 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	/* Allocate our own private page. */
 	if (unlikely(anon_vma_prepare(vma)))
 		goto oom;
-	folio = vma_alloc_zeroed_movable_folio(vma, vmf->address);
+	folio = alloc_anon_folio(vmf);
+	if (IS_ERR(folio))
+		return 0;
 	if (!folio)
 		goto oom;

Returning zero is weird. I think it should be a vm_fault_t code.

It's the same pattern that the existing code a little further down this function already implements:

	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl);
	if (!vmf->pte)
		goto release;

If we fail to map/lock the pte (due to a race), then we return 0 to allow user space to rerun the faulting instruction and cause the fault to happen again. The above code ends up calling "return ret;" and ret is 0.

Quoted 5 lines

This mixing of error pointers and NULL is going to cause problems. Normally when we have a mix of error pointers and NULL then the NULL is not an error but instead means that the feature has been deliberately turned off. I'm unable to figure out what the meaning is here.

There are 3 conditions that the function can return:

- folio successfully allocated
- folio failed to be allocated due to OOM
- fault needs to be tried again due to losing race

Previously only the first 2 conditions were possible and they were indicated by NULL/not-NULL. The new 3rd condition is only possible when THP is compile-time enabled. So it keeps the logic simpler to keep the NULL/not-NULL distinction for the first 2, and use the error code for the final one.

There are IS_ERR() and IS_ERR_OR_NULL() variants so I assume a pattern where you can have pointer, error or NULL is somewhat common already?

Thanks,
Ryan

Quoted 15 lines

It should return one or the other, or if it's a mix then add a giant comment explaining what they mean.

regards,
dan carpenter

Quoted 7 lines
mm/memory.c +3 -0
+	nr_pages = folio_nr_pages(folio);
+	addr = ALIGN_DOWN(vmf->address, nr_pages * PAGE_SIZE);
+
 	if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL))
 		goto oom_free_page;
 	folio_throttle_swaprate(folio, GFP_KERNEL);
#25 lore raw

Re: [PATCH v9 04/10] mm: thp: Support allocation of anonymous multi-size THP

Quoted 18 lines · Ryan Roberts
Quoted 13 lines · Dan Carpenter

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4267

Quoted 10 lines · Ryan Roberts
mm/memory.c +3 -1
@@ -4176,10 +4260,15 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	/* Allocate our own private page. */
 	if (unlikely(anon_vma_prepare(vma)))
 		goto oom;
-	folio = vma_alloc_zeroed_movable_folio(vma, vmf->address);
+	folio = alloc_anon_folio(vmf);
+	if (IS_ERR(folio))
+		return 0;
 	if (!folio)
 		goto oom;

Returning zero is weird. I think it should be a vm_fault_t code.

It's the same pattern that the existing code a little further down this function already implements:

Quoted 8 lines
	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl);
	if (!vmf->pte)
		goto release;

If we fail to map/lock the pte (due to a race), then we return 0 to allow user space to rerun the faulting instruction and cause the fault to happen again. The above code ends up calling "return ret;" and ret is 0.

Ah, okay. Thanks!

Quoted 12 lines
Quoted 5 lines

This mixing of error pointers and NULL is going to cause problems. Normally when we have a mix of error pointers and NULL then the NULL is not an error but instead means that the feature has been deliberately turned off. I'm unable to figure out what the meaning is here.

There are 3 conditions that the function can return:

- folio successfully allocated
- folio failed to be allocated due to OOM
- fault needs to be tried again due to losing race

Quoted 7 lines

Previously only the first 2 conditions were possible and they were indicated by NULL/not-NULL. The new 3rd condition is only possible when THP is compile-time enabled. So it keeps the logic simpler to keep the NULL/not-NULL distinction for the first 2, and use the error code for the final one.

There are IS_ERR() and IS_ERR_OR_NULL() variants so I assume a pattern where you can have pointer, error or NULL is somewhat common already?

People are confused by this a lot so I have written a blog about it:

https://staticthinking.wordpress.com/2022/08/01/mixing-error-pointers-and-null/

The IS_ERR_OR_NULL() function should be used like this:

int blink_leds()

{
	led = get_leds();
	if (IS_ERR_OR_NULL(led))
		return PTR_ERR(led);  <-- NULL means zero/success
	return led->blink();
}

In the case of alloc_anon_folio(), I would be tempted to create a wrapper around it where NULL becomes ERR_PTR(-ENOMEM). But this is obviously fast path code and I haven't benchmarked it.

Adding a comment is the other option.

regards,
dan carpenter

author reviewer
#26 lore raw

Re: [PATCH v9 04/10] mm: thp: Support allocation of anonymous multi-size THP

Quoted 47 lines · Dan Carpenter
Quoted 18 lines · Ryan Roberts
Quoted 13 lines · Dan Carpenter

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4267

Quoted 10 lines · Ryan Roberts
mm/memory.c +3 -1
@@ -4176,10 +4260,15 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	/* Allocate our own private page. */
 	if (unlikely(anon_vma_prepare(vma)))
 		goto oom;
-	folio = vma_alloc_zeroed_movable_folio(vma, vmf->address);
+	folio = alloc_anon_folio(vmf);
+	if (IS_ERR(folio))
+		return 0;
 	if (!folio)
 		goto oom;

Returning zero is weird. I think it should be a vm_fault_t code.

It's the same pattern that the existing code a little further down this function already implements:

Quoted 8 lines
	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl);
	if (!vmf->pte)
		goto release;

If we fail to map/lock the pte (due to a race), then we return 0 to allow user space to rerun the faulting instruction and cause the fault to happen again. The above code ends up calling "return ret;" and ret is 0.

Ah, okay. Thanks!

Quoted 17 lines
Quoted 5 lines

This mixing of error pointers and NULL is going to cause problems. Normally when we have a mix of error pointers and NULL then the NULL is not an error but instead means that the feature has been deliberately turned off. I'm unable to figure out what the meaning is here.

There are 3 conditions that the function can return:

- folio successfully allocated
- folio failed to be allocated due to OOM
- fault needs to be tried again due to losing race

Previously only the first 2 conditions were possible and they were indicated by NULL/not-NULL. The new 3rd condition is only possible when THP is compile-time enabled. So it keeps the logic simpler to keep the NULL/not-NULL distinction for the first 2, and use the error code for the final one.

Quoted 6 lines
Quoted 2 lines

There are IS_ERR() and IS_ERR_OR_NULL() variants so I assume a pattern where you can have pointer, error or NULL is somewhat common already?

People are confused by this a lot so I have written a blog about it:

https://staticthinking.wordpress.com/2022/08/01/mixing-error-pointers-and-null/

Nice; thanks for the pointer :)

Quoted 11 lines

The IS_ERR_OR_NULL() function should be used like this:

int blink_leds()

{
	led = get_leds();
	if (IS_ERR_OR_NULL(led))
		return PTR_ERR(led);  <-- NULL means zero/success
	return led->blink();
}
Quoted 5 lines

In the case of alloc_anon_folio(), I would be tempted to create a wrapper around it where NULL becomes ERR_PTR(-ENOMEM). But this is obviously fast path code and I haven't benchmarked it.

Adding a comment is the other option.

I'll add a comment; as you say this is a fast path, and I'm actively being burned in similar places (on another series I'm working on) where an additional check is regressing performance significantly so not keen on risking it here.

Andrew, I'll fold in the David's suggested ifdef improvement at the same time. Would you prefer an additional patch to squash in, or a whole new version of the series to swap out with the existing patches in mm-unstable?

Quoted 4 lines

regards,
dan carpenter

author reviewer
#27 lore raw

PATCH 1/1 mm: Resolve some multi-size THP review nits

In reply to Dan Carpenter

Tidy code based on review feedback for final version of multi-size THP:

- Comment added to explain alloc_anon_folio() error protocol
- ifdefery simplified for alloc_anon_folio()

---
Hi Andrew,

Hopefully this is the final tweak. Could you please squash this with the "mm: thp: Support allocation of anonymous multi-size THP" patch in mm-unstable?

Or if you prefer me to re-post the entire series, just let me know.

Thanks,
Ryan

 mm/memory.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
mm/memory.c +4 -6
index 8f0b936b90b5..3c530b639559 100644
@@ -4137,9 +4137,9 @@ static bool pte_range_none(pte_t *pte, int nr_pages)
 	return true;
 }
 
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
 static struct folio *alloc_anon_folio(struct vm_fault *vmf)
 {
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
 	struct vm_area_struct *vma = vmf->vma;
 	unsigned long orders;
 	struct folio *folio;
@@ -4199,12 +4199,9 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)
 	}
 
 fallback:
-	return vma_alloc_zeroed_movable_folio(vma, vmf->address);
-}
-#else
-#define alloc_anon_folio(vmf) \
-		vma_alloc_zeroed_movable_folio((vmf)->vma, (vmf)->address)
 #endif
+	return vma_alloc_zeroed_movable_folio(vmf->vma, vmf->address);
+}
 
 /*
  * We enter with non-exclusive mmap_lock (to exclude vma changes,
@@ -4260,6 +4257,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	/* Allocate our own private page. */
 	if (unlikely(anon_vma_prepare(vma)))
 		goto oom;
+	/* Returns NULL on OOM or ERR_PTR(-EAGAIN) if we must retry the fault */
 	folio = alloc_anon_folio(vmf);
 	if (IS_ERR(folio))
 		return 0;

--
2.25.1

author reviewer
#28 lore raw

PATCH v9 5/10 selftests/mm/kugepaged: Restore thp settings at exit

In reply to Ryan Roberts

Previously, the saved thp settings would be restored upon a signal or at the natural end of the test suite. But there are some tests that directly call exit() upon failure. In this case, the thp settings were not being restored, which could then influence other tests.

Fix this by installing an atexit() handler to do the actual restore. The signal handler can now just call exit() and the atexit handler is invoked.

---
 tools/testing/selftests/mm/khugepaged.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)
tools/testing/selftests/mm/khugepaged.c +11 -6
index 030667cb5533..fc47a1c4944c 100644
@@ -374,18 +374,22 @@ static void pop_settings(void)
 	write_settings(current_settings());
 }
 
-static void restore_settings(int sig)
+static void restore_settings_atexit(void)
 {
 	if (skip_settings_restore)
-		goto out;
+		return;
 
 	printf("Restore THP and khugepaged settings...");
 	write_settings(&saved_settings);
 	success("OK");
-	if (sig)
-		exit(EXIT_FAILURE);
-out:
-	exit(exit_status);
+
+	skip_settings_restore = true;
+}
+
+static void restore_settings(int sig)
+{
+	/* exit() will invoke the restore_settings_atexit handler. */
+	exit(sig ? EXIT_FAILURE : exit_status);
 }
 
 static void save_settings(void)
@@ -415,6 +419,7 @@ static void save_settings(void)
 
 	success("OK");
 
+	atexit(restore_settings_atexit);
 	signal(SIGTERM, restore_settings);
 	signal(SIGINT, restore_settings);
 	signal(SIGHUP, restore_settings);
Signature · 1 line
2.25.1
author reviewer
#29 lore raw

PATCH v9 6/10 selftests/mm: Factor out thp settings management

In reply to Ryan Roberts

The khugepaged test has a useful framework for save/restore/pop/push of all thp settings via the sysfs interface. This will be useful to explicitly control multi-size THP settings in other tests, so let's move it out of khugepaged and into its own thp_settings.[c|h] utility.

---
 tools/testing/selftests/mm/Makefile       |   4 +-
 tools/testing/selftests/mm/khugepaged.c   | 346 ++--------------------
 tools/testing/selftests/mm/thp_settings.c | 296 ++++++++++++++++++
 tools/testing/selftests/mm/thp_settings.h |  71 +++++
 4 files changed, 391 insertions(+), 326 deletions(-)
 create mode 100644 tools/testing/selftests/mm/thp_settings.c
 create mode 100644 tools/testing/selftests/mm/thp_settings.h
tools/testing/selftests/mm/Makefile +2 -2
index dede0bcf97a3..2453add65d12 100644
@@ -117,8 +117,8 @@ TEST_FILES += va_high_addr_switch.sh
 
 include ../lib.mk
 
-$(TEST_GEN_PROGS): vm_util.c
-$(TEST_GEN_FILES): vm_util.c
+$(TEST_GEN_PROGS): vm_util.c thp_settings.c
+$(TEST_GEN_FILES): vm_util.c thp_settings.c
 
 $(OUTPUT)/uffd-stress: uffd-common.c
 $(OUTPUT)/uffd-unit-tests: uffd-common.c
tools/testing/selftests/mm/khugepaged.c +22 -324

475 lines not on this page · raw message

tools/testing/selftests/mm/thp_settings.c new file +296 -0

296 lines not on this page · raw message

tools/testing/selftests/mm/thp_settings.h new file +71 -0

71 lines not on this page · raw message

Signature · 1 line
2.25.1
author reviewer
#30 lore raw

PATCH v9 7/10 selftests/mm: Support multi-size THP interface in thp_settings

In reply to Ryan Roberts

Save and restore the new per-size hugepage enabled setting, if available on the running kernel.

Since the number of per-size directories is not fixed, solve this as simply as possible by catering for a maximum number in the thp_settings struct (20). Each array index is the order. The value of THP_NEVER is changed to 0 so that all of these new settings default to THP_NEVER and the user only needs to fill in the ones they want to enable.

---
 tools/testing/selftests/mm/khugepaged.c   |  3 ++
 tools/testing/selftests/mm/thp_settings.c | 55 ++++++++++++++++++++++-
 tools/testing/selftests/mm/thp_settings.h | 11 ++++-
 3 files changed, 67 insertions(+), 2 deletions(-)
tools/testing/selftests/mm/khugepaged.c +3 -0
index b15e7fd70176..7bd3baa9d34b 100644
@@ -1141,6 +1141,7 @@ static void parse_test_type(int argc, const char **argv)
 
 int main(int argc, const char **argv)
 {
+	int hpage_pmd_order;
 	struct thp_settings default_settings = {
 		.thp_enabled = THP_MADVISE,
 		.thp_defrag = THP_DEFRAG_ALWAYS,
@@ -1175,11 +1176,13 @@ int main(int argc, const char **argv)
 		exit(EXIT_FAILURE);
 	}
 	hpage_pmd_nr = hpage_pmd_size / page_size;
+	hpage_pmd_order = __builtin_ctz(hpage_pmd_nr);
 
 	default_settings.khugepaged.max_ptes_none = hpage_pmd_nr - 1;
 	default_settings.khugepaged.max_ptes_swap = hpage_pmd_nr / 8;
 	default_settings.khugepaged.max_ptes_shared = hpage_pmd_nr / 2;
 	default_settings.khugepaged.pages_to_scan = hpage_pmd_nr * 8;
+	default_settings.hugepages[hpage_pmd_order].enabled = THP_INHERIT;
 
 	save_settings();
 	thp_push_settings(&default_settings);
tools/testing/selftests/mm/thp_settings.c +54 -1

89 lines not on this page · raw message

tools/testing/selftests/mm/thp_settings.h +10 -1
index ff3d98c30617..71cbff05f4c7 100644
@@ -7,9 +7,10 @@
 #include <stdint.h>
 
 enum thp_enabled {
+	THP_NEVER,
 	THP_ALWAYS,
+	THP_INHERIT,
 	THP_MADVISE,
-	THP_NEVER,
 };
 
 enum thp_defrag {
@@ -29,6 +30,12 @@ enum shmem_enabled {
 	SHMEM_FORCE,
 };
 
+#define NR_ORDERS 20
+
+struct hugepages_settings {
+	enum thp_enabled enabled;
+};
+
 struct khugepaged_settings {
 	bool defrag;
 	unsigned int alloc_sleep_millisecs;
@@ -46,6 +53,7 @@ struct thp_settings {
 	bool use_zero_page;
 	struct khugepaged_settings khugepaged;
 	unsigned long read_ahead_kb;
+	struct hugepages_settings hugepages[NR_ORDERS];
 };
 
 int read_file(const char *path, char *buf, size_t buflen);
@@ -67,5 +75,6 @@ void thp_restore_settings(void);
 void thp_save_settings(void);
 
 void thp_set_read_ahead_path(char *path);
+unsigned long thp_supported_orders(void);
 
 #endif /* __THP_SETTINGS_H__ */
Signature · 1 line
2.25.1
author reviewer
#31 lore raw

PATCH v9 8/10 selftests/mm/khugepaged: Enlighten for multi-size THP

In reply to Ryan Roberts

The collapse_max_ptes_none test was previously failing when a THP size less than PMD-size had enabled="always". The root cause is because the test faults in 1 page less than the threshold it set for collapsing. But when THP is enabled always, we "over allocate" and therefore the threshold is passed, and collapse unexpectedly succeeds.

Solve this by enlightening khugepaged selftest. Add a command line option to pass in the desired THP size that should be used for all anonymous allocations. The harness will then explicitly configure a THP size as requested and modify the collapse_max_ptes_none test so that it faults in the threshold minus the number of pages in the configured THP size. If no command line option is provided, default to order 0, as per previous behaviour.

I chose to use an order in the command line interface, since this makes the interface agnostic of base page size, making it easier to invoke from run_vmtests.sh.

---
 tools/testing/selftests/mm/khugepaged.c   | 48 +++++++++++++++++------
 tools/testing/selftests/mm/run_vmtests.sh |  2 +
 2 files changed, 39 insertions(+), 11 deletions(-)
tools/testing/selftests/mm/khugepaged.c +37 -11
index 7bd3baa9d34b..829320a519e7 100644
@@ -28,6 +28,7 @@
 static unsigned long hpage_pmd_size;
 static unsigned long page_size;
 static int hpage_pmd_nr;
+static int anon_order;
 
 #define PID_SMAPS "/proc/self/smaps"
 #define TEST_FILE "collapse_test_file"
@@ -607,6 +608,11 @@ static bool is_tmpfs(struct mem_ops *ops)
 	return ops == &__file_ops && finfo.type == VMA_SHMEM;
 }
 
+static bool is_anon(struct mem_ops *ops)
+{
+	return ops == &__anon_ops;
+}
+
 static void alloc_at_fault(void)
 {
 	struct thp_settings settings = *thp_current_settings();
@@ -673,6 +679,7 @@ static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *o
 	int max_ptes_none = hpage_pmd_nr / 2;
 	struct thp_settings settings = *thp_current_settings();
 	void *p;
+	int fault_nr_pages = is_anon(ops) ? 1 << anon_order : 1;
 
 	settings.khugepaged.max_ptes_none = max_ptes_none;
 	thp_push_settings(&settings);
@@ -686,10 +693,10 @@ static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *o
 		goto skip;
 	}
 
-	ops->fault(p, 0, (hpage_pmd_nr - max_ptes_none - 1) * page_size);
+	ops->fault(p, 0, (hpage_pmd_nr - max_ptes_none - fault_nr_pages) * page_size);
 	c->collapse("Maybe collapse with max_ptes_none exceeded", p, 1,
 		    ops, !c->enforce_pte_scan_limits);
-	validate_memory(p, 0, (hpage_pmd_nr - max_ptes_none - 1) * page_size);
+	validate_memory(p, 0, (hpage_pmd_nr - max_ptes_none - fault_nr_pages) * page_size);
 
 	if (c->enforce_pte_scan_limits) {
 		ops->fault(p, 0, (hpage_pmd_nr - max_ptes_none) * page_size);
@@ -1076,7 +1083,7 @@ static void madvise_retracted_page_tables(struct collapse_context *c,
 
 static void usage(void)
 {
-	fprintf(stderr, "\nUsage: ./khugepaged <test type> [dir]\n\n");
+	fprintf(stderr, "\nUsage: ./khugepaged [OPTIONS] <test type> [dir]\n\n");
 	fprintf(stderr, "\t<test type>\t: <context>:<mem_type>\n");
 	fprintf(stderr, "\t<context>\t: [all|khugepaged|madvise]\n");
 	fprintf(stderr, "\t<mem_type>\t: [all|anon|file|shmem]\n");
@@ -1085,15 +1092,34 @@ static void usage(void)
 	fprintf(stderr,	"\tCONFIG_READ_ONLY_THP_FOR_FS=y\n");
 	fprintf(stderr, "\n\tif [dir] is a (sub)directory of a tmpfs mount, tmpfs must be\n");
 	fprintf(stderr,	"\tmounted with huge=madvise option for khugepaged tests to work\n");
+	fprintf(stderr,	"\n\tSupported Options:\n");
+	fprintf(stderr,	"\t\t-h: This help message.\n");
+	fprintf(stderr,	"\t\t-s: mTHP size, expressed as page order.\n");
+	fprintf(stderr,	"\t\t    Defaults to 0. Use this size for anon allocations.\n");
 	exit(1);
 }
 
-static void parse_test_type(int argc, const char **argv)
+static void parse_test_type(int argc, char **argv)
 {
+	int opt;
 	char *buf;
 	const char *token;
 
-	if (argc == 1) {
+	while ((opt = getopt(argc, argv, "s:h")) != -1) {
+		switch (opt) {
+		case 's':
+			anon_order = atoi(optarg);
+			break;
+		case 'h':
+		default:
+			usage();
+		}
+	}
+
+	argv += optind;
+	argc -= optind;
+
+	if (argc == 0) {
 		/* Backwards compatibility */
 		khugepaged_context =  &__khugepaged_context;
 		madvise_context =  &__madvise_context;
@@ -1101,7 +1127,7 @@ static void parse_test_type(int argc, const char **argv)
 		return;
 	}
 
-	buf = strdup(argv[1]);
+	buf = strdup(argv[0]);
 	token = strsep(&buf, ":");
 
 	if (!strcmp(token, "all")) {
@@ -1135,11 +1161,13 @@ static void parse_test_type(int argc, const char **argv)
 	if (!file_ops)
 		return;
 
-	if (argc != 3)
+	if (argc != 2)
 		usage();
+
+	get_finfo(argv[1]);
 }
 
-int main(int argc, const char **argv)
+int main(int argc, char **argv)
 {
 	int hpage_pmd_order;
 	struct thp_settings default_settings = {
@@ -1164,9 +1192,6 @@ int main(int argc, const char **argv)
 
 	parse_test_type(argc, argv);
 
-	if (file_ops)
-		get_finfo(argv[2]);
-
 	setbuf(stdout, NULL);
 
 	page_size = getpagesize();
@@ -1183,6 +1208,7 @@ int main(int argc, const char **argv)
 	default_settings.khugepaged.max_ptes_shared = hpage_pmd_nr / 2;
 	default_settings.khugepaged.pages_to_scan = hpage_pmd_nr * 8;
 	default_settings.hugepages[hpage_pmd_order].enabled = THP_INHERIT;
+	default_settings.hugepages[anon_order].enabled = THP_ALWAYS;
 
 	save_settings();
 	thp_push_settings(&default_settings);
tools/testing/selftests/mm/run_vmtests.sh +2 -0
index c0212258b852..87f513f5cf91 100755
@@ -357,6 +357,8 @@ CATEGORY="cow" run_test ./cow
 
 CATEGORY="thp" run_test ./khugepaged
 
+CATEGORY="thp" run_test ./khugepaged -s 2
+
 CATEGORY="thp" run_test ./transhuge-stress -d 20
 
 CATEGORY="thp" run_test ./split_huge_page_test
Signature · 1 line
2.25.1
author reviewer
#32 lore raw

PATCH v9 9/10 selftests/mm/cow: Generalize do_run_with_thp() helper

In reply to Ryan Roberts

do_run_with_thp() prepares (PMD-sized) THP memory into different states before running tests. With the introduction of multi-size THP, we would like to reuse this logic to also test those smaller THP sizes. So let's add a thpsize parameter which tells the function what size THP it should operate on.

A separate commit will utilize this change to add new tests for multi-size THP, where available.

---
 tools/testing/selftests/mm/cow.c | 121 +++++++++++++++++--------------
 1 file changed, 67 insertions(+), 54 deletions(-)
tools/testing/selftests/mm/cow.c +67 -54
index 7324ce5363c0..4d0b5a125d3c 100644
@@ -32,7 +32,7 @@
 
 static size_t pagesize;
 static int pagemap_fd;
-static size_t thpsize;
+static size_t pmdsize;
 static int nr_hugetlbsizes;
 static size_t hugetlbsizes[10];
 static int gup_fd;
@@ -734,7 +734,7 @@ enum thp_run {
 	THP_RUN_PARTIAL_SHARED,
 };
 
-static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
+static void do_run_with_thp(test_fn fn, enum thp_run thp_run, size_t thpsize)
 {
 	char *mem, *mmap_mem, *tmp, *mremap_mem = MAP_FAILED;
 	size_t size, mmap_size, mremap_size;
@@ -759,11 +759,11 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 	}
 
 	/*
-	 * Try to populate a THP. Touch the first sub-page and test if we get
-	 * another sub-page populated automatically.
+	 * Try to populate a THP. Touch the first sub-page and test if
+	 * we get the last sub-page populated automatically.
 	 */
 	mem[0] = 0;
-	if (!pagemap_is_populated(pagemap_fd, mem + pagesize)) {
+	if (!pagemap_is_populated(pagemap_fd, mem + thpsize - pagesize)) {
 		ksft_test_result_skip("Did not get a THP populated\n");
 		goto munmap;
 	}
@@ -773,12 +773,14 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 	switch (thp_run) {
 	case THP_RUN_PMD:
 	case THP_RUN_PMD_SWAPOUT:
+		assert(thpsize == pmdsize);
 		break;
 	case THP_RUN_PTE:
 	case THP_RUN_PTE_SWAPOUT:
 		/*
 		 * Trigger PTE-mapping the THP by temporarily mapping a single
-		 * subpage R/O.
+		 * subpage R/O. This is a noop if the THP is not pmdsize (and
+		 * therefore already PTE-mapped).
 		 */
 		ret = mprotect(mem + pagesize, pagesize, PROT_READ);
 		if (ret) {
@@ -875,52 +877,60 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 		munmap(mremap_mem, mremap_size);
 }
 
-static void run_with_thp(test_fn fn, const char *desc)
+static void run_with_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PMD);
+	ksft_print_msg("[RUN] %s ... with THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PMD, size);
 }
 
-static void run_with_thp_swap(test_fn fn, const char *desc)
+static void run_with_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with swapped-out THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PMD_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with swapped-out THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PMD_SWAPOUT, size);
 }
 
-static void run_with_pte_mapped_thp(test_fn fn, const char *desc)
+static void run_with_pte_mapped_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with PTE-mapped THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PTE);
+	ksft_print_msg("[RUN] %s ... with PTE-mapped THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PTE, size);
 }
 
-static void run_with_pte_mapped_thp_swap(test_fn fn, const char *desc)
+static void run_with_pte_mapped_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with swapped-out, PTE-mapped THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PTE_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with swapped-out, PTE-mapped THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PTE_SWAPOUT, size);
 }
 
-static void run_with_single_pte_of_thp(test_fn fn, const char *desc)
+static void run_with_single_pte_of_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with single PTE of THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_SINGLE_PTE);
+	ksft_print_msg("[RUN] %s ... with single PTE of THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_SINGLE_PTE, size);
 }
 
-static void run_with_single_pte_of_thp_swap(test_fn fn, const char *desc)
+static void run_with_single_pte_of_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with single PTE of swapped-out THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_SINGLE_PTE_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with single PTE of swapped-out THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_SINGLE_PTE_SWAPOUT, size);
 }
 
-static void run_with_partial_mremap_thp(test_fn fn, const char *desc)
+static void run_with_partial_mremap_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with partially mremap()'ed THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PARTIAL_MREMAP);
+	ksft_print_msg("[RUN] %s ... with partially mremap()'ed THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PARTIAL_MREMAP, size);
 }
 
-static void run_with_partial_shared_thp(test_fn fn, const char *desc)
+static void run_with_partial_shared_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with partially shared THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PARTIAL_SHARED);
+	ksft_print_msg("[RUN] %s ... with partially shared THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PARTIAL_SHARED, size);
 }
 
 static void run_with_hugetlb(test_fn fn, const char *desc, size_t hugetlbsize)
@@ -1091,15 +1101,15 @@ static void run_anon_test_case(struct test_case const *test_case)
 
 	run_with_base_page(test_case->fn, test_case->desc);
 	run_with_base_page_swap(test_case->fn, test_case->desc);
-	if (thpsize) {
-		run_with_thp(test_case->fn, test_case->desc);
-		run_with_thp_swap(test_case->fn, test_case->desc);
-		run_with_pte_mapped_thp(test_case->fn, test_case->desc);
-		run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc);
-		run_with_single_pte_of_thp(test_case->fn, test_case->desc);
-		run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc);
-		run_with_partial_mremap_thp(test_case->fn, test_case->desc);
-		run_with_partial_shared_thp(test_case->fn, test_case->desc);
+	if (pmdsize) {
+		run_with_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_pte_mapped_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_single_pte_of_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_partial_mremap_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_partial_shared_thp(test_case->fn, test_case->desc, pmdsize);
 	}
 	for (i = 0; i < nr_hugetlbsizes; i++)
 		run_with_hugetlb(test_case->fn, test_case->desc,
@@ -1120,7 +1130,7 @@ static int tests_per_anon_test_case(void)
 {
 	int tests = 2 + nr_hugetlbsizes;
 
-	if (thpsize)
+	if (pmdsize)
 		tests += 8;
 	return tests;
 }
@@ -1329,7 +1339,7 @@ static void run_anon_thp_test_cases(void)
 {
 	int i;
 
-	if (!thpsize)
+	if (!pmdsize)
 		return;
 
 	ksft_print_msg("[INFO] Anonymous THP tests\n");
@@ -1338,13 +1348,13 @@ static void run_anon_thp_test_cases(void)
 		struct test_case const *test_case = &anon_thp_test_cases[i];
 
 		ksft_print_msg("[RUN] %s\n", test_case->desc);
-		do_run_with_thp(test_case->fn, THP_RUN_PMD);
+		do_run_with_thp(test_case->fn, THP_RUN_PMD, pmdsize);
 	}
 }
 
 static int tests_per_anon_thp_test_case(void)
 {
-	return thpsize ? 1 : 0;
+	return pmdsize ? 1 : 0;
 }
 
 typedef void (*non_anon_test_fn)(char *mem, const char *smem, size_t size);
@@ -1419,7 +1429,7 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 	}
 
 	/* For alignment purposes, we need twice the thp size. */
-	mmap_size = 2 * thpsize;
+	mmap_size = 2 * pmdsize;
 	mmap_mem = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE,
 			MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
 	if (mmap_mem == MAP_FAILED) {
@@ -1434,11 +1444,11 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 	}
 
 	/* We need a THP-aligned memory area. */
-	mem = (char *)(((uintptr_t)mmap_mem + thpsize) & ~(thpsize - 1));
-	smem = (char *)(((uintptr_t)mmap_smem + thpsize) & ~(thpsize - 1));
+	mem = (char *)(((uintptr_t)mmap_mem + pmdsize) & ~(pmdsize - 1));
+	smem = (char *)(((uintptr_t)mmap_smem + pmdsize) & ~(pmdsize - 1));
 
-	ret = madvise(mem, thpsize, MADV_HUGEPAGE);
-	ret |= madvise(smem, thpsize, MADV_HUGEPAGE);
+	ret = madvise(mem, pmdsize, MADV_HUGEPAGE);
+	ret |= madvise(smem, pmdsize, MADV_HUGEPAGE);
 	if (ret) {
 		ksft_test_result_fail("MADV_HUGEPAGE failed\n");
 		goto munmap;
@@ -1457,7 +1467,7 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 		goto munmap;
 	}
 
-	fn(mem, smem, thpsize);
+	fn(mem, smem, pmdsize);
 munmap:
 	munmap(mmap_mem, mmap_size);
 	if (mmap_smem != MAP_FAILED)
@@ -1650,7 +1660,7 @@ static void run_non_anon_test_case(struct non_anon_test_case const *test_case)
 	run_with_zeropage(test_case->fn, test_case->desc);
 	run_with_memfd(test_case->fn, test_case->desc);
 	run_with_tmpfile(test_case->fn, test_case->desc);
-	if (thpsize)
+	if (pmdsize)
 		run_with_huge_zeropage(test_case->fn, test_case->desc);
 	for (i = 0; i < nr_hugetlbsizes; i++)
 		run_with_memfd_hugetlb(test_case->fn, test_case->desc,
@@ -1671,7 +1681,7 @@ static int tests_per_non_anon_test_case(void)
 {
 	int tests = 3 + nr_hugetlbsizes;
 
-	if (thpsize)
+	if (pmdsize)
 		tests += 1;
 	return tests;
 }
@@ -1681,10 +1691,13 @@ int main(int argc, char **argv)
 	int err;
 
 	pagesize = getpagesize();
-	thpsize = read_pmd_pagesize();
-	if (thpsize)
+	pmdsize = read_pmd_pagesize();
+	if (pmdsize) {
+		ksft_print_msg("[INFO] detected PMD size: %zu KiB\n",
+			       pmdsize / 1024);
 		ksft_print_msg("[INFO] detected THP size: %zu KiB\n",
-			       thpsize / 1024);
+			       pmdsize / 1024);
+	}
 	nr_hugetlbsizes = detect_hugetlb_page_sizes(hugetlbsizes,
 						    ARRAY_SIZE(hugetlbsizes));
 	detect_huge_zeropage();
Signature · 1 line
2.25.1
Tested-by on v9 9/10
#33 lore raw

Re: [PATCH v9 09/10] selftests/mm/cow: Generalize do_run_with_thp() helper

Quoted 6 lines · Ryan Roberts

do_run_with_thp() prepares (PMD-sized) THP memory into different states before running tests. With the introduction of multi-size THP, we would like to reuse this logic to also test those smaller THP sizes. So let's add a thpsize parameter which tells the function what size THP it should operate on.

Quoted 7 lines

A separate commit will utilize this change to add new tests for multi-size THP, where available.

I am replying to all this time; Ryan, do you think it's okay to run 700 of selftests/mm/cow tests? Even on FVP, they did not take longer though.

Quoted 273 lines
---
 tools/testing/selftests/mm/cow.c | 121 +++++++++++++++++--------------
 1 file changed, 67 insertions(+), 54 deletions(-)
tools/testing/selftests/mm/cow.c +67 -54
index 7324ce5363c0..4d0b5a125d3c 100644
@@ -32,7 +32,7 @@
 
 static size_t pagesize;
 static int pagemap_fd;
-static size_t thpsize;
+static size_t pmdsize;
 static int nr_hugetlbsizes;
 static size_t hugetlbsizes[10];
 static int gup_fd;
@@ -734,7 +734,7 @@ enum thp_run {
 	THP_RUN_PARTIAL_SHARED,
 };
 
-static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
+static void do_run_with_thp(test_fn fn, enum thp_run thp_run, size_t thpsize)
 {
 	char *mem, *mmap_mem, *tmp, *mremap_mem = MAP_FAILED;
 	size_t size, mmap_size, mremap_size;
@@ -759,11 +759,11 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 	}
 
 	/*
-	 * Try to populate a THP. Touch the first sub-page and test if we get
-	 * another sub-page populated automatically.
+	 * Try to populate a THP. Touch the first sub-page and test if
+	 * we get the last sub-page populated automatically.
 	 */
 	mem[0] = 0;
-	if (!pagemap_is_populated(pagemap_fd, mem + pagesize)) {
+	if (!pagemap_is_populated(pagemap_fd, mem + thpsize - pagesize)) {
 		ksft_test_result_skip("Did not get a THP populated\n");
 		goto munmap;
 	}
@@ -773,12 +773,14 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 	switch (thp_run) {
 	case THP_RUN_PMD:
 	case THP_RUN_PMD_SWAPOUT:
+		assert(thpsize == pmdsize);
 		break;
 	case THP_RUN_PTE:
 	case THP_RUN_PTE_SWAPOUT:
 		/*
 		 * Trigger PTE-mapping the THP by temporarily mapping a single
-		 * subpage R/O.
+		 * subpage R/O. This is a noop if the THP is not pmdsize (and
+		 * therefore already PTE-mapped).
 		 */
 		ret = mprotect(mem + pagesize, pagesize, PROT_READ);
 		if (ret) {
@@ -875,52 +877,60 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 		munmap(mremap_mem, mremap_size);
 }
 
-static void run_with_thp(test_fn fn, const char *desc)
+static void run_with_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PMD);
+	ksft_print_msg("[RUN] %s ... with THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PMD, size);
 }
 
-static void run_with_thp_swap(test_fn fn, const char *desc)
+static void run_with_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with swapped-out THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PMD_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with swapped-out THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PMD_SWAPOUT, size);
 }
 
-static void run_with_pte_mapped_thp(test_fn fn, const char *desc)
+static void run_with_pte_mapped_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with PTE-mapped THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PTE);
+	ksft_print_msg("[RUN] %s ... with PTE-mapped THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PTE, size);
 }
 
-static void run_with_pte_mapped_thp_swap(test_fn fn, const char *desc)
+static void run_with_pte_mapped_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with swapped-out, PTE-mapped THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PTE_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with swapped-out, PTE-mapped THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PTE_SWAPOUT, size);
 }
 
-static void run_with_single_pte_of_thp(test_fn fn, const char *desc)
+static void run_with_single_pte_of_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with single PTE of THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_SINGLE_PTE);
+	ksft_print_msg("[RUN] %s ... with single PTE of THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_SINGLE_PTE, size);
 }
 
-static void run_with_single_pte_of_thp_swap(test_fn fn, const char *desc)
+static void run_with_single_pte_of_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with single PTE of swapped-out THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_SINGLE_PTE_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with single PTE of swapped-out THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_SINGLE_PTE_SWAPOUT, size);
 }
 
-static void run_with_partial_mremap_thp(test_fn fn, const char *desc)
+static void run_with_partial_mremap_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with partially mremap()'ed THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PARTIAL_MREMAP);
+	ksft_print_msg("[RUN] %s ... with partially mremap()'ed THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PARTIAL_MREMAP, size);
 }
 
-static void run_with_partial_shared_thp(test_fn fn, const char *desc)
+static void run_with_partial_shared_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with partially shared THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PARTIAL_SHARED);
+	ksft_print_msg("[RUN] %s ... with partially shared THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PARTIAL_SHARED, size);
 }
 
 static void run_with_hugetlb(test_fn fn, const char *desc, size_t hugetlbsize)
@@ -1091,15 +1101,15 @@ static void run_anon_test_case(struct test_case const *test_case)
 
 	run_with_base_page(test_case->fn, test_case->desc);
 	run_with_base_page_swap(test_case->fn, test_case->desc);
-	if (thpsize) {
-		run_with_thp(test_case->fn, test_case->desc);
-		run_with_thp_swap(test_case->fn, test_case->desc);
-		run_with_pte_mapped_thp(test_case->fn, test_case->desc);
-		run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc);
-		run_with_single_pte_of_thp(test_case->fn, test_case->desc);
-		run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc);
-		run_with_partial_mremap_thp(test_case->fn, test_case->desc);
-		run_with_partial_shared_thp(test_case->fn, test_case->desc);
+	if (pmdsize) {
+		run_with_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_pte_mapped_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_single_pte_of_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_partial_mremap_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_partial_shared_thp(test_case->fn, test_case->desc, pmdsize);
 	}
 	for (i = 0; i < nr_hugetlbsizes; i++)
 		run_with_hugetlb(test_case->fn, test_case->desc,
@@ -1120,7 +1130,7 @@ static int tests_per_anon_test_case(void)
 {
 	int tests = 2 + nr_hugetlbsizes;
 
-	if (thpsize)
+	if (pmdsize)
 		tests += 8;
 	return tests;
 }
@@ -1329,7 +1339,7 @@ static void run_anon_thp_test_cases(void)
 {
 	int i;
 
-	if (!thpsize)
+	if (!pmdsize)
 		return;
 
 	ksft_print_msg("[INFO] Anonymous THP tests\n");
@@ -1338,13 +1348,13 @@ static void run_anon_thp_test_cases(void)
 		struct test_case const *test_case = &anon_thp_test_cases[i];
 
 		ksft_print_msg("[RUN] %s\n", test_case->desc);
-		do_run_with_thp(test_case->fn, THP_RUN_PMD);
+		do_run_with_thp(test_case->fn, THP_RUN_PMD, pmdsize);
 	}
 }
 
 static int tests_per_anon_thp_test_case(void)
 {
-	return thpsize ? 1 : 0;
+	return pmdsize ? 1 : 0;
 }
 
 typedef void (*non_anon_test_fn)(char *mem, const char *smem, size_t size);
@@ -1419,7 +1429,7 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 	}
 
 	/* For alignment purposes, we need twice the thp size. */
-	mmap_size = 2 * thpsize;
+	mmap_size = 2 * pmdsize;
 	mmap_mem = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE,
 			MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
 	if (mmap_mem == MAP_FAILED) {
@@ -1434,11 +1444,11 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 	}
 
 	/* We need a THP-aligned memory area. */
-	mem = (char *)(((uintptr_t)mmap_mem + thpsize) & ~(thpsize - 1));
-	smem = (char *)(((uintptr_t)mmap_smem + thpsize) & ~(thpsize - 1));
+	mem = (char *)(((uintptr_t)mmap_mem + pmdsize) & ~(pmdsize - 1));
+	smem = (char *)(((uintptr_t)mmap_smem + pmdsize) & ~(pmdsize - 1));
 
-	ret = madvise(mem, thpsize, MADV_HUGEPAGE);
-	ret |= madvise(smem, thpsize, MADV_HUGEPAGE);
+	ret = madvise(mem, pmdsize, MADV_HUGEPAGE);
+	ret |= madvise(smem, pmdsize, MADV_HUGEPAGE);
 	if (ret) {
 		ksft_test_result_fail("MADV_HUGEPAGE failed\n");
 		goto munmap;
@@ -1457,7 +1467,7 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 		goto munmap;
 	}
 
-	fn(mem, smem, thpsize);
+	fn(mem, smem, pmdsize);
 munmap:
 	munmap(mmap_mem, mmap_size);
 	if (mmap_smem != MAP_FAILED)
@@ -1650,7 +1660,7 @@ static void run_non_anon_test_case(struct non_anon_test_case const *test_case)
 	run_with_zeropage(test_case->fn, test_case->desc);
 	run_with_memfd(test_case->fn, test_case->desc);
 	run_with_tmpfile(test_case->fn, test_case->desc);
-	if (thpsize)
+	if (pmdsize)
 		run_with_huge_zeropage(test_case->fn, test_case->desc);
 	for (i = 0; i < nr_hugetlbsizes; i++)
 		run_with_memfd_hugetlb(test_case->fn, test_case->desc,
@@ -1671,7 +1681,7 @@ static int tests_per_non_anon_test_case(void)
 {
 	int tests = 3 + nr_hugetlbsizes;
 
-	if (thpsize)
+	if (pmdsize)
 		tests += 1;
 	return tests;
 }
@@ -1681,10 +1691,13 @@ int main(int argc, char **argv)
 	int err;
 
 	pagesize = getpagesize();
-	thpsize = read_pmd_pagesize();
-	if (thpsize)
+	pmdsize = read_pmd_pagesize();
+	if (pmdsize) {
+		ksft_print_msg("[INFO] detected PMD size: %zu KiB\n",
+			       pmdsize / 1024);
 		ksft_print_msg("[INFO] detected THP size: %zu KiB\n",
-			       thpsize / 1024);
+			       pmdsize / 1024);
+	}
 	nr_hugetlbsizes = detect_hugetlb_page_sizes(hugetlbsizes,
 						    ARRAY_SIZE(hugetlbsizes));
 	detect_huge_zeropage();
Signature · 1 line
2.25.1
author reviewer
#34 lore raw

Re: [PATCH v9 09/10] selftests/mm/cow: Generalize do_run_with_thp() helper

Quoted 10 lines · Itaru Kitayama
Quoted 9 lines · Ryan Roberts

do_run_with_thp() prepares (PMD-sized) THP memory into different states before running tests. With the introduction of multi-size THP, we would like to reuse this logic to also test those smaller THP sizes. So let's add a thpsize parameter which tells the function what size THP it should operate on.

A separate commit will utilize this change to add new tests for multi-size THP, where available.

Quoted 6 lines
Quoted 4 lines

Thanks for testing!

Quoted 4 lines

I am replying to all this time; Ryan, do you think it's okay to run 700 of selftests/mm/cow tests? Even on FVP, they did not take longer though.

What exactly is your concern, the amount of time it takes to run the tests? I've found (at least on real HW) that the time it takes to run a test is dominated by accessing the folio's memory. So adding all of the new tests that test sizes between order-2 and PMD_ORDER-1 is ~equivalent to running the existing PMD_ORDER tests twice. And the runtime of those is barely noticable compared to the PUD_ORDER HugeTLB tests. So I don't think we are impacting runtime by much. Sounds like your experience says that's also true for FVP?

Quoted 274 lines
Quoted 274 lines
---
 tools/testing/selftests/mm/cow.c | 121 +++++++++++++++++--------------
 1 file changed, 67 insertions(+), 54 deletions(-)
tools/testing/selftests/mm/cow.c +67 -54
index 7324ce5363c0..4d0b5a125d3c 100644
@@ -32,7 +32,7 @@
 
 static size_t pagesize;
 static int pagemap_fd;
-static size_t thpsize;
+static size_t pmdsize;
 static int nr_hugetlbsizes;
 static size_t hugetlbsizes[10];
 static int gup_fd;
@@ -734,7 +734,7 @@ enum thp_run {
 	THP_RUN_PARTIAL_SHARED,
 };
 
-static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
+static void do_run_with_thp(test_fn fn, enum thp_run thp_run, size_t thpsize)
 {
 	char *mem, *mmap_mem, *tmp, *mremap_mem = MAP_FAILED;
 	size_t size, mmap_size, mremap_size;
@@ -759,11 +759,11 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 	}
 
 	/*
-	 * Try to populate a THP. Touch the first sub-page and test if we get
-	 * another sub-page populated automatically.
+	 * Try to populate a THP. Touch the first sub-page and test if
+	 * we get the last sub-page populated automatically.
 	 */
 	mem[0] = 0;
-	if (!pagemap_is_populated(pagemap_fd, mem + pagesize)) {
+	if (!pagemap_is_populated(pagemap_fd, mem + thpsize - pagesize)) {
 		ksft_test_result_skip("Did not get a THP populated\n");
 		goto munmap;
 	}
@@ -773,12 +773,14 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 	switch (thp_run) {
 	case THP_RUN_PMD:
 	case THP_RUN_PMD_SWAPOUT:
+		assert(thpsize == pmdsize);
 		break;
 	case THP_RUN_PTE:
 	case THP_RUN_PTE_SWAPOUT:
 		/*
 		 * Trigger PTE-mapping the THP by temporarily mapping a single
-		 * subpage R/O.
+		 * subpage R/O. This is a noop if the THP is not pmdsize (and
+		 * therefore already PTE-mapped).
 		 */
 		ret = mprotect(mem + pagesize, pagesize, PROT_READ);
 		if (ret) {
@@ -875,52 +877,60 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 		munmap(mremap_mem, mremap_size);
 }
 
-static void run_with_thp(test_fn fn, const char *desc)
+static void run_with_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PMD);
+	ksft_print_msg("[RUN] %s ... with THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PMD, size);
 }
 
-static void run_with_thp_swap(test_fn fn, const char *desc)
+static void run_with_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with swapped-out THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PMD_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with swapped-out THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PMD_SWAPOUT, size);
 }
 
-static void run_with_pte_mapped_thp(test_fn fn, const char *desc)
+static void run_with_pte_mapped_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with PTE-mapped THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PTE);
+	ksft_print_msg("[RUN] %s ... with PTE-mapped THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PTE, size);
 }
 
-static void run_with_pte_mapped_thp_swap(test_fn fn, const char *desc)
+static void run_with_pte_mapped_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with swapped-out, PTE-mapped THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PTE_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with swapped-out, PTE-mapped THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PTE_SWAPOUT, size);
 }
 
-static void run_with_single_pte_of_thp(test_fn fn, const char *desc)
+static void run_with_single_pte_of_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with single PTE of THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_SINGLE_PTE);
+	ksft_print_msg("[RUN] %s ... with single PTE of THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_SINGLE_PTE, size);
 }
 
-static void run_with_single_pte_of_thp_swap(test_fn fn, const char *desc)
+static void run_with_single_pte_of_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with single PTE of swapped-out THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_SINGLE_PTE_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with single PTE of swapped-out THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_SINGLE_PTE_SWAPOUT, size);
 }
 
-static void run_with_partial_mremap_thp(test_fn fn, const char *desc)
+static void run_with_partial_mremap_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with partially mremap()'ed THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PARTIAL_MREMAP);
+	ksft_print_msg("[RUN] %s ... with partially mremap()'ed THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PARTIAL_MREMAP, size);
 }
 
-static void run_with_partial_shared_thp(test_fn fn, const char *desc)
+static void run_with_partial_shared_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with partially shared THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PARTIAL_SHARED);
+	ksft_print_msg("[RUN] %s ... with partially shared THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PARTIAL_SHARED, size);
 }
 
 static void run_with_hugetlb(test_fn fn, const char *desc, size_t hugetlbsize)
@@ -1091,15 +1101,15 @@ static void run_anon_test_case(struct test_case const *test_case)
 
 	run_with_base_page(test_case->fn, test_case->desc);
 	run_with_base_page_swap(test_case->fn, test_case->desc);
-	if (thpsize) {
-		run_with_thp(test_case->fn, test_case->desc);
-		run_with_thp_swap(test_case->fn, test_case->desc);
-		run_with_pte_mapped_thp(test_case->fn, test_case->desc);
-		run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc);
-		run_with_single_pte_of_thp(test_case->fn, test_case->desc);
-		run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc);
-		run_with_partial_mremap_thp(test_case->fn, test_case->desc);
-		run_with_partial_shared_thp(test_case->fn, test_case->desc);
+	if (pmdsize) {
+		run_with_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_pte_mapped_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_single_pte_of_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_partial_mremap_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_partial_shared_thp(test_case->fn, test_case->desc, pmdsize);
 	}
 	for (i = 0; i < nr_hugetlbsizes; i++)
 		run_with_hugetlb(test_case->fn, test_case->desc,
@@ -1120,7 +1130,7 @@ static int tests_per_anon_test_case(void)
 {
 	int tests = 2 + nr_hugetlbsizes;
 
-	if (thpsize)
+	if (pmdsize)
 		tests += 8;
 	return tests;
 }
@@ -1329,7 +1339,7 @@ static void run_anon_thp_test_cases(void)
 {
 	int i;
 
-	if (!thpsize)
+	if (!pmdsize)
 		return;
 
 	ksft_print_msg("[INFO] Anonymous THP tests\n");
@@ -1338,13 +1348,13 @@ static void run_anon_thp_test_cases(void)
 		struct test_case const *test_case = &anon_thp_test_cases[i];
 
 		ksft_print_msg("[RUN] %s\n", test_case->desc);
-		do_run_with_thp(test_case->fn, THP_RUN_PMD);
+		do_run_with_thp(test_case->fn, THP_RUN_PMD, pmdsize);
 	}
 }
 
 static int tests_per_anon_thp_test_case(void)
 {
-	return thpsize ? 1 : 0;
+	return pmdsize ? 1 : 0;
 }
 
 typedef void (*non_anon_test_fn)(char *mem, const char *smem, size_t size);
@@ -1419,7 +1429,7 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 	}
 
 	/* For alignment purposes, we need twice the thp size. */
-	mmap_size = 2 * thpsize;
+	mmap_size = 2 * pmdsize;
 	mmap_mem = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE,
 			MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
 	if (mmap_mem == MAP_FAILED) {
@@ -1434,11 +1444,11 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 	}
 
 	/* We need a THP-aligned memory area. */
-	mem = (char *)(((uintptr_t)mmap_mem + thpsize) & ~(thpsize - 1));
-	smem = (char *)(((uintptr_t)mmap_smem + thpsize) & ~(thpsize - 1));
+	mem = (char *)(((uintptr_t)mmap_mem + pmdsize) & ~(pmdsize - 1));
+	smem = (char *)(((uintptr_t)mmap_smem + pmdsize) & ~(pmdsize - 1));
 
-	ret = madvise(mem, thpsize, MADV_HUGEPAGE);
-	ret |= madvise(smem, thpsize, MADV_HUGEPAGE);
+	ret = madvise(mem, pmdsize, MADV_HUGEPAGE);
+	ret |= madvise(smem, pmdsize, MADV_HUGEPAGE);
 	if (ret) {
 		ksft_test_result_fail("MADV_HUGEPAGE failed\n");
 		goto munmap;
@@ -1457,7 +1467,7 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 		goto munmap;
 	}
 
-	fn(mem, smem, thpsize);
+	fn(mem, smem, pmdsize);
 munmap:
 	munmap(mmap_mem, mmap_size);
 	if (mmap_smem != MAP_FAILED)
@@ -1650,7 +1660,7 @@ static void run_non_anon_test_case(struct non_anon_test_case const *test_case)
 	run_with_zeropage(test_case->fn, test_case->desc);
 	run_with_memfd(test_case->fn, test_case->desc);
 	run_with_tmpfile(test_case->fn, test_case->desc);
-	if (thpsize)
+	if (pmdsize)
 		run_with_huge_zeropage(test_case->fn, test_case->desc);
 	for (i = 0; i < nr_hugetlbsizes; i++)
 		run_with_memfd_hugetlb(test_case->fn, test_case->desc,
@@ -1671,7 +1681,7 @@ static int tests_per_non_anon_test_case(void)
 {
 	int tests = 3 + nr_hugetlbsizes;
 
-	if (thpsize)
+	if (pmdsize)
 		tests += 1;
 	return tests;
 }
@@ -1681,10 +1691,13 @@ int main(int argc, char **argv)
 	int err;
 
 	pagesize = getpagesize();
-	thpsize = read_pmd_pagesize();
-	if (thpsize)
+	pmdsize = read_pmd_pagesize();
+	if (pmdsize) {
+		ksft_print_msg("[INFO] detected PMD size: %zu KiB\n",
+			       pmdsize / 1024);
 		ksft_print_msg("[INFO] detected THP size: %zu KiB\n",
-			       thpsize / 1024);
+			       pmdsize / 1024);
+	}
 	nr_hugetlbsizes = detect_hugetlb_page_sizes(hugetlbsizes,
 						    ARRAY_SIZE(hugetlbsizes));
 	detect_huge_zeropage();
Signature · 1 line
2.25.1
#35 lore raw

Re: [PATCH v9 09/10] selftests/mm/cow: Generalize do_run_with_thp() helper

Quoted 25 lines · Ryan Roberts
Quoted 10 lines · Itaru Kitayama
Quoted 9 lines · Ryan Roberts

do_run_with_thp() prepares (PMD-sized) THP memory into different states before running tests. With the introduction of multi-size THP, we would like to reuse this logic to also test those smaller THP sizes. So let's add a thpsize parameter which tells the function what size THP it should operate on.

A separate commit will utilize this change to add new tests for multi-size THP, where available.

Quoted 6 lines
Quoted 4 lines

Thanks for testing!

Quoted 4 lines

I am replying to all this time; Ryan, do you think it's okay to run 700 of selftests/mm/cow tests? Even on FVP, they did not take longer though.

Quoted 7 lines

What exactly is your concern, the amount of time it takes to run the tests? I've found (at least on real HW) that the time it takes to run a test is dominated by accessing the folio's memory. So adding all of the new tests that test sizes between order-2 and PMD_ORDER-1 is ~equivalent to running the existing PMD_ORDER tests twice. And the runtime of those is barely noticable compared to the PUD_ORDER HugeTLB tests. So I don't think we are impacting runtime by much. Sounds like your experience says that's also true for FVP?

My primary concern was the time amount of time, but going back from
mm-unstable/mm-stable, which contains your multi THP changes to Linus' master, I see what you were saying - the total number of tests for the "cow" program is the same. And I am convinced that the total time won't be changed much.

On FVP EVP RevC, if those kselftests tests are not focusing on stress testing, tests are processed reasonably "fast".

Lastly, as you tried to come up a series, the way mm kselftests executed is not so intuitive to me, as I am starting tests from the run_kselftest.sh script, I don't think mm tests are run using the -t (specific test) or -c (entire mm collection) options.

Thanks,
Itaru.

Quoted 276 lines
Quoted 275 lines
Quoted 275 lines

---
 tools/testing/selftests/mm/cow.c | 121 +++++++++++++++++--------------
 1 file changed, 67 insertions(+), 54 deletions(-)
tools/testing/selftests/mm/cow.c +67 -54
index 7324ce5363c0..4d0b5a125d3c 100644
@@ -32,7 +32,7 @@
 
 static size_t pagesize;
 static int pagemap_fd;
-static size_t thpsize;
+static size_t pmdsize;
 static int nr_hugetlbsizes;
 static size_t hugetlbsizes[10];
 static int gup_fd;
@@ -734,7 +734,7 @@ enum thp_run {
 	THP_RUN_PARTIAL_SHARED,
 };
 
-static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
+static void do_run_with_thp(test_fn fn, enum thp_run thp_run, size_t thpsize)
 {
 	char *mem, *mmap_mem, *tmp, *mremap_mem = MAP_FAILED;
 	size_t size, mmap_size, mremap_size;
@@ -759,11 +759,11 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 	}
 
 	/*
-	 * Try to populate a THP. Touch the first sub-page and test if we get
-	 * another sub-page populated automatically.
+	 * Try to populate a THP. Touch the first sub-page and test if
+	 * we get the last sub-page populated automatically.
 	 */
 	mem[0] = 0;
-	if (!pagemap_is_populated(pagemap_fd, mem + pagesize)) {
+	if (!pagemap_is_populated(pagemap_fd, mem + thpsize - pagesize)) {
 		ksft_test_result_skip("Did not get a THP populated\n");
 		goto munmap;
 	}
@@ -773,12 +773,14 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 	switch (thp_run) {
 	case THP_RUN_PMD:
 	case THP_RUN_PMD_SWAPOUT:
+		assert(thpsize == pmdsize);
 		break;
 	case THP_RUN_PTE:
 	case THP_RUN_PTE_SWAPOUT:
 		/*
 		 * Trigger PTE-mapping the THP by temporarily mapping a single
-		 * subpage R/O.
+		 * subpage R/O. This is a noop if the THP is not pmdsize (and
+		 * therefore already PTE-mapped).
 		 */
 		ret = mprotect(mem + pagesize, pagesize, PROT_READ);
 		if (ret) {
@@ -875,52 +877,60 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 		munmap(mremap_mem, mremap_size);
 }
 
-static void run_with_thp(test_fn fn, const char *desc)
+static void run_with_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PMD);
+	ksft_print_msg("[RUN] %s ... with THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PMD, size);
 }
 
-static void run_with_thp_swap(test_fn fn, const char *desc)
+static void run_with_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with swapped-out THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PMD_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with swapped-out THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PMD_SWAPOUT, size);
 }
 
-static void run_with_pte_mapped_thp(test_fn fn, const char *desc)
+static void run_with_pte_mapped_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with PTE-mapped THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PTE);
+	ksft_print_msg("[RUN] %s ... with PTE-mapped THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PTE, size);
 }
 
-static void run_with_pte_mapped_thp_swap(test_fn fn, const char *desc)
+static void run_with_pte_mapped_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with swapped-out, PTE-mapped THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PTE_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with swapped-out, PTE-mapped THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PTE_SWAPOUT, size);
 }
 
-static void run_with_single_pte_of_thp(test_fn fn, const char *desc)
+static void run_with_single_pte_of_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with single PTE of THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_SINGLE_PTE);
+	ksft_print_msg("[RUN] %s ... with single PTE of THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_SINGLE_PTE, size);
 }
 
-static void run_with_single_pte_of_thp_swap(test_fn fn, const char *desc)
+static void run_with_single_pte_of_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with single PTE of swapped-out THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_SINGLE_PTE_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with single PTE of swapped-out THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_SINGLE_PTE_SWAPOUT, size);
 }
 
-static void run_with_partial_mremap_thp(test_fn fn, const char *desc)
+static void run_with_partial_mremap_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with partially mremap()'ed THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PARTIAL_MREMAP);
+	ksft_print_msg("[RUN] %s ... with partially mremap()'ed THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PARTIAL_MREMAP, size);
 }
 
-static void run_with_partial_shared_thp(test_fn fn, const char *desc)
+static void run_with_partial_shared_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with partially shared THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PARTIAL_SHARED);
+	ksft_print_msg("[RUN] %s ... with partially shared THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PARTIAL_SHARED, size);
 }
 
 static void run_with_hugetlb(test_fn fn, const char *desc, size_t hugetlbsize)
@@ -1091,15 +1101,15 @@ static void run_anon_test_case(struct test_case const *test_case)
 
 	run_with_base_page(test_case->fn, test_case->desc);
 	run_with_base_page_swap(test_case->fn, test_case->desc);
-	if (thpsize) {
-		run_with_thp(test_case->fn, test_case->desc);
-		run_with_thp_swap(test_case->fn, test_case->desc);
-		run_with_pte_mapped_thp(test_case->fn, test_case->desc);
-		run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc);
-		run_with_single_pte_of_thp(test_case->fn, test_case->desc);
-		run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc);
-		run_with_partial_mremap_thp(test_case->fn, test_case->desc);
-		run_with_partial_shared_thp(test_case->fn, test_case->desc);
+	if (pmdsize) {
+		run_with_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_pte_mapped_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_single_pte_of_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_partial_mremap_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_partial_shared_thp(test_case->fn, test_case->desc, pmdsize);
 	}
 	for (i = 0; i < nr_hugetlbsizes; i++)
 		run_with_hugetlb(test_case->fn, test_case->desc,
@@ -1120,7 +1130,7 @@ static int tests_per_anon_test_case(void)
 {
 	int tests = 2 + nr_hugetlbsizes;
 
-	if (thpsize)
+	if (pmdsize)
 		tests += 8;
 	return tests;
 }
@@ -1329,7 +1339,7 @@ static void run_anon_thp_test_cases(void)
 {
 	int i;
 
-	if (!thpsize)
+	if (!pmdsize)
 		return;
 
 	ksft_print_msg("[INFO] Anonymous THP tests\n");
@@ -1338,13 +1348,13 @@ static void run_anon_thp_test_cases(void)
 		struct test_case const *test_case = &anon_thp_test_cases[i];
 
 		ksft_print_msg("[RUN] %s\n", test_case->desc);
-		do_run_with_thp(test_case->fn, THP_RUN_PMD);
+		do_run_with_thp(test_case->fn, THP_RUN_PMD, pmdsize);
 	}
 }
 
 static int tests_per_anon_thp_test_case(void)
 {
-	return thpsize ? 1 : 0;
+	return pmdsize ? 1 : 0;
 }
 
 typedef void (*non_anon_test_fn)(char *mem, const char *smem, size_t size);
@@ -1419,7 +1429,7 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 	}
 
 	/* For alignment purposes, we need twice the thp size. */
-	mmap_size = 2 * thpsize;
+	mmap_size = 2 * pmdsize;
 	mmap_mem = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE,
 			MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
 	if (mmap_mem == MAP_FAILED) {
@@ -1434,11 +1444,11 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 	}
 
 	/* We need a THP-aligned memory area. */
-	mem = (char *)(((uintptr_t)mmap_mem + thpsize) & ~(thpsize - 1));
-	smem = (char *)(((uintptr_t)mmap_smem + thpsize) & ~(thpsize - 1));
+	mem = (char *)(((uintptr_t)mmap_mem + pmdsize) & ~(pmdsize - 1));
+	smem = (char *)(((uintptr_t)mmap_smem + pmdsize) & ~(pmdsize - 1));
 
-	ret = madvise(mem, thpsize, MADV_HUGEPAGE);
-	ret |= madvise(smem, thpsize, MADV_HUGEPAGE);
+	ret = madvise(mem, pmdsize, MADV_HUGEPAGE);
+	ret |= madvise(smem, pmdsize, MADV_HUGEPAGE);
 	if (ret) {
 		ksft_test_result_fail("MADV_HUGEPAGE failed\n");
 		goto munmap;
@@ -1457,7 +1467,7 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 		goto munmap;
 	}
 
-	fn(mem, smem, thpsize);
+	fn(mem, smem, pmdsize);
 munmap:
 	munmap(mmap_mem, mmap_size);
 	if (mmap_smem != MAP_FAILED)
@@ -1650,7 +1660,7 @@ static void run_non_anon_test_case(struct non_anon_test_case const *test_case)
 	run_with_zeropage(test_case->fn, test_case->desc);
 	run_with_memfd(test_case->fn, test_case->desc);
 	run_with_tmpfile(test_case->fn, test_case->desc);
-	if (thpsize)
+	if (pmdsize)
 		run_with_huge_zeropage(test_case->fn, test_case->desc);
 	for (i = 0; i < nr_hugetlbsizes; i++)
 		run_with_memfd_hugetlb(test_case->fn, test_case->desc,
@@ -1671,7 +1681,7 @@ static int tests_per_non_anon_test_case(void)
 {
 	int tests = 3 + nr_hugetlbsizes;
 
-	if (thpsize)
+	if (pmdsize)
 		tests += 1;
 	return tests;
 }
@@ -1681,10 +1691,13 @@ int main(int argc, char **argv)
 	int err;
 
 	pagesize = getpagesize();
-	thpsize = read_pmd_pagesize();
-	if (thpsize)
+	pmdsize = read_pmd_pagesize();
+	if (pmdsize) {
+		ksft_print_msg("[INFO] detected PMD size: %zu KiB\n",
+			       pmdsize / 1024);
 		ksft_print_msg("[INFO] detected THP size: %zu KiB\n",
-			       thpsize / 1024);
+			       pmdsize / 1024);
+	}
 	nr_hugetlbsizes = detect_hugetlb_page_sizes(hugetlbsizes,
 						    ARRAY_SIZE(hugetlbsizes));
 	detect_huge_zeropage();
Signature · 1 line
2.25.1
author reviewer
#36 lore raw

PATCH v9 10/10 selftests/mm/cow: Add tests for anonymous multi-size THP

In reply to Ryan Roberts

Add tests similar to the existing PMD-sized THP tests, but which operate on memory backed by (PTE-mapped) multi-size THP. This reuses all the existing infrastructure. If the test suite detects that multi-size THP is not supported by the kernel, the new tests are skipped.

---
 tools/testing/selftests/mm/cow.c | 84 +++++++++++++++++++++++++++-----
 1 file changed, 72 insertions(+), 12 deletions(-)
tools/testing/selftests/mm/cow.c +72 -12
index 4d0b5a125d3c..37b4d7d28ae9 100644
@@ -29,15 +29,49 @@
 #include "../../../../mm/gup_test.h"
 #include "../kselftest.h"
 #include "vm_util.h"
+#include "thp_settings.h"
 
 static size_t pagesize;
 static int pagemap_fd;
 static size_t pmdsize;
+static int nr_thpsizes;
+static size_t thpsizes[20];
 static int nr_hugetlbsizes;
 static size_t hugetlbsizes[10];
 static int gup_fd;
 static bool has_huge_zeropage;
 
+static int sz2ord(size_t size)
+{
+	return __builtin_ctzll(size / pagesize);
+}
+
+static int detect_thp_sizes(size_t sizes[], int max)
+{
+	int count = 0;
+	unsigned long orders;
+	size_t kb;
+	int i;
+
+	/* thp not supported at all. */
+	if (!pmdsize)
+		return 0;
+
+	orders = 1UL << sz2ord(pmdsize);
+	orders |= thp_supported_orders();
+
+	for (i = 0; orders && count < max; i++) {
+		if (!(orders & (1UL << i)))
+			continue;
+		orders &= ~(1UL << i);
+		kb = (pagesize >> 10) << i;
+		sizes[count++] = kb * 1024;
+		ksft_print_msg("[INFO] detected THP size: %zu KiB\n", kb);
+	}
+
+	return count;
+}
+
 static void detect_huge_zeropage(void)
 {
 	int fd = open("/sys/kernel/mm/transparent_hugepage/use_zero_page",
@@ -1101,15 +1135,27 @@ static void run_anon_test_case(struct test_case const *test_case)
 
 	run_with_base_page(test_case->fn, test_case->desc);
 	run_with_base_page_swap(test_case->fn, test_case->desc);
-	if (pmdsize) {
-		run_with_thp(test_case->fn, test_case->desc, pmdsize);
-		run_with_thp_swap(test_case->fn, test_case->desc, pmdsize);
-		run_with_pte_mapped_thp(test_case->fn, test_case->desc, pmdsize);
-		run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc, pmdsize);
-		run_with_single_pte_of_thp(test_case->fn, test_case->desc, pmdsize);
-		run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc, pmdsize);
-		run_with_partial_mremap_thp(test_case->fn, test_case->desc, pmdsize);
-		run_with_partial_shared_thp(test_case->fn, test_case->desc, pmdsize);
+	for (i = 0; i < nr_thpsizes; i++) {
+		size_t size = thpsizes[i];
+		struct thp_settings settings = *thp_current_settings();
+
+		settings.hugepages[sz2ord(pmdsize)].enabled = THP_NEVER;
+		settings.hugepages[sz2ord(size)].enabled = THP_ALWAYS;
+		thp_push_settings(&settings);
+
+		if (size == pmdsize) {
+			run_with_thp(test_case->fn, test_case->desc, size);
+			run_with_thp_swap(test_case->fn, test_case->desc, size);
+		}
+
+		run_with_pte_mapped_thp(test_case->fn, test_case->desc, size);
+		run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc, size);
+		run_with_single_pte_of_thp(test_case->fn, test_case->desc, size);
+		run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc, size);
+		run_with_partial_mremap_thp(test_case->fn, test_case->desc, size);
+		run_with_partial_shared_thp(test_case->fn, test_case->desc, size);
+
+		thp_pop_settings();
 	}
 	for (i = 0; i < nr_hugetlbsizes; i++)
 		run_with_hugetlb(test_case->fn, test_case->desc,
@@ -1130,8 +1176,9 @@ static int tests_per_anon_test_case(void)
 {
 	int tests = 2 + nr_hugetlbsizes;
 
+	tests += 6 * nr_thpsizes;
 	if (pmdsize)
-		tests += 8;
+		tests += 2;
 	return tests;
 }
 
@@ -1689,15 +1736,23 @@ static int tests_per_non_anon_test_case(void)
 int main(int argc, char **argv)
 {
 	int err;
+	struct thp_settings default_settings;
 
 	pagesize = getpagesize();
 	pmdsize = read_pmd_pagesize();
 	if (pmdsize) {
+		/* Only if THP is supported. */
+		thp_read_settings(&default_settings);
+		default_settings.hugepages[sz2ord(pmdsize)].enabled = THP_INHERIT;
+		thp_save_settings();
+		thp_push_settings(&default_settings);
+
 		ksft_print_msg("[INFO] detected PMD size: %zu KiB\n",
 			       pmdsize / 1024);
-		ksft_print_msg("[INFO] detected THP size: %zu KiB\n",
-			       pmdsize / 1024);
+
+		nr_thpsizes = detect_thp_sizes(thpsizes, ARRAY_SIZE(thpsizes));
 	}
+
 	nr_hugetlbsizes = detect_hugetlb_page_sizes(hugetlbsizes,
 						    ARRAY_SIZE(hugetlbsizes));
 	detect_huge_zeropage();
@@ -1716,6 +1771,11 @@ int main(int argc, char **argv)
 	run_anon_thp_test_cases();
 	run_non_anon_test_cases();
 
+	if (pmdsize) {
+		/* Only if THP is supported. */
+		thp_restore_settings();
+	}
+
 	err = ksft_get_fail_cnt();
 	if (err)
 		ksft_exit_fail_msg("%d out of %d tests failed\n",
Signature · 1 line
2.25.1
maintainer
#37 lore raw
In reply to Ryan Roberts
Quoted 5 lines · On Thu, 7 Dec 2023 16:12:01 +0000 Ryan Roberts <[email protected]> wrote:

Hi All,

This is v9 (and hopefully the last) of a series to implement multi-size THP (mTHP) for anonymous memory (previously called "small-sized THP" and "large anonymous folios").

A general point on the [0/N] intro. Bear in mind that this is (intended to be) for ever. Five years hence, people won't be interested in knowing which version the patchset was, in seeing what changed from the previous iteration, etc. This is all important and useful info, of course. But it's best suited for being below the "^---$" separator.

Also, those five-years-from-now people won't want to have to go click on some link to find the performance testing results and suchlike. It's better to paste such important info right into their faces.

author reviewer
#38 lore raw
Quoted 15 lines · Andrew Morton
Quoted 5 lines · On Thu, 7 Dec 2023 16:12:01 +0000 Ryan Roberts <[email protected]> wrote:

Hi All,

This is v9 (and hopefully the last) of a series to implement multi-size THP (mTHP) for anonymous memory (previously called "small-sized THP" and "large anonymous folios").

A general point on the [0/N] intro. Bear in mind that this is (intended to be) for ever. Five years hence, people won't be interested in knowing which version the patchset was, in seeing what changed from the previous iteration, etc. This is all important and useful info, of course. But it's best suited for being below the "^---$" separator.

Quoted 3 lines

Also, those five-years-from-now people won't want to have to go click on some link to find the performance testing results and suchlike. It's better to paste such important info right into their faces.

Sorry about this, Andrew - you've given me feedback on this before, and I've been trying to improve this. I'm obviously not there yet. Will fix for next time.