Skip to contentSkip to conversation

variable-order, large folios for anonymous memory

PATCH In review Ryan Roberts 4 participants 54 messages Jun 26, 2023 – Jun 29, 2023

Subsystems (9)
Memory management x86 MM core arm64 +5 more subsystems
lore
Share & tools
lore
#0 lore raw

cover variable-order, large folios for anonymous memory

Hi All,

Following on from the previous RFCv2 [1], this series implements variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults:

- Since SW (the kernel) is dealing with larger chunks of memory than base pages, there are efficiency savings to be had; fewer page faults, batched PTE and RMAP manipulation, fewer items on lists, etc. In short, we reduce kernel overhead. This should benefit all architectures.
- Since we are now mapping physically contiguous chunks of memory, we can take advantage of HW TLB compression techniques. A reduction in TLB pressure speeds up kernel and user space. arm64 systems have 2 mechanisms to coalesce TLB entries; "the contiguous bit" (architectural) and HPA (uarch).

This patch set deals with the SW side of things only and based on feedback from the RFC, aims to be the most minimal initial change, upon which future incremental changes can be added. For this reason, the new behaviour is hidden behind a new Kconfig switch, CONFIG_LARGE_ANON_FOLIO, which is disabled by default. Although the code has been refactored to parameterize the desired order of the allocation, when the feature is disabled (by forcing the order to be always 0) my performance tests measure no regression. So I'm hoping this will be a suitable mechanism to allow incremental submissions to the kernel without affecting the rest of the world.

The patches are based on top of v6.4 plus Matthew Wilcox's set_ptes() series [2], which is a hard dependency. I'm not sure of Matthew's exact plans for getting that series into the kernel, but I'm hoping we can start the review process on this patch set independently. I have a branch at [3].

I've posted a separate series concerning the HW part (contpte mapping) for arm64 at [4].

Performance
-----------

Below results show 2 benchmarks; kernel compilation and speedometer 2.0 (a javascript benchmark running in Chromium). Both cases are running on Ampere Altra with 1 NUMA node enabled, Ubuntu 22.04 and XFS filesystem. Each benchmark is repeated 15 times over 5 reboots and averaged.

All improvements are relative to baseline-4k. 'anonfolio-basic' is this series. 'anonfolio' is the full patch set similar to the RFC with the additional changes to the extra 3 fault paths. The rest of the configs are described at [4].

Kernel Compilation (smaller is better):

| kernel          |   real-time |   kern-time |   user-time |
|:----------------|------------:|------------:|------------:|
| baseline-4k     |        0.0% |        0.0% |        0.0% |
| anonfolio-basic |       -5.3% |      -42.9% |       -0.6% |
| anonfolio       |       -5.4% |      -46.0% |       -0.3% |
| contpte         |       -6.8% |      -45.7% |       -2.1% |
| exefolio        |       -8.4% |      -46.4% |       -3.7% |
| baseline-16k    |       -8.7% |      -49.2% |       -3.7% |
| baseline-64k    |      -10.5% |      -66.0% |       -3.5% |

Speedometer 2.0 (bigger is better):

| kernel          |   runs_per_min |
|:----------------|---------------:|
| baseline-4k     |           0.0% |
| anonfolio-basic |           0.7% |
| anonfolio       |           1.2% |
| contpte         |           3.1% |
| exefolio        |           4.2% |
| baseline-16k    |           5.3% |

Changes since RFCv2
-------------------

- Simplified series to bare minimum (on David Hildenbrand's advice)
- Removed changes to 3 fault paths:
- write fault on zero page: wp_page_copy()
- write fault on non-exclusive CoW page: wp_page_copy()
- write fault on exclusive CoW page: do_wp_page()/wp_page_reuse()
- Only 1 fault path change remains:
- write fault on unallocated address: do_anonymous_page()
- Removed support patches that are no longer needed
- Added Kconfig CONFIG_LARGE_ANON_FOLIO and friends
- Whole feature defaults to off
- Arch opts-in to allowing feature and provides max allocation order

Future Work
-----------

Once this series is in, there are some more incremental changes I plan to follow up with:

- Add the other 3 fault path changes back in
- Properly support pte-mapped folios for:
- numa balancing (do_numa_page())
- fix assumptions about exclusivity for large folios in madvise()
- compaction (although I think this is already a problem for large folios in the file cache so perhaps someone is working on it?)

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://gitlab.arm.com/linux-arm/linux-rr/-/tree/features/granule_perf/anonfolio-lkml_v1
[4] https://lore.kernel.org/linux-arm-kernel/[email protected]/

Thanks,
Ryan

Ryan Roberts (10):
mm: Expose clear_huge_page() unconditionally
mm: pass gfp flags and order to vma_alloc_zeroed_movable_folio()
mm: Introduce try_vma_alloc_movable_folio()
mm: Implement folio_add_new_anon_rmap_range()
mm: Implement folio_remove_rmap_range()
mm: Allow deferred splitting of arbitrary large anon folios
mm: Batch-zap large anonymous folio PTE mappings
mm: Kconfig hooks to determine max anon folio allocation order
arm64: mm: Declare support for large anonymous folios
mm: Allocate large folios for anonymous memory

 arch/alpha/include/asm/page.h   |   5 +-
 arch/arm64/Kconfig              |  13 ++
 arch/arm64/include/asm/page.h   |   3 +-
 arch/arm64/mm/fault.c           |   7 +-
 arch/ia64/include/asm/page.h    |   5 +-
 arch/m68k/include/asm/page_no.h |   7 +-
 arch/s390/include/asm/page.h    |   5 +-
 arch/x86/include/asm/page.h     |   5 +-
 include/linux/highmem.h         |  23 ++-
 include/linux/mm.h              |   3 +-
 include/linux/rmap.h            |   4 +
 mm/Kconfig                      |  39 ++++
 mm/memory.c                     | 324 ++++++++++++++++++++++++++++++--
 mm/rmap.c                       | 107 ++++++++++-
 14 files changed, 506 insertions(+), 44 deletions(-)

--
2.25.1

#1 lore raw

PATCH 1/10 mm: Expose clear_huge_page() unconditionally

In preparation for extending vma_alloc_zeroed_movable_folio() to allocate a arbitrary order folio, expose clear_huge_page() unconditionally, so that it can be used to zero the allocated folio in the generic implementation of vma_alloc_zeroed_movable_folio().

---
 include/linux/mm.h | 3 ++-
 mm/memory.c        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
include/linux/mm.h +2 -1
index 7f1741bd870a..7e3bf45e6491 100644
@@ -3684,10 +3684,11 @@ enum mf_action_page_type {
  */
 extern const struct attribute_group memory_failure_attr_group;
 
-#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
 extern void clear_huge_page(struct page *page,
 			    unsigned long addr_hint,
 			    unsigned int pages_per_huge_page);
+
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
 int copy_user_large_folio(struct folio *dst, struct folio *src,
 			  unsigned long addr_hint,
 			  struct vm_area_struct *vma);
mm/memory.c +1 -1
index fb30f7523550..3d4ea668c4d1 100644
@@ -5741,7 +5741,6 @@ void __might_fault(const char *file, int line)
 EXPORT_SYMBOL(__might_fault);
 #endif
 
-#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
 /*
  * Process all subpages of the specified huge page with the specified
  * operation.  The target subpage will be processed last to keep its
@@ -5839,6 +5838,7 @@ void clear_huge_page(struct page *page,
 	process_huge_page(addr_hint, pages_per_huge_page, clear_subpage, page);
 }
 
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
 static int copy_user_gigantic_page(struct folio *dst, struct folio *src,
 				     unsigned long addr,
 				     struct vm_area_struct *vma,
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH v1 01/10] mm: Expose clear_huge_page() unconditionally

Quoted 17 lines · On Mon, Jun 26, 2023 at 11:14 AM Ryan Roberts <[email protected]> wrote:

In preparation for extending vma_alloc_zeroed_movable_folio() to allocate a arbitrary order folio, expose clear_huge_page() unconditionally, so that it can be used to zero the allocated folio in the generic implementation of vma_alloc_zeroed_movable_folio().

---
 include/linux/mm.h | 3 ++-
 mm/memory.c        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
include/linux/mm.h +0 -0
index 7f1741bd870a..7e3bf45e6491 100644

include/linux/mm.h enum mf_action_page_type { · line 3691

Quoted 9 lines
include/linux/mm.h +2 -1
@@ -3684,10 +3684,11 @@ enum mf_action_page_type {
  */
 extern const struct attribute_group memory_failure_attr_group;
 
-#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
 extern void clear_huge_page(struct page *page,
                            unsigned long addr_hint,
                            unsigned int pages_per_huge_page);
+
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)

We might not want to depend on THP eventually. Right now, we still have to, unless splitting is optional, which seems to contradict 06/10. (deferred_split_folio() is a nop without THP.)

#3 lore raw

Re: [PATCH v1 01/10] mm: Expose clear_huge_page() unconditionally

Quoted 21 lines · Yu Zhao
Quoted 20 lines · On Mon, Jun 26, 2023 at 11:14 AM Ryan Roberts <[email protected]> wrote:

In preparation for extending vma_alloc_zeroed_movable_folio() to allocate a arbitrary order folio, expose clear_huge_page() unconditionally, so that it can be used to zero the allocated folio in the generic implementation of vma_alloc_zeroed_movable_folio().

---
 include/linux/mm.h | 3 ++-
 mm/memory.c        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
include/linux/mm.h +0 -0
index 7f1741bd870a..7e3bf45e6491 100644
@@ -3684,10 +3684,11 @@ enum mf_action_page_type {
  */
 extern const struct attribute_group memory_failure_attr_group;
 
Quoted 10 lines

include/linux/mm.h enum mf_action_page_type { · line 3691

Quoted 6 lines
include/linux/mm.h +2 -1
-#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
 extern void clear_huge_page(struct page *page,
                            unsigned long addr_hint,
                            unsigned int pages_per_huge_page);
+
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)

We might not want to depend on THP eventually. Right now, we still have to, unless splitting is optional, which seems to contradict 06/10. (deferred_split_folio() is a nop without THP.)

Yes, I agree - for large anon folios to work, we depend on THP. But I don't think that helps us here.

In the next patch, I give vma_alloc_zeroed_movable_folio() an extra order parameter. So the generic/default version of the function now needs a way to clear a compound page.

I guess I could do something like:

static inline
struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
unsigned long vaddr, gfp_t gfp, int order)
{
struct folio *folio;

	folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp,
					order, vma, vaddr, false);
 	if (folio) {
#ifdef CONFIG_LARGE_FOLIO
		clear_huge_page(&folio->page, vaddr, 1U << order);
#else
		BUG_ON(order != 0);
		clear_user_highpage(&folio->page, vaddr);
#endif
	}

return folio;
}

But that's pretty messy and there's no reason why other users might come along that pass order != 0 and will be surprised by the BUG_ON.

#4 lore raw

Re: [PATCH v1 01/10] mm: Expose clear_huge_page() unconditionally

Quoted 61 lines · Ryan Roberts
Quoted 21 lines · Yu Zhao
Quoted 20 lines · Ryan Roberts

In preparation for extending vma_alloc_zeroed_movable_folio() to allocate a arbitrary order folio, expose clear_huge_page() unconditionally, so that it can be used to zero the allocated folio in the generic implementation of vma_alloc_zeroed_movable_folio().

---
 include/linux/mm.h | 3 ++-
 mm/memory.c        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
include/linux/mm.h +0 -0
index 7f1741bd870a..7e3bf45e6491 100644
@@ -3684,10 +3684,11 @@ enum mf_action_page_type {
  */
 extern const struct attribute_group memory_failure_attr_group;
 
Quoted 10 lines

include/linux/mm.h enum mf_action_page_type { · line 3691

Quoted 6 lines
include/linux/mm.h +2 -1
-#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
 extern void clear_huge_page(struct page *page,
                            unsigned long addr_hint,
                            unsigned int pages_per_huge_page);
+
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)

We might not want to depend on THP eventually. Right now, we still have to, unless splitting is optional, which seems to contradict 06/10. (deferred_split_folio() is a nop without THP.)

Yes, I agree - for large anon folios to work, we depend on THP. But I don't think that helps us here.

In the next patch, I give vma_alloc_zeroed_movable_folio() an extra order parameter. So the generic/default version of the function now needs a way to clear a compound page.

I guess I could do something like:

static inline
struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
unsigned long vaddr, gfp_t gfp, int order)

 {
        struct folio *folio;

        folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp,
                                        order, vma, vaddr, false);
        if (folio) {
#ifdef CONFIG_LARGE_FOLIO
                clear_huge_page(&folio->page, vaddr, 1U << order);
#else
                BUG_ON(order != 0);
                clear_user_highpage(&folio->page, vaddr);
#endif
        }

        return folio;
Quoted 4 lines

}

But that's pretty messy and there's no reason why other users might come along that pass order != 0 and will be surprised by the BUG_ON.

#ifdef CONFIG_LARGE_ANON_FOLIO // depends on CONFIG_TRANSPARENT_HUGE_PAGE struct folio *alloc_anon_folio(struct vm_area_struct *vma, unsigned long vaddr, int order)
{
// how do_huge_pmd_anonymous_page() allocs and clears vma_alloc_folio(..., *true*);
}
#else
#define alloc_anon_folio(vma, addr, order) vma_alloc_zeroed_movable_folio(vma, addr)
#endif

#5 lore raw

Re: [PATCH v1 01/10] mm: Expose clear_huge_page() unconditionally

Quoted 67 lines · Yu Zhao
Quoted 61 lines · Ryan Roberts
Quoted 21 lines · Yu Zhao
Quoted 20 lines · Ryan Roberts

In preparation for extending vma_alloc_zeroed_movable_folio() to allocate a arbitrary order folio, expose clear_huge_page() unconditionally, so that it can be used to zero the allocated folio in the generic implementation of vma_alloc_zeroed_movable_folio().

---
 include/linux/mm.h | 3 ++-
 mm/memory.c        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
include/linux/mm.h +0 -0
index 7f1741bd870a..7e3bf45e6491 100644
@@ -3684,10 +3684,11 @@ enum mf_action_page_type {
  */
 extern const struct attribute_group memory_failure_attr_group;
 
Quoted 10 lines

include/linux/mm.h enum mf_action_page_type { · line 3691

Quoted 6 lines
include/linux/mm.h +2 -1
-#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
 extern void clear_huge_page(struct page *page,
                            unsigned long addr_hint,
                            unsigned int pages_per_huge_page);
+
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)

We might not want to depend on THP eventually. Right now, we still have to, unless splitting is optional, which seems to contradict 06/10. (deferred_split_folio() is a nop without THP.)

Yes, I agree - for large anon folios to work, we depend on THP. But I don't think that helps us here.

In the next patch, I give vma_alloc_zeroed_movable_folio() an extra order parameter. So the generic/default version of the function now needs a way to clear a compound page.

I guess I could do something like:

static inline
struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
unsigned long vaddr, gfp_t gfp, int order)

 {
        struct folio *folio;

        folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp,
                                        order, vma, vaddr, false);
        if (folio) {
#ifdef CONFIG_LARGE_FOLIO
                clear_huge_page(&folio->page, vaddr, 1U << order);
#else
                BUG_ON(order != 0);
                clear_user_highpage(&folio->page, vaddr);
#endif
        }

        return folio;
Quoted 4 lines

}

But that's pretty messy and there's no reason why other users might come along that pass order != 0 and will be surprised by the BUG_ON.

Quoted 6 lines

#ifdef CONFIG_LARGE_ANON_FOLIO // depends on CONFIG_TRANSPARENT_HUGE_PAGE struct folio *alloc_anon_folio(struct vm_area_struct *vma, unsigned long vaddr, int order)
{
// how do_huge_pmd_anonymous_page() allocs and clears vma_alloc_folio(..., *true*);

This controls the mem allocation policy (see mempolicy.c::vma_alloc_folio()) not clearing. Clearing is done in __do_huge_pmd_anonymous_page():

  clear_huge_page(page, vmf->address, HPAGE_PMD_NR);
Quoted 5 lines

}
#else
#define alloc_anon_folio(vma, addr, order)
vma_alloc_zeroed_movable_folio(vma, addr)
#endif

Sorry I don't get this at all... If you are suggesting to bypass vma_alloc_zeroed_movable_folio() entirely for the LARGE_ANON_FOLIO case, I don't think that works because the arch code adds its own gfp flags there. For example, arm64 adds __GFP_ZEROTAGS for VM_MTE VMAs.

Perhaps we can do away with an arch-owned vma_alloc_zeroed_movable_folio() and replace it with a new arch_get_zeroed_movable_gfp_flags() then alloc_anon_folio() add in those flags?

But I still think the cleanest, simplest change is just to unconditionally expose clear_huge_page() as I've done it.

#6 lore raw

Re: [PATCH v1 01/10] mm: Expose clear_huge_page() unconditionally

Quoted 76 lines · Ryan Roberts
Quoted 67 lines · Yu Zhao
Quoted 61 lines · Ryan Roberts
Quoted 21 lines · Yu Zhao
Quoted 20 lines · Ryan Roberts

In preparation for extending vma_alloc_zeroed_movable_folio() to allocate a arbitrary order folio, expose clear_huge_page() unconditionally, so that it can be used to zero the allocated folio in the generic implementation of vma_alloc_zeroed_movable_folio().

---
 include/linux/mm.h | 3 ++-
 mm/memory.c        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
include/linux/mm.h +0 -0
index 7f1741bd870a..7e3bf45e6491 100644
@@ -3684,10 +3684,11 @@ enum mf_action_page_type {
  */
 extern const struct attribute_group memory_failure_attr_group;
 
Quoted 10 lines

include/linux/mm.h enum mf_action_page_type { · line 3691

Quoted 6 lines
include/linux/mm.h +2 -1
-#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
 extern void clear_huge_page(struct page *page,
                            unsigned long addr_hint,
                            unsigned int pages_per_huge_page);
+
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)

We might not want to depend on THP eventually. Right now, we still have to, unless splitting is optional, which seems to contradict 06/10. (deferred_split_folio() is a nop without THP.)

Yes, I agree - for large anon folios to work, we depend on THP. But I don't think that helps us here.

In the next patch, I give vma_alloc_zeroed_movable_folio() an extra order parameter. So the generic/default version of the function now needs a way to clear a compound page.

I guess I could do something like:

static inline
struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
unsigned long vaddr, gfp_t gfp, int order)

 {
        struct folio *folio;

        folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp,
                                        order, vma, vaddr, false);
        if (folio) {
#ifdef CONFIG_LARGE_FOLIO
                clear_huge_page(&folio->page, vaddr, 1U << order);
#else
                BUG_ON(order != 0);
                clear_user_highpage(&folio->page, vaddr);
#endif
        }

        return folio;
Quoted 4 lines

}

But that's pretty messy and there's no reason why other users might come along that pass order != 0 and will be surprised by the BUG_ON.

Quoted 6 lines

#ifdef CONFIG_LARGE_ANON_FOLIO // depends on CONFIG_TRANSPARENT_HUGE_PAGE struct folio *alloc_anon_folio(struct vm_area_struct *vma, unsigned long vaddr, int order)
{
// how do_huge_pmd_anonymous_page() allocs and clears vma_alloc_folio(..., *true*);

Quoted 4 lines

This controls the mem allocation policy (see mempolicy.c::vma_alloc_folio()) not clearing. Clearing is done in __do_huge_pmd_anonymous_page():

  clear_huge_page(page, vmf->address, HPAGE_PMD_NR);

Sorry for rushing this previously. This is what I meant. The #ifdef makes it safe to use clear_huge_page() without 01/10. I highlighted the last parameter to vma_alloc_folio() only because it's different from what you chose (not implying it clears the folio).

Quoted 8 lines
Quoted 5 lines

}
#else
#define alloc_anon_folio(vma, addr, order)
vma_alloc_zeroed_movable_folio(vma, addr)
#endif

Sorry I don't get this at all... If you are suggesting to bypass vma_alloc_zeroed_movable_folio() entirely for the LARGE_ANON_FOLIO case

Correct.

Quoted 3 lines

I don't
think that works because the arch code adds its own gfp flags there. For example, arm64 adds __GFP_ZEROTAGS for VM_MTE VMAs.

I think it's the opposite: it should be safer to reuse the THP code because
1. It's an existing case that has been working for PMD_ORDER folios mapped by PTEs, and it's an arch-independent API which would be easier to review.
2. Use vma_alloc_zeroed_movable_folio() for large folios is a *new* case. It's an arch-*dependent* API which I have no idea what VM_MTE does (should do) to large folios and don't plan to answer that for now.

Quoted 6 lines

Perhaps we can do away with an arch-owned vma_alloc_zeroed_movable_folio() and replace it with a new arch_get_zeroed_movable_gfp_flags() then alloc_anon_folio() add in those flags?

But I still think the cleanest, simplest change is just to unconditionally expose clear_huge_page() as I've done it.

The fundamental choice there as I see it is to whether the first step of large anon folios should lean toward the THP code base or the base page code base (I'm a big fan of the answer "Neither -- we should create something entirely new instead"). My POV is that the THP code base would allow us to move faster, since it's proven to work for a very similar case (PMD_ORDER folios mapped by PTEs).

#7 lore raw

Re: [PATCH v1 01/10] mm: Expose clear_huge_page() unconditionally

Quoted 101 lines · Yu Zhao
Quoted 76 lines · Ryan Roberts
Quoted 67 lines · Yu Zhao
Quoted 61 lines · Ryan Roberts
Quoted 21 lines · Yu Zhao
Quoted 20 lines · Ryan Roberts

In preparation for extending vma_alloc_zeroed_movable_folio() to allocate a arbitrary order folio, expose clear_huge_page() unconditionally, so that it can be used to zero the allocated folio in the generic implementation of vma_alloc_zeroed_movable_folio().

---
 include/linux/mm.h | 3 ++-
 mm/memory.c        | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
include/linux/mm.h +0 -0
index 7f1741bd870a..7e3bf45e6491 100644
@@ -3684,10 +3684,11 @@ enum mf_action_page_type {
  */
 extern const struct attribute_group memory_failure_attr_group;
 
Quoted 10 lines

include/linux/mm.h enum mf_action_page_type { · line 3691

Quoted 6 lines
include/linux/mm.h +2 -1
-#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
 extern void clear_huge_page(struct page *page,
                            unsigned long addr_hint,
                            unsigned int pages_per_huge_page);
+
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)

We might not want to depend on THP eventually. Right now, we still have to, unless splitting is optional, which seems to contradict 06/10. (deferred_split_folio() is a nop without THP.)

Yes, I agree - for large anon folios to work, we depend on THP. But I don't think that helps us here.

In the next patch, I give vma_alloc_zeroed_movable_folio() an extra order parameter. So the generic/default version of the function now needs a way to clear a compound page.

I guess I could do something like:

static inline
struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
unsigned long vaddr, gfp_t gfp, int order)

 {
        struct folio *folio;

        folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp,
                                        order, vma, vaddr, false);
        if (folio) {
#ifdef CONFIG_LARGE_FOLIO
                clear_huge_page(&folio->page, vaddr, 1U << order);
#else
                BUG_ON(order != 0);
                clear_user_highpage(&folio->page, vaddr);
#endif
        }

        return folio;
Quoted 4 lines

}

But that's pretty messy and there's no reason why other users might come along that pass order != 0 and will be surprised by the BUG_ON.

Quoted 6 lines

#ifdef CONFIG_LARGE_ANON_FOLIO // depends on CONFIG_TRANSPARENT_HUGE_PAGE struct folio *alloc_anon_folio(struct vm_area_struct *vma, unsigned long vaddr, int order)
{
// how do_huge_pmd_anonymous_page() allocs and clears vma_alloc_folio(..., *true*);

Quoted 4 lines

This controls the mem allocation policy (see mempolicy.c::vma_alloc_folio()) not clearing. Clearing is done in __do_huge_pmd_anonymous_page():

  clear_huge_page(page, vmf->address, HPAGE_PMD_NR);

Sorry for rushing this previously. This is what I meant. The #ifdef makes it safe to use clear_huge_page() without 01/10. I highlighted the last parameter to vma_alloc_folio() only because it's different from what you chose (not implying it clears the folio).>

Quoted 8 lines
Quoted 5 lines

}
#else
#define alloc_anon_folio(vma, addr, order)
vma_alloc_zeroed_movable_folio(vma, addr)
#endif

Sorry I don't get this at all... If you are suggesting to bypass vma_alloc_zeroed_movable_folio() entirely for the LARGE_ANON_FOLIO case

Correct.

Quoted 3 lines

I don't
think that works because the arch code adds its own gfp flags there. For example, arm64 adds __GFP_ZEROTAGS for VM_MTE VMAs.

Quoted 8 lines

I think it's the opposite: it should be safer to reuse the THP code because
1. It's an existing case that has been working for PMD_ORDER folios mapped by PTEs, and it's an arch-independent API which would be easier to review.
2. Use vma_alloc_zeroed_movable_folio() for large folios is a *new* case. It's an arch-*dependent* API which I have no idea what VM_MTE does (should do) to large folios and don't plan to answer that for now.

I've done some archaology on this now, and convinced myself that your suggestion is a good one - sorry for doubting it!

If you are interested here are the details: Only arm64 and ia64 do something non-standard in vma_alloc_zeroed_movable_folio(). ia64 flushes the dcache for the folio - but given it does not support THP this is not a problem for the THP path. arm64 adds the __GFP_ZEROTAGS flag which means that the MTE tags will be zeroed at the same time as the page is zeroed. This is a perf optimization - if its not performed then it will be done at set_pte_at(), which is how this works for the THP path.

So on that basis, I agree we can use your proposed alloc_anon_folio() approach. arm64 will lose the MTE optimization but that can be added back later if needed. So no need to unconditionally expose clear_huge_page() and no need to modify all the arch vma_alloc_zeroed_movable_folio() implementations.

Thanks,
Ryan

Quoted 14 lines
Quoted 7 lines

Perhaps we can do away with an arch-owned vma_alloc_zeroed_movable_folio() and replace it with a new arch_get_zeroed_movable_gfp_flags() then alloc_anon_folio() add in those flags?

But I still think the cleanest, simplest change is just to unconditionally expose clear_huge_page() as I've done it.

The fundamental choice there as I see it is to whether the first step of large anon folios should lean toward the THP code base or the base page code base (I'm a big fan of the answer "Neither -- we should create something entirely new instead"). My POV is that the THP code base would allow us to move faster, since it's proven to work for a very similar case (PMD_ORDER folios mapped by PTEs).

#8 lore raw

PATCH 2/10 mm: pass gfp flags and order to vma_alloc_zeroed_movable_folio()

In reply to Ryan Roberts

Allow allocation of large folios with vma_alloc_zeroed_movable_folio(). This prepares the ground for large anonymous folios. The generic implementation of vma_alloc_zeroed_movable_folio() now uses clear_huge_page() to zero the allocated folio since it may now be a non-0 order.

Currently the function is always called with order 0 and no extra gfp flags, so no functional change intended. But a subsequent commit will take advantage of the new parameters to allocate large folios. The extra gfp flags will be used to control the reclaim policy.

---
 arch/alpha/include/asm/page.h   |  5 +++--
 arch/arm64/include/asm/page.h   |  3 ++-
 arch/arm64/mm/fault.c           |  7 ++++---
 arch/ia64/include/asm/page.h    |  5 +++--
 arch/m68k/include/asm/page_no.h |  7 ++++---
 arch/s390/include/asm/page.h    |  5 +++--
 arch/x86/include/asm/page.h     |  5 +++--
 include/linux/highmem.h         | 23 +++++++++++++----------
 mm/memory.c                     |  5 +++--
 9 files changed, 38 insertions(+), 27 deletions(-)
arch/alpha/include/asm/page.h +3 -2
index 4db1ebc0ed99..6fc7fe91b6cb 100644
@@ -17,8 +17,9 @@
 extern void clear_page(void *page);
 #define clear_user_page(page, vaddr, pg)	clear_page(page)
 
-#define vma_alloc_zeroed_movable_folio(vma, vaddr) \
-	vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false)
+#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \
+	vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \
+			order, vma, vaddr, false)
 
 extern void copy_page(void * _to, void * _from);
 #define copy_user_page(to, from, vaddr, pg)	copy_page(to, from)
arch/arm64/include/asm/page.h +2 -1
index 2312e6ee595f..47710852f872 100644
@@ -30,7 +30,8 @@ void copy_highpage(struct page *to, struct page *from);
 #define __HAVE_ARCH_COPY_HIGHPAGE
 
 struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
-						unsigned long vaddr);
+						unsigned long vaddr,
+						gfp_t gfp, int order);
 #define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio
 
 void tag_clear_highpage(struct page *to);
arch/arm64/mm/fault.c +4 -3
index 6045a5117ac1..0a43c3b3f190 100644
@@ -961,9 +961,10 @@ NOKPROBE_SYMBOL(do_debug_exception);
  * Used during anonymous page fault handling.
  */
 struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
-						unsigned long vaddr)
+						unsigned long vaddr,
+						gfp_t gfp, int order)
 {
-	gfp_t flags = GFP_HIGHUSER_MOVABLE | __GFP_ZERO;
+	gfp_t flags = GFP_HIGHUSER_MOVABLE | __GFP_ZERO | gfp;
 
 	/*
 	 * If the page is mapped with PROT_MTE, initialise the tags at the
@@ -973,7 +974,7 @@ struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
 	if (vma->vm_flags & VM_MTE)
 		flags |= __GFP_ZEROTAGS;
 
-	return vma_alloc_folio(flags, 0, vma, vaddr, false);
+	return vma_alloc_folio(flags, order, vma, vaddr, false);
 }
 
 void tag_clear_highpage(struct page *page)
arch/ia64/include/asm/page.h +3 -2
index 310b09c3342d..ebdf04274023 100644
@@ -82,10 +82,11 @@ do {						\
 } while (0)
 
 
-#define vma_alloc_zeroed_movable_folio(vma, vaddr)			\
+#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order)		\
 ({									\
 	struct folio *folio = vma_alloc_folio(				\
-		GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false); \
+		GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp),		\
+		order, vma, vaddr, false);				\
 	if (folio)							\
 		flush_dcache_folio(folio);				\
 	folio;								\
arch/m68k/include/asm/page_no.h +4 -3
index 060e4c0e7605..4a2fe57fef5e 100644
@@ -3,7 +3,7 @@
 #define _M68K_PAGE_NO_H
 
 #ifndef __ASSEMBLY__
- 
+
 extern unsigned long memory_start;
 extern unsigned long memory_end;
 
@@ -13,8 +13,9 @@ extern unsigned long memory_end;
 #define clear_user_page(page, vaddr, pg)	clear_page(page)
 #define copy_user_page(to, from, vaddr, pg)	copy_page(to, from)
 
-#define vma_alloc_zeroed_movable_folio(vma, vaddr) \
-	vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false)
+#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \
+	vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \
+			order, vma, vaddr, false)
 
 #define __pa(vaddr)		((unsigned long)(vaddr))
 #define __va(paddr)		((void *)((unsigned long)(paddr)))
arch/s390/include/asm/page.h +3 -2
index 8a2a3b5d1e29..b749564140f1 100644
@@ -73,8 +73,9 @@ static inline void copy_page(void *to, void *from)
 #define clear_user_page(page, vaddr, pg)	clear_page(page)
 #define copy_user_page(to, from, vaddr, pg)	copy_page(to, from)
 
-#define vma_alloc_zeroed_movable_folio(vma, vaddr) \
-	vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false)
+#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \
+	vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \
+			order, vma, vaddr, false)
 
 /*
  * These are used to make use of C type-checking..
arch/x86/include/asm/page.h +3 -2
index d18e5c332cb9..34deab1a8dae 100644
@@ -34,8 +34,9 @@ static inline void copy_user_page(void *to, void *from, unsigned long vaddr,
 	copy_page(to, from);
 }
 
-#define vma_alloc_zeroed_movable_folio(vma, vaddr) \
-	vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false)
+#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \
+	vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \
+			order, vma, vaddr, false)
 
 #ifndef __pa
 #define __pa(x)		__phys_addr((unsigned long)(x))
include/linux/highmem.h +13 -10
index 4de1dbcd3ef6..b9a9b0340557 100644
@@ -209,26 +209,29 @@ static inline void clear_user_highpage(struct page *page, unsigned long vaddr)
 
 #ifndef vma_alloc_zeroed_movable_folio
 /**
- * vma_alloc_zeroed_movable_folio - Allocate a zeroed page for a VMA.
- * @vma: The VMA the page is to be allocated for.
- * @vaddr: The virtual address the page will be inserted into.
- *
- * This function will allocate a page suitable for inserting into this
- * VMA at this virtual address.  It may be allocated from highmem or
+ * vma_alloc_zeroed_movable_folio - Allocate a zeroed folio for a VMA.
+ * @vma: The start VMA the folio is to be allocated for.
+ * @vaddr: The virtual address the folio will be inserted into.
+ * @gfp: Additional gfp falgs to mix in or 0.
+ * @order: The order of the folio (2^order pages).
+ *
+ * This function will allocate a folio suitable for inserting into this
+ * VMA starting at this virtual address.  It may be allocated from highmem or
  * the movable zone.  An architecture may provide its own implementation.
  *
- * Return: A folio containing one allocated and zeroed page or NULL if
+ * Return: A folio containing 2^order allocated and zeroed pages or NULL if
  * we are out of memory.
  */
 static inline
 struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
-				   unsigned long vaddr)
+				   unsigned long vaddr, gfp_t gfp, int order)
 {
 	struct folio *folio;
 
-	folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vaddr, false);
+	folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp,
+					order, vma, vaddr, false);
 	if (folio)
-		clear_user_highpage(&folio->page, vaddr);
+		clear_huge_page(&folio->page, vaddr, 1U << order);
 
 	return folio;
 }
mm/memory.c +3 -2
index 3d4ea668c4d1..367bbbb29d91 100644
@@ -3073,7 +3073,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)
 		goto oom;
 
 	if (is_zero_pfn(pte_pfn(vmf->orig_pte))) {
-		new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address);
+		new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address,
+									0, 0);
 		if (!new_folio)
 			goto oom;
 	} else {
@@ -4087,7 +4088,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	/* Allocate our own private page. */
 	if (unlikely(anon_vma_prepare(vma)))
 		goto oom;
-	folio = vma_alloc_zeroed_movable_folio(vma, vmf->address);
+	folio = vma_alloc_zeroed_movable_folio(vma, vmf->address, 0, 0);
 	if (!folio)
 		goto oom;
 
Signature · 1 line
2.25.1
#9 lore raw

Re: [PATCH v1 02/10] mm: pass gfp flags and order to vma_alloc_zeroed_movable_folio()

Quoted 30 lines · On Mon, Jun 26, 2023 at 11:14 AM Ryan Roberts <[email protected]> wrote:

Allow allocation of large folios with vma_alloc_zeroed_movable_folio(). This prepares the ground for large anonymous folios. The generic implementation of vma_alloc_zeroed_movable_folio() now uses clear_huge_page() to zero the allocated folio since it may now be a non-0 order.

Currently the function is always called with order 0 and no extra gfp flags, so no functional change intended. But a subsequent commit will take advantage of the new parameters to allocate large folios. The extra gfp flags will be used to control the reclaim policy.

---
 arch/alpha/include/asm/page.h   |  5 +++--
 arch/arm64/include/asm/page.h   |  3 ++-
 arch/arm64/mm/fault.c           |  7 ++++---
 arch/ia64/include/asm/page.h    |  5 +++--
 arch/m68k/include/asm/page_no.h |  7 ++++---
 arch/s390/include/asm/page.h    |  5 +++--
 arch/x86/include/asm/page.h     |  5 +++--
 include/linux/highmem.h         | 23 +++++++++++++----------
 mm/memory.c                     |  5 +++--
 9 files changed, 38 insertions(+), 27 deletions(-)
arch/alpha/include/asm/page.h +0 -0
index 4db1ebc0ed99..6fc7fe91b6cb 100644

arch/alpha/include/asm/page.h line 22

Quoted 8 lines
arch/alpha/include/asm/page.h +3 -2
@@ -17,8 +17,9 @@
 extern void clear_page(void *page);
 #define clear_user_page(page, vaddr, pg)       clear_page(page)
 
-#define vma_alloc_zeroed_movable_folio(vma, vaddr) \
-       vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false)
+#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \
+       vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \
+                       order, vma, vaddr, false)

I don't think we need to worry about gfp if we want to make a minimum series. There would be many discussion points around it, e.g., I already disagree with what you chose: GFP_TRANSHUGE_LIGHT would be more suitable than __GFP_NORETRY, and there are even better options than GFP_TRANSHUGE_LIGHT.

#10 lore raw

Re: [PATCH v1 02/10] mm: pass gfp flags and order to vma_alloc_zeroed_movable_folio()

Quoted 34 lines · Yu Zhao
Quoted 33 lines · On Mon, Jun 26, 2023 at 11:14 AM Ryan Roberts <[email protected]> wrote:

Allow allocation of large folios with vma_alloc_zeroed_movable_folio(). This prepares the ground for large anonymous folios. The generic implementation of vma_alloc_zeroed_movable_folio() now uses clear_huge_page() to zero the allocated folio since it may now be a non-0 order.

Currently the function is always called with order 0 and no extra gfp flags, so no functional change intended. But a subsequent commit will take advantage of the new parameters to allocate large folios. The extra gfp flags will be used to control the reclaim policy.

---
 arch/alpha/include/asm/page.h   |  5 +++--
 arch/arm64/include/asm/page.h   |  3 ++-
 arch/arm64/mm/fault.c           |  7 ++++---
 arch/ia64/include/asm/page.h    |  5 +++--
 arch/m68k/include/asm/page_no.h |  7 ++++---
 arch/s390/include/asm/page.h    |  5 +++--
 arch/x86/include/asm/page.h     |  5 +++--
 include/linux/highmem.h         | 23 +++++++++++++----------
 mm/memory.c                     |  5 +++--
 9 files changed, 38 insertions(+), 27 deletions(-)
arch/alpha/include/asm/page.h +0 -0
index 4db1ebc0ed99..6fc7fe91b6cb 100644
@@ -17,8 +17,9 @@
 extern void clear_page(void *page);
 #define clear_user_page(page, vaddr, pg)       clear_page(page)
 
Quoted 11 lines

arch/alpha/include/asm/page.h line 22

Quoted 5 lines
arch/alpha/include/asm/page.h +3 -2
-#define vma_alloc_zeroed_movable_folio(vma, vaddr) \
-       vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false)
+#define vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order) \
+       vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO | (gfp), \
+                       order, vma, vaddr, false)

I don't think we need to worry about gfp if we want to make a minimum series. There would be many discussion points around it, e.g., I already disagree with what you chose: GFP_TRANSHUGE_LIGHT would be more suitable than __GFP_NORETRY, and there are even better options than GFP_TRANSHUGE_LIGHT.

OK, but disagreeing about what the GFP flags should be is different from disagreeing about whether we need a mechanism for specifying them. Given I need to do the changes to add order I thought it was sensible to add the gfp flags at the same time.

I'll follow your advice and remove the gfp flag addition for now.

#11 lore raw

PATCH 3/10 mm: Introduce try_vma_alloc_movable_folio()

In reply to Ryan Roberts

Opportunistically attempt to allocate high-order folios in highmem, optionally zeroed. Retry with lower orders all the way to order-0, until success. Although, of note, order-1 allocations are skipped since a large folio must be at least order-2 to work with the THP machinery. The user must check what they got with folio_order().

This will be used to oportunistically allocate large folios for anonymous memory with a sensible fallback under memory pressure.

For attempts to allocate non-0 orders, we set __GFP_NORETRY to prevent high latency due to reclaim, instead preferring to just try for a lower order. The same approach is used by the readahead code when allocating large folios.

---
 mm/memory.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
mm/memory.c +33 -0
index 367bbbb29d91..53896d46e686 100644
@@ -3001,6 +3001,39 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
 	return 0;
 }
 
+static inline struct folio *vma_alloc_movable_folio(struct vm_area_struct *vma,
+				unsigned long vaddr, int order, bool zeroed)
+{
+	gfp_t gfp = order > 0 ? __GFP_NORETRY | __GFP_NOWARN : 0;
+
+	if (zeroed)
+		return vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order);
+	else
+		return vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp, order, vma,
+								vaddr, false);
+}
+
+/*
+ * Opportunistically attempt to allocate high-order folios, retrying with lower
+ * orders all the way to order-0, until success. order-1 allocations are skipped
+ * since a folio must be at least order-2 to work with the THP machinery. The
+ * user must check what they got with folio_order(). vaddr can be any virtual
+ * address that will be mapped by the allocated folio.
+ */
+static struct folio *try_vma_alloc_movable_folio(struct vm_area_struct *vma,
+				unsigned long vaddr, int order, bool zeroed)
+{
+	struct folio *folio;
+
+	for (; order > 1; order--) {
+		folio = vma_alloc_movable_folio(vma, vaddr, order, zeroed);
+		if (folio)
+			return folio;
+	}
+
+	return vma_alloc_movable_folio(vma, vaddr, 0, zeroed);
+}
+
 /*
  * Handle write page faults for pages that can be reused in the current vma
  *
Signature · 1 line
2.25.1
#12 lore raw

Re: [PATCH v1 03/10] mm: Introduce try_vma_alloc_movable_folio()

Quoted 50 lines · On Mon, Jun 26, 2023 at 11:14 AM Ryan Roberts <[email protected]> wrote:

Opportunistically attempt to allocate high-order folios in highmem, optionally zeroed. Retry with lower orders all the way to order-0, until success. Although, of note, order-1 allocations are skipped since a large folio must be at least order-2 to work with the THP machinery. The user must check what they got with folio_order().

This will be used to oportunistically allocate large folios for anonymous memory with a sensible fallback under memory pressure.

For attempts to allocate non-0 orders, we set __GFP_NORETRY to prevent high latency due to reclaim, instead preferring to just try for a lower order. The same approach is used by the readahead code when allocating large folios.

---
 mm/memory.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
mm/memory.c +22 -0
index 367bbbb29d91..53896d46e686 100644
@@ -3001,6 +3001,39 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
        return 0;
 }
 
+static inline struct folio *vma_alloc_movable_folio(struct vm_area_struct *vma,
+                               unsigned long vaddr, int order, bool zeroed)
+{
+       gfp_t gfp = order > 0 ? __GFP_NORETRY | __GFP_NOWARN : 0;
+
+       if (zeroed)
+               return vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order);
+       else
+               return vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp, order, vma,
+                                                               vaddr, false);
+}
+
+/*
+ * Opportunistically attempt to allocate high-order folios, retrying with lower
+ * orders all the way to order-0, until success. order-1 allocations are skipped
+ * since a folio must be at least order-2 to work with the THP machinery. The
+ * user must check what they got with folio_order(). vaddr can be any virtual
+ * address that will be mapped by the allocated folio.
+ */
+static struct folio *try_vma_alloc_movable_folio(struct vm_area_struct *vma,
+                               unsigned long vaddr, int order, bool zeroed)
+{

mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 3035

Quoted 10 lines
mm/memory.c +10 -0
+       struct folio *folio;
+
+       for (; order > 1; order--) {
+               folio = vma_alloc_movable_folio(vma, vaddr, order, zeroed);
+               if (folio)
+                       return folio;
+       }
+
+       return vma_alloc_movable_folio(vma, vaddr, 0, zeroed);
+}

I'd drop this patch. Instead, in do_anonymous_page():

if (IS_ENABLED(CONFIG_ARCH_WANTS_PTE_ORDER))
folio = vma_alloc_zeroed_movable_folio(vma, addr,
CONFIG_ARCH_WANTS_PTE_ORDER))

  if (!folio)
    folio = vma_alloc_zeroed_movable_folio(vma, addr, 0);
#13 lore raw

Re: [PATCH v1 03/10] mm: Introduce try_vma_alloc_movable_folio()

Quoted 65 lines · Yu Zhao
Quoted 50 lines · Ryan Roberts

Opportunistically attempt to allocate high-order folios in highmem, optionally zeroed. Retry with lower orders all the way to order-0, until success. Although, of note, order-1 allocations are skipped since a large folio must be at least order-2 to work with the THP machinery. The user must check what they got with folio_order().

This will be used to oportunistically allocate large folios for anonymous memory with a sensible fallback under memory pressure.

For attempts to allocate non-0 orders, we set __GFP_NORETRY to prevent high latency due to reclaim, instead preferring to just try for a lower order. The same approach is used by the readahead code when allocating large folios.

---
 mm/memory.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
mm/memory.c +22 -0
index 367bbbb29d91..53896d46e686 100644
@@ -3001,6 +3001,39 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
        return 0;
 }
 
+static inline struct folio *vma_alloc_movable_folio(struct vm_area_struct *vma,
+                               unsigned long vaddr, int order, bool zeroed)
+{
+       gfp_t gfp = order > 0 ? __GFP_NORETRY | __GFP_NOWARN : 0;
+
+       if (zeroed)
+               return vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order);
+       else
+               return vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp, order, vma,
+                                                               vaddr, false);
+}
+
+/*
+ * Opportunistically attempt to allocate high-order folios, retrying with lower
+ * orders all the way to order-0, until success. order-1 allocations are skipped
+ * since a folio must be at least order-2 to work with the THP machinery. The
+ * user must check what they got with folio_order(). vaddr can be any virtual
+ * address that will be mapped by the allocated folio.
+ */
+static struct folio *try_vma_alloc_movable_folio(struct vm_area_struct *vma,
+                               unsigned long vaddr, int order, bool zeroed)
+{

mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 3035

Quoted 10 lines
mm/memory.c +10 -0
+       struct folio *folio;
+
+       for (; order > 1; order--) {
+               folio = vma_alloc_movable_folio(vma, vaddr, order, zeroed);
+               if (folio)
+                       return folio;
+       }
+
+       return vma_alloc_movable_folio(vma, vaddr, 0, zeroed);
+}

I'd drop this patch. Instead, in do_anonymous_page():

Quoted 6 lines

if (IS_ENABLED(CONFIG_ARCH_WANTS_PTE_ORDER))
folio = vma_alloc_zeroed_movable_folio(vma, addr,
CONFIG_ARCH_WANTS_PTE_ORDER))

  if (!folio)
    folio = vma_alloc_zeroed_movable_folio(vma, addr, 0);

I meant a runtime function arch_wants_pte_order() (Its default implementation would return 0.)

#14 lore raw

Re: [PATCH v1 03/10] mm: Introduce try_vma_alloc_movable_folio()

Quoted 70 lines · Yu Zhao
Quoted 69 lines · Yu Zhao
Quoted 50 lines · Ryan Roberts

Opportunistically attempt to allocate high-order folios in highmem, optionally zeroed. Retry with lower orders all the way to order-0, until success. Although, of note, order-1 allocations are skipped since a large folio must be at least order-2 to work with the THP machinery. The user must check what they got with folio_order().

This will be used to oportunistically allocate large folios for anonymous memory with a sensible fallback under memory pressure.

For attempts to allocate non-0 orders, we set __GFP_NORETRY to prevent high latency due to reclaim, instead preferring to just try for a lower order. The same approach is used by the readahead code when allocating large folios.

---
 mm/memory.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
mm/memory.c +22 -0
index 367bbbb29d91..53896d46e686 100644
@@ -3001,6 +3001,39 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
        return 0;
 }
 
+static inline struct folio *vma_alloc_movable_folio(struct vm_area_struct *vma,
+                               unsigned long vaddr, int order, bool zeroed)
+{
+       gfp_t gfp = order > 0 ? __GFP_NORETRY | __GFP_NOWARN : 0;
+
+       if (zeroed)
+               return vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order);
+       else
+               return vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp, order, vma,
+                                                               vaddr, false);
+}
+
+/*
+ * Opportunistically attempt to allocate high-order folios, retrying with lower
+ * orders all the way to order-0, until success. order-1 allocations are skipped
+ * since a folio must be at least order-2 to work with the THP machinery. The
+ * user must check what they got with folio_order(). vaddr can be any virtual
+ * address that will be mapped by the allocated folio.
+ */
+static struct folio *try_vma_alloc_movable_folio(struct vm_area_struct *vma,
+                               unsigned long vaddr, int order, bool zeroed)
+{

mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 3035

Quoted 10 lines
mm/memory.c +10 -0
+       struct folio *folio;
+
+       for (; order > 1; order--) {
+               folio = vma_alloc_movable_folio(vma, vaddr, order, zeroed);
+               if (folio)
+                       return folio;
+       }
+
+       return vma_alloc_movable_folio(vma, vaddr, 0, zeroed);
+}

I'd drop this patch. Instead, in do_anonymous_page():

if (IS_ENABLED(CONFIG_ARCH_WANTS_PTE_ORDER))
folio = vma_alloc_zeroed_movable_folio(vma, addr,
CONFIG_ARCH_WANTS_PTE_ORDER))

Quoted 5 lines
Quoted 2 lines
  if (!folio)
    folio = vma_alloc_zeroed_movable_folio(vma, addr, 0);

I meant a runtime function arch_wants_pte_order() (Its default implementation would return 0.)

There are a bunch of things which you are implying here which I'll try to make explicit:

I think you are implying that we shouldn't retry allocation with intermediate orders; but only try the order requested by the arch (arch_wants_pte_order()) and 0. Correct? For arm64 at least, I would like the VMA's THP hint to be a factor in determining the preferred order (see patches 8 and 9). So I would add a vma parameter to arch_wants_pte_order() to allow for this.

For the case where the THP hint is present, then the arch will request 2M (if the page size is 16K or 64K). If that fails to allocate, there is still value in allocating a 64K folio (which is order 2 in the 16K case). Without the retry with intermediate orders logic, we would not get this.

We can't just blindly allocate a folio of arch_wants_pte_order() size because it might overlap with existing populated PTEs, or cross the bounds of the VMA (or a number of other things - see calc_anon_folio_order_alloc() in patch 10). Are you implying that if there is any kind of issue like this, then we should go directly to order 0? I can kind of see the argument from a minimizing fragmentation perspective, but for best possible performance I think we are better off "packing the bin" with intermediate orders.

You're also implying that a runtime arch_wants_pte_order() function is better than the Kconfig stuff I did in patch 8. On reflection, I agree with you here. I think you mentioned that AMD supports coalescing 8 pages on some CPUs - so you would probably want runtime logic to determine if you are on an appropriate AMD CPU as part of the decision in that function?

The real reason for the existance of try_vma_alloc_movable_folio() is that I'm reusing it on the other fault paths (which are no longer part of this series). But I guess that's not a good reason to keep this until we get to those patches.

#15 lore raw

Re: [PATCH v1 03/10] mm: Introduce try_vma_alloc_movable_folio()

Quoted 91 lines · Ryan Roberts
Quoted 70 lines · Yu Zhao
Quoted 69 lines · Yu Zhao
Quoted 50 lines · Ryan Roberts

Opportunistically attempt to allocate high-order folios in highmem, optionally zeroed. Retry with lower orders all the way to order-0, until success. Although, of note, order-1 allocations are skipped since a large folio must be at least order-2 to work with the THP machinery. The user must check what they got with folio_order().

This will be used to oportunistically allocate large folios for anonymous memory with a sensible fallback under memory pressure.

For attempts to allocate non-0 orders, we set __GFP_NORETRY to prevent high latency due to reclaim, instead preferring to just try for a lower order. The same approach is used by the readahead code when allocating large folios.

---
 mm/memory.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
mm/memory.c +22 -0
index 367bbbb29d91..53896d46e686 100644
@@ -3001,6 +3001,39 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
        return 0;
 }
 
+static inline struct folio *vma_alloc_movable_folio(struct vm_area_struct *vma,
+                               unsigned long vaddr, int order, bool zeroed)
+{
+       gfp_t gfp = order > 0 ? __GFP_NORETRY | __GFP_NOWARN : 0;
+
+       if (zeroed)
+               return vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order);
+       else
+               return vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp, order, vma,
+                                                               vaddr, false);
+}
+
+/*
+ * Opportunistically attempt to allocate high-order folios, retrying with lower
+ * orders all the way to order-0, until success. order-1 allocations are skipped
+ * since a folio must be at least order-2 to work with the THP machinery. The
+ * user must check what they got with folio_order(). vaddr can be any virtual
+ * address that will be mapped by the allocated folio.
+ */
+static struct folio *try_vma_alloc_movable_folio(struct vm_area_struct *vma,
+                               unsigned long vaddr, int order, bool zeroed)
+{

mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 3035

Quoted 10 lines
mm/memory.c +10 -0
+       struct folio *folio;
+
+       for (; order > 1; order--) {
+               folio = vma_alloc_movable_folio(vma, vaddr, order, zeroed);
+               if (folio)
+                       return folio;
+       }
+
+       return vma_alloc_movable_folio(vma, vaddr, 0, zeroed);
+}

I'd drop this patch. Instead, in do_anonymous_page():

if (IS_ENABLED(CONFIG_ARCH_WANTS_PTE_ORDER))
folio = vma_alloc_zeroed_movable_folio(vma, addr,
CONFIG_ARCH_WANTS_PTE_ORDER))

Quoted 5 lines
Quoted 2 lines
  if (!folio)
    folio = vma_alloc_zeroed_movable_folio(vma, addr, 0);

I meant a runtime function arch_wants_pte_order() (Its default implementation would return 0.)

There are a bunch of things which you are implying here which I'll try to make explicit:

I think you are implying that we shouldn't retry allocation with intermediate orders; but only try the order requested by the arch (arch_wants_pte_order()) and 0. Correct? For arm64 at least, I would like the VMA's THP hint to be a factor in determining the preferred order (see patches 8 and 9). So I would add a vma parameter to arch_wants_pte_order() to allow for this.

For the case where the THP hint is present, then the arch will request 2M (if the page size is 16K or 64K). If that fails to allocate, there is still value in allocating a 64K folio (which is order 2 in the 16K case). Without the retry with intermediate orders logic, we would not get this.

Quoted 7 lines

We can't just blindly allocate a folio of arch_wants_pte_order() size because it might overlap with existing populated PTEs, or cross the bounds of the VMA (or a number of other things - see calc_anon_folio_order_alloc() in patch 10). Are you implying that if there is any kind of issue like this, then we should go directly to order 0? I can kind of see the argument from a minimizing fragmentation perspective, but for best possible performance I think we are better off "packing the bin" with intermediate orders.

One drawback of the retry is that it could introduce large tail latency (by memory zeroing, memory reclaiming or existing populated PTEs). That may not be appreciated by some applications. Thanks.

Regards
Yin, Fengwei

Quoted 10 lines

You're also implying that a runtime arch_wants_pte_order() function is better than the Kconfig stuff I did in patch 8. On reflection, I agree with you here. I think you mentioned that AMD supports coalescing 8 pages on some CPUs - so you would probably want runtime logic to determine if you are on an appropriate AMD CPU as part of the decision in that function?

The real reason for the existance of try_vma_alloc_movable_folio() is that I'm reusing it on the other fault paths (which are no longer part of this series). But I guess that's not a good reason to keep this until we get to those patches.

#16 lore raw

Re: [PATCH v1 03/10] mm: Introduce try_vma_alloc_movable_folio()

Quoted 102 lines · Yin Fengwei
Quoted 91 lines · Ryan Roberts
Quoted 70 lines · Yu Zhao
Quoted 69 lines · Yu Zhao
Quoted 50 lines · Ryan Roberts

Opportunistically attempt to allocate high-order folios in highmem, optionally zeroed. Retry with lower orders all the way to order-0, until success. Although, of note, order-1 allocations are skipped since a large folio must be at least order-2 to work with the THP machinery. The user must check what they got with folio_order().

This will be used to oportunistically allocate large folios for anonymous memory with a sensible fallback under memory pressure.

For attempts to allocate non-0 orders, we set __GFP_NORETRY to prevent high latency due to reclaim, instead preferring to just try for a lower order. The same approach is used by the readahead code when allocating large folios.

---
 mm/memory.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
mm/memory.c +22 -0
index 367bbbb29d91..53896d46e686 100644
@@ -3001,6 +3001,39 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)
        return 0;
 }
 
+static inline struct folio *vma_alloc_movable_folio(struct vm_area_struct *vma,
+                               unsigned long vaddr, int order, bool zeroed)
+{
+       gfp_t gfp = order > 0 ? __GFP_NORETRY | __GFP_NOWARN : 0;
+
+       if (zeroed)
+               return vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order);
+       else
+               return vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp, order, vma,
+                                                               vaddr, false);
+}
+
+/*
+ * Opportunistically attempt to allocate high-order folios, retrying with lower
+ * orders all the way to order-0, until success. order-1 allocations are skipped
+ * since a folio must be at least order-2 to work with the THP machinery. The
+ * user must check what they got with folio_order(). vaddr can be any virtual
+ * address that will be mapped by the allocated folio.
+ */
+static struct folio *try_vma_alloc_movable_folio(struct vm_area_struct *vma,
+                               unsigned long vaddr, int order, bool zeroed)
+{

mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 3035

Quoted 10 lines
mm/memory.c +10 -0
+       struct folio *folio;
+
+       for (; order > 1; order--) {
+               folio = vma_alloc_movable_folio(vma, vaddr, order, zeroed);
+               if (folio)
+                       return folio;
+       }
+
+       return vma_alloc_movable_folio(vma, vaddr, 0, zeroed);
+}

I'd drop this patch. Instead, in do_anonymous_page():

if (IS_ENABLED(CONFIG_ARCH_WANTS_PTE_ORDER))
folio = vma_alloc_zeroed_movable_folio(vma, addr,
CONFIG_ARCH_WANTS_PTE_ORDER))

Quoted 5 lines
Quoted 2 lines
  if (!folio)
    folio = vma_alloc_zeroed_movable_folio(vma, addr, 0);

I meant a runtime function arch_wants_pte_order() (Its default implementation would return 0.)

There are a bunch of things which you are implying here which I'll try to make explicit:

I think you are implying that we shouldn't retry allocation with intermediate orders; but only try the order requested by the arch (arch_wants_pte_order()) and 0. Correct? For arm64 at least, I would like the VMA's THP hint to be a factor in determining the preferred order (see patches 8 and 9). So I would add a vma parameter to arch_wants_pte_order() to allow for this.

For the case where the THP hint is present, then the arch will request 2M (if the page size is 16K or 64K). If that fails to allocate, there is still value in allocating a 64K folio (which is order 2 in the 16K case). Without the retry with intermediate orders logic, we would not get this.

Quoted 7 lines

We can't just blindly allocate a folio of arch_wants_pte_order() size because it might overlap with existing populated PTEs, or cross the bounds of the VMA (or a number of other things - see calc_anon_folio_order_alloc() in patch 10). Are you implying that if there is any kind of issue like this, then we should go directly to order 0? I can kind of see the argument from a minimizing fragmentation perspective, but for best possible performance I think we are better off "packing the bin" with intermediate orders.

Quoted 3 lines

One drawback of the retry is that it could introduce large tail latency (by memory zeroing, memory reclaiming or existing populated PTEs). That may not be appreciated by some applications. Thanks.

Good point. based on all the discussion, I think the conclusion is:

- ask the arch to for preferred folio order with runtime function
- check the folio will fit (racy) - if does not fit fall back to order-0
- allocate the folio
- take the ptl
- check the folio still fits (not racy) - if does not fit fall back to order-0

So in the worst case the latency will be allocating and zeroing a large folio, then allocating and zeroing an order-0 folio. Which is obviously better than iterating through every order from preferred to 0.

I'll work this flow into a v2.

Quoted 15 lines

Regards
Yin, Fengwei

Quoted 10 lines

You're also implying that a runtime arch_wants_pte_order() function is better than the Kconfig stuff I did in patch 8. On reflection, I agree with you here. I think you mentioned that AMD supports coalescing 8 pages on some CPUs - so you would probably want runtime logic to determine if you are on an appropriate AMD CPU as part of the decision in that function?

The real reason for the existance of try_vma_alloc_movable_folio() is that I'm reusing it on the other fault paths (which are no longer part of this series). But I guess that's not a good reason to keep this until we get to those patches.

#17 lore raw

PATCH 4/10 mm: Implement folio_add_new_anon_rmap_range()

In reply to Ryan Roberts

Like folio_add_new_anon_rmap() but batch-rmaps a range of pages belonging to a folio, for effciency savings. All pages are accounted as small pages.

---
 include/linux/rmap.h |  2 ++
 mm/rmap.c            | 43 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)
include/linux/rmap.h +2 -0
index a3825ce81102..15433a3d0cbf 100644
@@ -196,6 +196,8 @@ void page_add_new_anon_rmap(struct page *, struct vm_area_struct *,
 		unsigned long address);
 void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *,
 		unsigned long address);
+void folio_add_new_anon_rmap_range(struct folio *folio, struct page *page,
+		int nr, struct vm_area_struct *vma, unsigned long address);
 void page_add_file_rmap(struct page *, struct vm_area_struct *,
 		bool compound);
 void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr,
mm/rmap.c +43 -0

49 lines not on this page · raw message

Signature · 1 line
2.25.1
#18 lore raw

Re: [PATCH v1 04/10] mm: Implement folio_add_new_anon_rmap_range()

Quoted 16 lines · On Mon, Jun 26, 2023 at 11:14 AM Ryan Roberts <[email protected]> wrote:

Like folio_add_new_anon_rmap() but batch-rmaps a range of pages belonging to a folio, for effciency savings. All pages are accounted as small pages.

---
 include/linux/rmap.h |  2 ++
 mm/rmap.c            | 43 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)
include/linux/rmap.h +0 -0
index a3825ce81102..15433a3d0cbf 100644

include/linux/rmap.h void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, · line 200

Quoted 5 lines
include/linux/rmap.h +2 -0
@@ -196,6 +196,8 @@ void page_add_new_anon_rmap(struct page *, struct vm_area_struct *,
                unsigned long address);
 void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *,
                unsigned long address);
+void folio_add_new_anon_rmap_range(struct folio *folio, struct page *page,
+               int nr, struct vm_area_struct *vma, unsigned long address);

We should update folio_add_new_anon_rmap() to support large() && !folio_test_pmd_mappable() folios instead.

I double checked all places currently using folio_add_new_anon_rmap(), and as expected, none actually allocates large() && !folio_test_pmd_mappable() and maps it one by one, which makes the cases simpler, i.e.,
if (!large())

    // the existing basepage case
  else if (!folio_test_pmd_mappable())
    // our new case
  else
    // the existing THP case
Quoted 25 lines
include/linux/rmap.h +0 -0
 void page_add_file_rmap(struct page *, struct vm_area_struct *,
                bool compound);
 void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr,
mm/rmap.c +14 -0
index 1d8369549424..4050bcea7ae7 100644
@@ -1305,6 +1305,49 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
        __page_set_anon_rmap(folio, &folio->page, vma, address, 1);
 }
 
+/**
+ * folio_add_new_anon_rmap_range - Add mapping to a set of pages within a new
+ * anonymous potentially large folio.
+ * @folio:      The folio containing the pages to be mapped
+ * @page:       First page in the folio to be mapped
+ * @nr:         Number of pages to be mapped
+ * @vma:        the vm area in which the mapping is added
+ * @address:    the user virtual address of the first page to be mapped
+ *
+ * Like folio_add_new_anon_rmap() but batch-maps a range of pages within a folio
+ * using non-THP accounting. Like folio_add_new_anon_rmap(), the inc-and-test is
+ * bypassed and the folio does not have to be locked. All pages in the folio are
+ * individually accounted.
+ *

mm/rmap.c void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, · line 1331

Quoted 10 lines
mm/rmap.c +10 -0
+ * As the folio is new, it's assumed to be mapped exclusively by a single
+ * process.
+ */
+void folio_add_new_anon_rmap_range(struct folio *folio, struct page *page,
+               int nr, struct vm_area_struct *vma, unsigned long address)
+{
+       int i;
+
+       VM_BUG_ON_VMA(address < vma->vm_start ||
+                     address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

BTW, VM_BUG_ON* shouldn't be used in new code:
Documentation/process/coding-style.rst

#19 lore raw

Re: [PATCH v1 04/10] mm: Implement folio_add_new_anon_rmap_range()

Quoted 31 lines · Yu Zhao
Quoted 16 lines · On Mon, Jun 26, 2023 at 11:14 AM Ryan Roberts <[email protected]> wrote:

Like folio_add_new_anon_rmap() but batch-rmaps a range of pages belonging to a folio, for effciency savings. All pages are accounted as small pages.

---
 include/linux/rmap.h |  2 ++
 mm/rmap.c            | 43 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)
include/linux/rmap.h +0 -0
index a3825ce81102..15433a3d0cbf 100644

include/linux/rmap.h void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, · line 200

Quoted 5 lines
include/linux/rmap.h +2 -0
@@ -196,6 +196,8 @@ void page_add_new_anon_rmap(struct page *, struct vm_area_struct *,
                unsigned long address);
 void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *,
                unsigned long address);
+void folio_add_new_anon_rmap_range(struct folio *folio, struct page *page,
+               int nr, struct vm_area_struct *vma, unsigned long address);

We should update folio_add_new_anon_rmap() to support large() && !folio_test_pmd_mappable() folios instead.

I double checked all places currently using folio_add_new_anon_rmap(), and as expected, none actually allocates large() && !folio_test_pmd_mappable() and maps it one by one, which makes the cases simpler, i.e.,
if (!large())

Quoted 5 lines
    // the existing basepage case
  else if (!folio_test_pmd_mappable())
    // our new case
  else
    // the existing THP case

I don't have a strong opinion either way. Happy to go with this suggestion. But the reason I did it as a new function was because I was following the pattern in
[1] which adds a new folio_add_file_rmap_range() function.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 31 lines
Quoted 31 lines
include/linux/rmap.h +0 -0
 void page_add_file_rmap(struct page *, struct vm_area_struct *,
                bool compound);
 void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr,
mm/rmap.c +19 -0
index 1d8369549424..4050bcea7ae7 100644
@@ -1305,6 +1305,49 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
        __page_set_anon_rmap(folio, &folio->page, vma, address, 1);
 }
 
+/**
+ * folio_add_new_anon_rmap_range - Add mapping to a set of pages within a new
+ * anonymous potentially large folio.
+ * @folio:      The folio containing the pages to be mapped
+ * @page:       First page in the folio to be mapped
+ * @nr:         Number of pages to be mapped
+ * @vma:        the vm area in which the mapping is added
+ * @address:    the user virtual address of the first page to be mapped
+ *
+ * Like folio_add_new_anon_rmap() but batch-maps a range of pages within a folio
+ * using non-THP accounting. Like folio_add_new_anon_rmap(), the inc-and-test is
+ * bypassed and the folio does not have to be locked. All pages in the folio are
+ * individually accounted.
+ *
+ * As the folio is new, it's assumed to be mapped exclusively by a single
+ * process.
+ */
+void folio_add_new_anon_rmap_range(struct folio *folio, struct page *page,
+               int nr, struct vm_area_struct *vma, unsigned long address)
Quoted 8 lines

mm/rmap.c void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, · line 1331

Quoted 5 lines
mm/rmap.c +5 -0
+{
+       int i;
+
+       VM_BUG_ON_VMA(address < vma->vm_start ||
+                     address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

BTW, VM_BUG_ON* shouldn't be used in new code:
Documentation/process/coding-style.rst

Thanks, sorry about that. Was copy-pasting from folio_add_new_anon_rmap().

#20 lore raw

Re: [PATCH v1 04/10] mm: Implement folio_add_new_anon_rmap_range()

Quoted 38 lines · Ryan Roberts
Quoted 31 lines · Yu Zhao
Quoted 16 lines · On Mon, Jun 26, 2023 at 11:14 AM Ryan Roberts <[email protected]> wrote:

Like folio_add_new_anon_rmap() but batch-rmaps a range of pages belonging to a folio, for effciency savings. All pages are accounted as small pages.

---
 include/linux/rmap.h |  2 ++
 mm/rmap.c            | 43 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)
include/linux/rmap.h +0 -0
index a3825ce81102..15433a3d0cbf 100644

include/linux/rmap.h void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, · line 200

Quoted 5 lines
include/linux/rmap.h +2 -0
@@ -196,6 +196,8 @@ void page_add_new_anon_rmap(struct page *, struct vm_area_struct *,
                unsigned long address);
 void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *,
                unsigned long address);
+void folio_add_new_anon_rmap_range(struct folio *folio, struct page *page,
+               int nr, struct vm_area_struct *vma, unsigned long address);

We should update folio_add_new_anon_rmap() to support large() && !folio_test_pmd_mappable() folios instead.

I double checked all places currently using folio_add_new_anon_rmap(), and as expected, none actually allocates large() && !folio_test_pmd_mappable() and maps it one by one, which makes the cases simpler, i.e.,
if (!large())

Quoted 5 lines
    // the existing basepage case
  else if (!folio_test_pmd_mappable())
    // our new case
  else
    // the existing THP case
Quoted 5 lines

I don't have a strong opinion either way. Happy to go with this suggestion. But the reason I did it as a new function was because I was following the pattern in
[1] which adds a new folio_add_file_rmap_range() function.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Oh. There is different here:
For page cache, large folio could be created by previous file access. But later file access by other process just need map partial large folio. In this case, we need _range for filemap.

But for anonymous, I suppose we always map whole folio in. So I agree with Yu. We don't need _range for folio_add_new_anon_rmap(). Thanks.

Regards
Yin, Fengwei

Quoted 44 lines
Quoted 33 lines
Quoted 33 lines
include/linux/rmap.h +0 -0
 void page_add_file_rmap(struct page *, struct vm_area_struct *,
                bool compound);
 void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr,
mm/rmap.c +19 -0
index 1d8369549424..4050bcea7ae7 100644
@@ -1305,6 +1305,49 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
        __page_set_anon_rmap(folio, &folio->page, vma, address, 1);
 }
 
+/**
+ * folio_add_new_anon_rmap_range - Add mapping to a set of pages within a new
+ * anonymous potentially large folio.
+ * @folio:      The folio containing the pages to be mapped
+ * @page:       First page in the folio to be mapped
+ * @nr:         Number of pages to be mapped
+ * @vma:        the vm area in which the mapping is added
+ * @address:    the user virtual address of the first page to be mapped
+ *
+ * Like folio_add_new_anon_rmap() but batch-maps a range of pages within a folio
+ * using non-THP accounting. Like folio_add_new_anon_rmap(), the inc-and-test is
+ * bypassed and the folio does not have to be locked. All pages in the folio are
+ * individually accounted.
+ *
+ * As the folio is new, it's assumed to be mapped exclusively by a single
+ * process.
+ */
+void folio_add_new_anon_rmap_range(struct folio *folio, struct page *page,
+               int nr, struct vm_area_struct *vma, unsigned long address)
Quoted 8 lines

mm/rmap.c void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, · line 1331

Quoted 5 lines
mm/rmap.c +5 -0
+{
+       int i;
+
+       VM_BUG_ON_VMA(address < vma->vm_start ||
+                     address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

BTW, VM_BUG_ON* shouldn't be used in new code:
Documentation/process/coding-style.rst

Thanks, sorry about that. Was copy-pasting from folio_add_new_anon_rmap().

#21 lore raw

Re: [PATCH v1 04/10] mm: Implement folio_add_new_anon_rmap_range()

Quoted 45 lines · Yin Fengwei
Quoted 42 lines · Ryan Roberts
Quoted 31 lines · Yu Zhao
Quoted 16 lines · On Mon, Jun 26, 2023 at 11:14 AM Ryan Roberts <[email protected]> wrote:

Like folio_add_new_anon_rmap() but batch-rmaps a range of pages belonging to a folio, for effciency savings. All pages are accounted as small pages.

---
 include/linux/rmap.h |  2 ++
 mm/rmap.c            | 43 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)
include/linux/rmap.h +0 -0
index a3825ce81102..15433a3d0cbf 100644

include/linux/rmap.h void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, · line 200

Quoted 5 lines
include/linux/rmap.h +2 -0
@@ -196,6 +196,8 @@ void page_add_new_anon_rmap(struct page *, struct vm_area_struct *,
                unsigned long address);
 void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *,
                unsigned long address);
+void folio_add_new_anon_rmap_range(struct folio *folio, struct page *page,
+               int nr, struct vm_area_struct *vma, unsigned long address);

We should update folio_add_new_anon_rmap() to support large() && !folio_test_pmd_mappable() folios instead.

I double checked all places currently using folio_add_new_anon_rmap(), and as expected, none actually allocates large() && !folio_test_pmd_mappable() and maps it one by one, which makes the cases simpler, i.e.,
if (!large())

Quoted 5 lines
    // the existing basepage case
  else if (!folio_test_pmd_mappable())
    // our new case
  else
    // the existing THP case

I don't have a strong opinion either way. Happy to go with this suggestion. But the reason I did it as a new function was because I was following the pattern in
[1] which adds a new folio_add_file_rmap_range() function.

Quoted 8 lines
Quoted 1 line

[1] https://lore.kernel.org/linux-mm/[email protected]/

Oh. There is different here:
For page cache, large folio could be created by previous file access. But later file access by other process just need map partial large folio. In this case, we need _range for filemap.

But for anonymous, I suppose we always map whole folio in. So I agree with Yu. We don't need _range for folio_add_new_anon_rmap(). Thanks.

Yes that makes sense - thanks. I'll merge the new case into folio_add_new_anon_rmap() for v2.

Quoted 49 lines

Regards
Yin, Fengwei

Quoted 44 lines
Quoted 33 lines
Quoted 33 lines
include/linux/rmap.h +0 -0
 void page_add_file_rmap(struct page *, struct vm_area_struct *,
                bool compound);
 void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr,
mm/rmap.c +19 -0
index 1d8369549424..4050bcea7ae7 100644
@@ -1305,6 +1305,49 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
        __page_set_anon_rmap(folio, &folio->page, vma, address, 1);
 }
 
+/**
+ * folio_add_new_anon_rmap_range - Add mapping to a set of pages within a new
+ * anonymous potentially large folio.
+ * @folio:      The folio containing the pages to be mapped
+ * @page:       First page in the folio to be mapped
+ * @nr:         Number of pages to be mapped
+ * @vma:        the vm area in which the mapping is added
+ * @address:    the user virtual address of the first page to be mapped
+ *
+ * Like folio_add_new_anon_rmap() but batch-maps a range of pages within a folio
+ * using non-THP accounting. Like folio_add_new_anon_rmap(), the inc-and-test is
+ * bypassed and the folio does not have to be locked. All pages in the folio are
+ * individually accounted.
+ *
+ * As the folio is new, it's assumed to be mapped exclusively by a single
+ * process.
+ */
+void folio_add_new_anon_rmap_range(struct folio *folio, struct page *page,
+               int nr, struct vm_area_struct *vma, unsigned long address)
Quoted 8 lines

mm/rmap.c void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, · line 1331

Quoted 5 lines
mm/rmap.c +5 -0
+{
+       int i;
+
+       VM_BUG_ON_VMA(address < vma->vm_start ||
+                     address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

BTW, VM_BUG_ON* shouldn't be used in new code:
Documentation/process/coding-style.rst

Thanks, sorry about that. Was copy-pasting from folio_add_new_anon_rmap().

#22 lore raw

Re: [PATCH v1 04/10] mm: Implement folio_add_new_anon_rmap_range()

In reply to Yu Zhao
Quoted 31 lines · Yu Zhao
Quoted 16 lines · On Mon, Jun 26, 2023 at 11:14 AM Ryan Roberts <[email protected]> wrote:

Like folio_add_new_anon_rmap() but batch-rmaps a range of pages belonging to a folio, for effciency savings. All pages are accounted as small pages.

---
 include/linux/rmap.h |  2 ++
 mm/rmap.c            | 43 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)
include/linux/rmap.h +0 -0
index a3825ce81102..15433a3d0cbf 100644

include/linux/rmap.h void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, · line 200

Quoted 5 lines
include/linux/rmap.h +2 -0
@@ -196,6 +196,8 @@ void page_add_new_anon_rmap(struct page *, struct vm_area_struct *,
                unsigned long address);
 void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *,
                unsigned long address);
+void folio_add_new_anon_rmap_range(struct folio *folio, struct page *page,
+               int nr, struct vm_area_struct *vma, unsigned long address);

We should update folio_add_new_anon_rmap() to support large() && !folio_test_pmd_mappable() folios instead.

I double checked all places currently using folio_add_new_anon_rmap(), and as expected, none actually allocates large() && !folio_test_pmd_mappable() and maps it one by one, which makes the cases simpler, i.e.,
if (!large())

Quoted 5 lines
    // the existing basepage case
  else if (!folio_test_pmd_mappable())
    // our new case
  else
    // the existing THP case

I suppose we can merge the new case and existing THP case.

Regards
Yin, Fengwei

Quoted 39 lines
Quoted 26 lines
include/linux/rmap.h +0 -0
 void page_add_file_rmap(struct page *, struct vm_area_struct *,
                bool compound);
 void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr,
mm/rmap.c +14 -0
index 1d8369549424..4050bcea7ae7 100644
@@ -1305,6 +1305,49 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
        __page_set_anon_rmap(folio, &folio->page, vma, address, 1);
 }
 
+/**
+ * folio_add_new_anon_rmap_range - Add mapping to a set of pages within a new
+ * anonymous potentially large folio.
+ * @folio:      The folio containing the pages to be mapped
+ * @page:       First page in the folio to be mapped
+ * @nr:         Number of pages to be mapped
+ * @vma:        the vm area in which the mapping is added
+ * @address:    the user virtual address of the first page to be mapped
+ *
+ * Like folio_add_new_anon_rmap() but batch-maps a range of pages within a folio
+ * using non-THP accounting. Like folio_add_new_anon_rmap(), the inc-and-test is
+ * bypassed and the folio does not have to be locked. All pages in the folio are
+ * individually accounted.
+ *

mm/rmap.c void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, · line 1331

Quoted 10 lines
mm/rmap.c +10 -0
+ * As the folio is new, it's assumed to be mapped exclusively by a single
+ * process.
+ */
+void folio_add_new_anon_rmap_range(struct folio *folio, struct page *page,
+               int nr, struct vm_area_struct *vma, unsigned long address)
+{
+       int i;
+
+       VM_BUG_ON_VMA(address < vma->vm_start ||
+                     address + (nr << PAGE_SHIFT) > vma->vm_end, vma);

BTW, VM_BUG_ON* shouldn't be used in new code:
Documentation/process/coding-style.rst

#23 lore raw

PATCH 5/10 mm: Implement folio_remove_rmap_range()

In reply to Ryan Roberts

Like page_remove_rmap() but batch-removes the rmap for a range of pages belonging to a folio, for effciency savings. All pages are accounted as small pages.

---
 include/linux/rmap.h |  2 ++
 mm/rmap.c            | 62 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)
include/linux/rmap.h +2 -0
index 15433a3d0cbf..50f50e4cb0f8 100644
@@ -204,6 +204,8 @@ void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr,
 		struct vm_area_struct *, bool compound);
 void page_remove_rmap(struct page *, struct vm_area_struct *,
 		bool compound);
+void folio_remove_rmap_range(struct folio *folio, struct page *page,
+		int nr, struct vm_area_struct *vma);
 
 void hugepage_add_anon_rmap(struct page *, struct vm_area_struct *,
 		unsigned long address, rmap_t flags);
mm/rmap.c +62 -0

68 lines not on this page · raw message

Signature · 1 line
2.25.1
#24 lore raw

Re: [PATCH v1 05/10] mm: Implement folio_remove_rmap_range()

Quoted 10 lines · On Mon, Jun 26, 2023 at 11:14 AM Ryan Roberts <[email protected]> wrote:

Like page_remove_rmap() but batch-removes the rmap for a range of pages belonging to a folio, for effciency savings. All pages are accounted as small pages.

---
 include/linux/rmap.h |  2 ++
 mm/rmap.c            | 62 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)

Sorry for nagging: this can be included in a followup series.

#25 lore raw

PATCH 6/10 mm: Allow deferred splitting of arbitrary large anon folios

In reply to Ryan Roberts

With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.

---
 mm/rmap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/rmap.c +1 -1
index ac1d93d43f2b..3d11c5fb6090 100644
@@ -1567,7 +1567,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,
 		 * page of the folio is unmapped and at least one page
 		 * is still mapped.
 		 */
-		if (folio_test_pmd_mappable(folio) && folio_test_anon(folio))
+		if (folio_test_large(folio) && folio_test_anon(folio))
 			if (!compound || nr < nr_pmdmapped)
 				deferred_split_folio(folio);
 	}
Signature · 1 line
2.25.1
Reviewed-by on v1 6/10
#26 lore raw

Re: [PATCH v1 06/10] mm: Allow deferred splitting of arbitrary large anon folios

Quoted 8 lines · On Mon, Jun 26, 2023 at 11:15 AM Ryan Roberts <[email protected]> wrote:

With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.

Reviewed-by on v1 6/10
#27 lore raw

Re: [PATCH v1 06/10] mm: Allow deferred splitting of arbitrary large anon folios

In reply to Ryan Roberts
Quoted 7 lines · Ryan Roberts

With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.

Regards
Yin, Fengwei

Quoted 17 lines
---
 mm/rmap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/rmap.c +1 -1
index ac1d93d43f2b..3d11c5fb6090 100644
@@ -1567,7 +1567,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,
 		 * page of the folio is unmapped and at least one page
 		 * is still mapped.
 		 */
-		if (folio_test_pmd_mappable(folio) && folio_test_anon(folio))
+		if (folio_test_large(folio) && folio_test_anon(folio))
 			if (!compound || nr < nr_pmdmapped)
 				deferred_split_folio(folio);
 	}
#28 lore raw

PATCH 7/10 mm: Batch-zap large anonymous folio PTE mappings

In reply to Ryan Roberts

This allows batching the rmap removal with folio_remove_rmap_range(), which means we avoid spuriously adding a partially unmapped folio to the deferrred split queue in the common case, which reduces split queue lock contention.

Previously each page was removed from the rmap individually with page_remove_rmap(). If the first page belonged to a large folio, this would cause page_remove_rmap() to conclude that the folio was now partially mapped and add the folio to the deferred split queue. But subsequent calls would cause the folio to become fully unmapped, meaning there is no value to adding it to the split queue.

---
 mm/memory.c | 119 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 119 insertions(+)
mm/memory.c +119 -0
index 53896d46e686..9165ed1b9fc2 100644
@@ -914,6 +914,57 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma
 	return 0;
 }
 
+static inline unsigned long page_addr(struct page *page,
+				struct page *anchor, unsigned long anchor_addr)
+{
+	unsigned long offset;
+	unsigned long addr;
+
+	offset = (page_to_pfn(page) - page_to_pfn(anchor)) << PAGE_SHIFT;
+	addr = anchor_addr + offset;
+
+	if (anchor > page) {
+		if (addr > anchor_addr)
+			return 0;
+	} else {
+		if (addr < anchor_addr)
+			return ULONG_MAX;
+	}
+
+	return addr;
+}
+
+static int calc_anon_folio_map_pgcount(struct folio *folio,
+				       struct page *page, pte_t *pte,
+				       unsigned long addr, unsigned long end)
+{
+	pte_t ptent;
+	int floops;
+	int i;
+	unsigned long pfn;
+
+	end = min(page_addr(&folio->page + folio_nr_pages(folio), page, addr),
+		  end);
+	floops = (end - addr) >> PAGE_SHIFT;
+	pfn = page_to_pfn(page);
+	pfn++;
+	pte++;
+
+	for (i = 1; i < floops; i++) {
+		ptent = ptep_get(pte);
+
+		if (!pte_present(ptent) ||
+		    pte_pfn(ptent) != pfn) {
+			return i;
+		}
+
+		pfn++;
+		pte++;
+	}
+
+	return floops;
+}
+
 /*
  * Copy one pte.  Returns 0 if succeeded, or -EAGAIN if one preallocated page
  * is required to copy this pte.
@@ -1379,6 +1430,44 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma,
 	pte_install_uffd_wp_if_needed(vma, addr, pte, pteval);
 }
 
+static unsigned long zap_anon_pte_range(struct mmu_gather *tlb,
+					struct vm_area_struct *vma,
+					struct page *page, pte_t *pte,
+					unsigned long addr, unsigned long end,
+					bool *full_out)
+{
+	struct folio *folio = page_folio(page);
+	struct mm_struct *mm = tlb->mm;
+	pte_t ptent;
+	int pgcount;
+	int i;
+	bool full;
+
+	pgcount = calc_anon_folio_map_pgcount(folio, page, pte, addr, end);
+
+	for (i = 0; i < pgcount;) {
+		ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm);
+		tlb_remove_tlb_entry(tlb, pte, addr);
+		full = __tlb_remove_page(tlb, page, 0);
+
+		if (unlikely(page_mapcount(page) < 1))
+			print_bad_pte(vma, addr, ptent, page);
+
+		i++;
+		page++;
+		pte++;
+		addr += PAGE_SIZE;
+
+		if (unlikely(full))
+			break;
+	}
+
+	folio_remove_rmap_range(folio, page - i, i, vma);
+
+	*full_out = full;
+	return i;
+}
+
 static unsigned long zap_pte_range(struct mmu_gather *tlb,
 				struct vm_area_struct *vma, pmd_t *pmd,
 				unsigned long addr, unsigned long end,
@@ -1415,6 +1504,36 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 			page = vm_normal_page(vma, addr, ptent);
 			if (unlikely(!should_zap_page(details, page)))
 				continue;
+
+			/*
+			 * Batch zap large anonymous folio mappings. This allows
+			 * batching the rmap removal, which means we avoid
+			 * spuriously adding a partially unmapped folio to the
+			 * deferrred split queue in the common case, which
+			 * reduces split queue lock contention. Require the VMA
+			 * to be anonymous to ensure that none of the PTEs in
+			 * the range require zap_install_uffd_wp_if_needed().
+			 */
+			if (page && PageAnon(page) && vma_is_anonymous(vma)) {
+				bool full;
+				int pgcount;
+
+				pgcount = zap_anon_pte_range(tlb, vma,
+						page, pte, addr, end, &full);
+
+				rss[mm_counter(page)] -= pgcount;
+				pgcount--;
+				pte += pgcount;
+				addr += pgcount << PAGE_SHIFT;
+
+				if (unlikely(full)) {
+					force_flush = 1;
+					addr += PAGE_SIZE;
+					break;
+				}
+				continue;
+			}
+
 			ptent = ptep_get_and_clear_full(mm, addr, pte,
 							tlb->fullmm);
 			tlb_remove_tlb_entry(tlb, pte, addr);
Signature · 1 line
2.25.1
#29 lore raw

Re: [PATCH v1 07/10] mm: Batch-zap large anonymous folio PTE mappings

Quoted 13 lines · On Mon, Jun 26, 2023 at 11:15 AM Ryan Roberts <[email protected]> wrote:

This allows batching the rmap removal with folio_remove_rmap_range(), which means we avoid spuriously adding a partially unmapped folio to the deferrred split queue in the common case, which reduces split queue lock contention.

Previously each page was removed from the rmap individually with page_remove_rmap(). If the first page belonged to a large folio, this would cause page_remove_rmap() to conclude that the folio was now partially mapped and add the folio to the deferred split queue. But subsequent calls would cause the folio to become fully unmapped, meaning there is no value to adding it to the split queue.

Quoted 4 lines
---
 mm/memory.c | 119 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 119 insertions(+)

We don't really need this patch for the series to work. So again, I'd split it out.

#30 lore raw

Re: [PATCH v1 07/10] mm: Batch-zap large anonymous folio PTE mappings

Quoted 14 lines · Yu Zhao
Quoted 13 lines · On Mon, Jun 26, 2023 at 11:15 AM Ryan Roberts <[email protected]> wrote:

This allows batching the rmap removal with folio_remove_rmap_range(), which means we avoid spuriously adding a partially unmapped folio to the deferrred split queue in the common case, which reduces split queue lock contention.

Previously each page was removed from the rmap individually with page_remove_rmap(). If the first page belonged to a large folio, this would cause page_remove_rmap() to conclude that the folio was now partially mapped and add the folio to the deferred split queue. But subsequent calls would cause the folio to become fully unmapped, meaning there is no value to adding it to the split queue.

Quoted 7 lines
Quoted 4 lines
---
 mm/memory.c | 119 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 119 insertions(+)

We don't really need this patch for the series to work. So again, I'd split it out.

The reason I included it in the MVP was that without it I was seeing high lock contention for the split queue lock, which was significantly eating the performance gains. But since then Yin Fengwei's patch to make this more efficient has been accepted so perhaps that solves the problem and in that case we can drop this as you suggest. If I still see a reasonable perf improvement without it, I'll drop for v2.

#31 lore raw

PATCH 8/10 mm: Kconfig hooks to determine max anon folio allocation order

In reply to Ryan Roberts

For variable-order anonymous folios, we need to determine the order that we will allocate. From a SW perspective, the higher the order we allocate, the less overhead we will have; fewer faults, fewer folios in lists, etc. But of course there will also be more memory wastage as the order increases.

From a HW perspective, there are memory block sizes that can be beneficial to reducing TLB pressure. arm64, for example, has the ability to map "contpte" sized chunks (64K for a 4K base page, 2M for 16K and 64K base pages) such that one of these chunks only uses a single TLB entry.

So we let the architecture specify the order of the maximally beneficial mapping unit when PTE-mapped. Furthermore, because in some cases, this order may be quite big (and therefore potentially wasteful of memory), allow the arch to specify 2 values; One is the max order for a mapping that _would not_ use THP if all size and alignment constraints were met, and the other is the max order for a mapping that _would_ use THP if all those constraints were met.

Implement this with Kconfig by introducing some new options to allow the architecture to declare that it supports large anonymous folios along with these 2 preferred max order values. Then introduce a user-facing option, LARGE_ANON_FOLIO, which defaults to disabled and can only be enabled if the architecture has declared its support. When disabled, it forces the max order values, LARGE_ANON_FOLIO_NOTHP_ORDER_MAX and LARGE_ANON_FOLIO_THP_ORDER_MAX to 0, meaning only a single page is ever allocated.

---
 mm/Kconfig  | 39 +++++++++++++++++++++++++++++++++++++++
 mm/memory.c |  8 ++++++++
 2 files changed, 47 insertions(+)
mm/Kconfig +39 -0
index 7672a22647b4..f4ba48c37b75 100644
@@ -1208,4 +1208,43 @@ config PER_VMA_LOCK
 
 source "mm/damon/Kconfig"
 
+config ARCH_SUPPORTS_LARGE_ANON_FOLIO
+	def_bool n
+	help
+	  An arch should select this symbol if wants to allow LARGE_ANON_FOLIO
+	  to be enabled. It must also set the following integer values:
+	  - ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+	  - ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+
+config ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+	int
+	help
+	  The maximum size of folio to allocate for an anonymous VMA PTE-mapping
+	  that does not have the MADV_HUGEPAGE hint set.
+
+config ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+	int
+	help
+	  The maximum size of folio to allocate for an anonymous VMA PTE-mapping
+	  that has the MADV_HUGEPAGE hint set.
+
+config LARGE_ANON_FOLIO
+	bool "Allocate large folios for anonymous memory"
+	depends on ARCH_SUPPORTS_LARGE_ANON_FOLIO
+	default n
+	help
+	  Use large (bigger than order-0) folios to back anonymous memory where
+	  possible. This reduces the number of page faults, as well as other
+	  per-page overheads to improve performance for many workloads.
+
+config LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+	int
+	default 0 if !LARGE_ANON_FOLIO
+	default ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+
+config LARGE_ANON_FOLIO_THP_ORDER_MAX
+	int
+	default 0 if !LARGE_ANON_FOLIO
+	default ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+
 endmenu
mm/memory.c +8 -0
index 9165ed1b9fc2..a8f7e2b28d7a 100644
@@ -3153,6 +3153,14 @@ static struct folio *try_vma_alloc_movable_folio(struct vm_area_struct *vma,
 	return vma_alloc_movable_folio(vma, vaddr, 0, zeroed);
 }
 
+static inline int max_anon_folio_order(struct vm_area_struct *vma)
+{
+	if (hugepage_vma_check(vma, vma->vm_flags, false, true, true))
+		return CONFIG_LARGE_ANON_FOLIO_THP_ORDER_MAX;
+	else
+		return CONFIG_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX;
+}
+
 /*
  * Handle write page faults for pages that can be reused in the current vma
  *
Signature · 1 line
2.25.1
#32 lore raw

Re: [PATCH v1 08/10] mm: Kconfig hooks to determine max anon folio allocation order

Quoted 74 lines · On Mon, Jun 26, 2023 at 11:15 AM Ryan Roberts <[email protected]> wrote:

For variable-order anonymous folios, we need to determine the order that we will allocate. From a SW perspective, the higher the order we allocate, the less overhead we will have; fewer faults, fewer folios in lists, etc. But of course there will also be more memory wastage as the order increases.

From a HW perspective, there are memory block sizes that can be beneficial to reducing TLB pressure. arm64, for example, has the ability to map "contpte" sized chunks (64K for a 4K base page, 2M for 16K and 64K base pages) such that one of these chunks only uses a single TLB entry.

So we let the architecture specify the order of the maximally beneficial mapping unit when PTE-mapped. Furthermore, because in some cases, this order may be quite big (and therefore potentially wasteful of memory), allow the arch to specify 2 values; One is the max order for a mapping that _would not_ use THP if all size and alignment constraints were met, and the other is the max order for a mapping that _would_ use THP if all those constraints were met.

Implement this with Kconfig by introducing some new options to allow the architecture to declare that it supports large anonymous folios along with these 2 preferred max order values. Then introduce a user-facing option, LARGE_ANON_FOLIO, which defaults to disabled and can only be enabled if the architecture has declared its support. When disabled, it forces the max order values, LARGE_ANON_FOLIO_NOTHP_ORDER_MAX and LARGE_ANON_FOLIO_THP_ORDER_MAX to 0, meaning only a single page is ever allocated.

---
 mm/Kconfig  | 39 +++++++++++++++++++++++++++++++++++++++
 mm/memory.c |  8 ++++++++
 2 files changed, 47 insertions(+)
mm/Kconfig +30 -0
index 7672a22647b4..f4ba48c37b75 100644
@@ -1208,4 +1208,43 @@ config PER_VMA_LOCK
 
 source "mm/damon/Kconfig"
 
+config ARCH_SUPPORTS_LARGE_ANON_FOLIO
+       def_bool n
+       help
+         An arch should select this symbol if wants to allow LARGE_ANON_FOLIO
+         to be enabled. It must also set the following integer values:
+         - ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+         - ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+
+config ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+       int
+       help
+         The maximum size of folio to allocate for an anonymous VMA PTE-mapping
+         that does not have the MADV_HUGEPAGE hint set.
+
+config ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+       int
+       help
+         The maximum size of folio to allocate for an anonymous VMA PTE-mapping
+         that has the MADV_HUGEPAGE hint set.
+
+config LARGE_ANON_FOLIO
+       bool "Allocate large folios for anonymous memory"
+       depends on ARCH_SUPPORTS_LARGE_ANON_FOLIO
+       default n
+       help
+         Use large (bigger than order-0) folios to back anonymous memory where
+         possible. This reduces the number of page faults, as well as other
+         per-page overheads to improve performance for many workloads.
+
+config LARGE_ANON_FOLIO_NOTHP_ORDER_MAX

mm/Kconfig config PER_VMA_LOCK · line 1250

Quoted 10 lines
mm/Kconfig +9 -0
+       int
+       default 0 if !LARGE_ANON_FOLIO
+       default ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+
+config LARGE_ANON_FOLIO_THP_ORDER_MAX
+       int
+       default 0 if !LARGE_ANON_FOLIO
+       default ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+
 endmenu

I don't think an MVP should add this many Kconfigs. One Kconfig sounds reasonable to me for now.

#33 lore raw

Re: [PATCH v1 08/10] mm: Kconfig hooks to determine max anon folio allocation order

Quoted 80 lines · Yu Zhao
Quoted 79 lines · On Mon, Jun 26, 2023 at 11:15 AM Ryan Roberts <[email protected]> wrote:

For variable-order anonymous folios, we need to determine the order that we will allocate. From a SW perspective, the higher the order we allocate, the less overhead we will have; fewer faults, fewer folios in lists, etc. But of course there will also be more memory wastage as the order increases.

From a HW perspective, there are memory block sizes that can be beneficial to reducing TLB pressure. arm64, for example, has the ability to map "contpte" sized chunks (64K for a 4K base page, 2M for 16K and 64K base pages) such that one of these chunks only uses a single TLB entry.

So we let the architecture specify the order of the maximally beneficial mapping unit when PTE-mapped. Furthermore, because in some cases, this order may be quite big (and therefore potentially wasteful of memory), allow the arch to specify 2 values; One is the max order for a mapping that _would not_ use THP if all size and alignment constraints were met, and the other is the max order for a mapping that _would_ use THP if all those constraints were met.

Implement this with Kconfig by introducing some new options to allow the architecture to declare that it supports large anonymous folios along with these 2 preferred max order values. Then introduce a user-facing option, LARGE_ANON_FOLIO, which defaults to disabled and can only be enabled if the architecture has declared its support. When disabled, it forces the max order values, LARGE_ANON_FOLIO_NOTHP_ORDER_MAX and LARGE_ANON_FOLIO_THP_ORDER_MAX to 0, meaning only a single page is ever allocated.

---
 mm/Kconfig  | 39 +++++++++++++++++++++++++++++++++++++++
 mm/memory.c |  8 ++++++++
 2 files changed, 47 insertions(+)
mm/Kconfig +35 -0
index 7672a22647b4..f4ba48c37b75 100644
@@ -1208,4 +1208,43 @@ config PER_VMA_LOCK
 
 source "mm/damon/Kconfig"
 
+config ARCH_SUPPORTS_LARGE_ANON_FOLIO
+       def_bool n
+       help
+         An arch should select this symbol if wants to allow LARGE_ANON_FOLIO
+         to be enabled. It must also set the following integer values:
+         - ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+         - ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+
+config ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+       int
+       help
+         The maximum size of folio to allocate for an anonymous VMA PTE-mapping
+         that does not have the MADV_HUGEPAGE hint set.
+
+config ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+       int
+       help
+         The maximum size of folio to allocate for an anonymous VMA PTE-mapping
+         that has the MADV_HUGEPAGE hint set.
+
+config LARGE_ANON_FOLIO
+       bool "Allocate large folios for anonymous memory"
+       depends on ARCH_SUPPORTS_LARGE_ANON_FOLIO
+       default n
+       help
+         Use large (bigger than order-0) folios to back anonymous memory where
+         possible. This reduces the number of page faults, as well as other
+         per-page overheads to improve performance for many workloads.
+
+config LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+       int
+       default 0 if !LARGE_ANON_FOLIO
+       default ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+
+config LARGE_ANON_FOLIO_THP_ORDER_MAX
Quoted 8 lines

mm/Kconfig config PER_VMA_LOCK · line 1250

Quoted 5 lines
mm/Kconfig +4 -0
+       int
+       default 0 if !LARGE_ANON_FOLIO
+       default ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+
 endmenu

I don't think an MVP should add this many Kconfigs. One Kconfig sounds reasonable to me for now.

If we move to arch_wants_pte_order() as you suggested (in your response to patch
3) then I agree we can remove most of these. I still think we might want 2 though. For an arch that does not implement arch_wants_pte_order() we wouldn't want LARGE_ANON_FOLIO to show up in menuconfig so we would still need ARCH_SUPPORTS_LARGE_ANON_FOLIO:

config ARCH_SUPPORTS_LARGE_ANON_FOLIO
def_bool n
help
An arch should select this symbol if wants to allow LARGE_ANON_FOLIO to be enabled. In this case, It must also define arch_wants_pte_order()

config LARGE_ANON_FOLIO
bool "Allocate large folios for anonymous memory"
depends on ARCH_SUPPORTS_LARGE_ANON_FOLIO
default n
help
Use large (bigger than order-0) folios to back anonymous memory where possible. This reduces the number of page faults, as well as other per-page overheads to improve performance for many workloads.

What do you think?

#34 lore raw

Re: [PATCH v1 08/10] mm: Kconfig hooks to determine max anon folio allocation order

In reply to Ryan Roberts
Quoted 73 lines · On Mon, Jun 26, 2023 at 10:15 AM Ryan Roberts <[email protected]> wrote:

For variable-order anonymous folios, we need to determine the order that we will allocate. From a SW perspective, the higher the order we allocate, the less overhead we will have; fewer faults, fewer folios in lists, etc. But of course there will also be more memory wastage as the order increases.

From a HW perspective, there are memory block sizes that can be beneficial to reducing TLB pressure. arm64, for example, has the ability to map "contpte" sized chunks (64K for a 4K base page, 2M for 16K and 64K base pages) such that one of these chunks only uses a single TLB entry.

So we let the architecture specify the order of the maximally beneficial mapping unit when PTE-mapped. Furthermore, because in some cases, this order may be quite big (and therefore potentially wasteful of memory), allow the arch to specify 2 values; One is the max order for a mapping that _would not_ use THP if all size and alignment constraints were met, and the other is the max order for a mapping that _would_ use THP if all those constraints were met.

Implement this with Kconfig by introducing some new options to allow the architecture to declare that it supports large anonymous folios along with these 2 preferred max order values. Then introduce a user-facing option, LARGE_ANON_FOLIO, which defaults to disabled and can only be enabled if the architecture has declared its support. When disabled, it forces the max order values, LARGE_ANON_FOLIO_NOTHP_ORDER_MAX and LARGE_ANON_FOLIO_THP_ORDER_MAX to 0, meaning only a single page is ever allocated.

---
 mm/Kconfig  | 39 +++++++++++++++++++++++++++++++++++++++
 mm/memory.c |  8 ++++++++
 2 files changed, 47 insertions(+)
mm/Kconfig +29 -0
index 7672a22647b4..f4ba48c37b75 100644
@@ -1208,4 +1208,43 @@ config PER_VMA_LOCK
 
 source "mm/damon/Kconfig"
 
+config ARCH_SUPPORTS_LARGE_ANON_FOLIO
+       def_bool n
+       help
+         An arch should select this symbol if wants to allow LARGE_ANON_FOLIO
+         to be enabled. It must also set the following integer values:
+         - ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+         - ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+
+config ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+       int
+       help
+         The maximum size of folio to allocate for an anonymous VMA PTE-mapping
+         that does not have the MADV_HUGEPAGE hint set.
+
+config ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+       int
+       help
+         The maximum size of folio to allocate for an anonymous VMA PTE-mapping
+         that has the MADV_HUGEPAGE hint set.
+
+config LARGE_ANON_FOLIO
+       bool "Allocate large folios for anonymous memory"
+       depends on ARCH_SUPPORTS_LARGE_ANON_FOLIO
+       default n
+       help
+         Use large (bigger than order-0) folios to back anonymous memory where
+         possible. This reduces the number of page faults, as well as other
+         per-page overheads to improve performance for many workloads.
+

mm/Kconfig config PER_VMA_LOCK · line 1249

Quoted 10 lines
mm/Kconfig +10 -0
+config LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+       int
+       default 0 if !LARGE_ANON_FOLIO
+       default ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+
+config LARGE_ANON_FOLIO_THP_ORDER_MAX
+       int
+       default 0 if !LARGE_ANON_FOLIO
+       default ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+

IMHO I don't think we need all of the new kconfigs. Ideally the large anon folios could be supported by all arches, although some of them may not benefit from larger TLB entries due to lack of hardware support.t

For now with a minimum implementation, I think you could define a macro or a function that returns the hardware preferred order.

Quoted 24 lines
mm/Kconfig +0 -0
 endmenu
mm/memory.c +8 -1
index 9165ed1b9fc2..a8f7e2b28d7a 100644
@@ -3153,6 +3153,14 @@ static struct folio *try_vma_alloc_movable_folio(struct vm_area_struct *vma,
        return vma_alloc_movable_folio(vma, vaddr, 0, zeroed);
 }
 
+static inline int max_anon_folio_order(struct vm_area_struct *vma)
+{
+       if (hugepage_vma_check(vma, vma->vm_flags, false, true, true))
+               return CONFIG_LARGE_ANON_FOLIO_THP_ORDER_MAX;
+       else
+               return CONFIG_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX;
+}
+
 /*
  * Handle write page faults for pages that can be reused in the current vma
  *
--

2.25.1

#35 lore raw

Re: [PATCH v1 08/10] mm: Kconfig hooks to determine max anon folio allocation order

Quoted 90 lines · Yang Shi
Quoted 73 lines · On Mon, Jun 26, 2023 at 10:15 AM Ryan Roberts <[email protected]> wrote:

For variable-order anonymous folios, we need to determine the order that we will allocate. From a SW perspective, the higher the order we allocate, the less overhead we will have; fewer faults, fewer folios in lists, etc. But of course there will also be more memory wastage as the order increases.

From a HW perspective, there are memory block sizes that can be beneficial to reducing TLB pressure. arm64, for example, has the ability to map "contpte" sized chunks (64K for a 4K base page, 2M for 16K and 64K base pages) such that one of these chunks only uses a single TLB entry.

So we let the architecture specify the order of the maximally beneficial mapping unit when PTE-mapped. Furthermore, because in some cases, this order may be quite big (and therefore potentially wasteful of memory), allow the arch to specify 2 values; One is the max order for a mapping that _would not_ use THP if all size and alignment constraints were met, and the other is the max order for a mapping that _would_ use THP if all those constraints were met.

Implement this with Kconfig by introducing some new options to allow the architecture to declare that it supports large anonymous folios along with these 2 preferred max order values. Then introduce a user-facing option, LARGE_ANON_FOLIO, which defaults to disabled and can only be enabled if the architecture has declared its support. When disabled, it forces the max order values, LARGE_ANON_FOLIO_NOTHP_ORDER_MAX and LARGE_ANON_FOLIO_THP_ORDER_MAX to 0, meaning only a single page is ever allocated.

---
 mm/Kconfig  | 39 +++++++++++++++++++++++++++++++++++++++
 mm/memory.c |  8 ++++++++
 2 files changed, 47 insertions(+)
mm/Kconfig +29 -0
index 7672a22647b4..f4ba48c37b75 100644
@@ -1208,4 +1208,43 @@ config PER_VMA_LOCK
 
 source "mm/damon/Kconfig"
 
+config ARCH_SUPPORTS_LARGE_ANON_FOLIO
+       def_bool n
+       help
+         An arch should select this symbol if wants to allow LARGE_ANON_FOLIO
+         to be enabled. It must also set the following integer values:
+         - ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+         - ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+
+config ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+       int
+       help
+         The maximum size of folio to allocate for an anonymous VMA PTE-mapping
+         that does not have the MADV_HUGEPAGE hint set.
+
+config ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+       int
+       help
+         The maximum size of folio to allocate for an anonymous VMA PTE-mapping
+         that has the MADV_HUGEPAGE hint set.
+
+config LARGE_ANON_FOLIO
+       bool "Allocate large folios for anonymous memory"
+       depends on ARCH_SUPPORTS_LARGE_ANON_FOLIO
+       default n
+       help
+         Use large (bigger than order-0) folios to back anonymous memory where
+         possible. This reduces the number of page faults, as well as other
+         per-page overheads to improve performance for many workloads.
+

mm/Kconfig config PER_VMA_LOCK · line 1249

Quoted 10 lines
mm/Kconfig +10 -0
+config LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+       int
+       default 0 if !LARGE_ANON_FOLIO
+       default ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+
+config LARGE_ANON_FOLIO_THP_ORDER_MAX
+       int
+       default 0 if !LARGE_ANON_FOLIO
+       default ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+

IMHO I don't think we need all of the new kconfigs. Ideally the large anon folios could be supported by all arches, although some of them may not benefit from larger TLB entries due to lack of hardware support.t

Quoted 2 lines

For now with a minimum implementation, I think you could define a macro or a function that returns the hardware preferred order.

Thanks for the feedback - that aligns with what Yu Zhao suggested. I'm implementing it for v2.

Thanks,
Ryan

Quoted 25 lines
Quoted 25 lines
mm/Kconfig +0 -0
 endmenu
mm/memory.c +8 -1
index 9165ed1b9fc2..a8f7e2b28d7a 100644
@@ -3153,6 +3153,14 @@ static struct folio *try_vma_alloc_movable_folio(struct vm_area_struct *vma,
        return vma_alloc_movable_folio(vma, vaddr, 0, zeroed);
 }
 
+static inline int max_anon_folio_order(struct vm_area_struct *vma)
+{
+       if (hugepage_vma_check(vma, vma->vm_flags, false, true, true))
+               return CONFIG_LARGE_ANON_FOLIO_THP_ORDER_MAX;
+       else
+               return CONFIG_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX;
+}
+
 /*
  * Handle write page faults for pages that can be reused in the current vma
  *
--

2.25.1

#36 lore raw

PATCH 9/10 arm64: mm: Declare support for large anonymous folios

In reply to Ryan Roberts

For the unhinted case, when THP is not permitted for the vma, don't allow anything bigger than 64K. This means we don't waste too much memory. Additionally, for 4K pages this is the contpte size, and for 16K, this is (usually) the HPA size when the uarch feature is implemented. For the hinted case, when THP is permitted for the vma, allow the contpte size for all page size configurations; 64K for 4K, 2M for 16K and 2M for 64K.

---
 arch/arm64/Kconfig | 13 +++++++++++++
 1 file changed, 13 insertions(+)
arch/arm64/Kconfig +13 -0
index 343e1e1cae10..0e91b5bc8cd9 100644
@@ -243,6 +243,7 @@ config ARM64
 	select TRACE_IRQFLAGS_SUPPORT
 	select TRACE_IRQFLAGS_NMI_SUPPORT
 	select HAVE_SOFTIRQ_ON_OWN_STACK
+	select ARCH_SUPPORTS_LARGE_ANON_FOLIO
 	help
 	  ARM 64-bit (AArch64) Linux support.
 
@@ -281,6 +282,18 @@ config ARM64_CONT_PMD_SHIFT
 	default 5 if ARM64_16K_PAGES
 	default 4
 
+config ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+	int
+	default 0 if ARM64_64K_PAGES	# 64K (1 page)
+	default 2 if ARM64_16K_PAGES	# 64K (4 pages; benefits from HPA where HW supports it)
+	default 4 if ARM64_4K_PAGES	# 64K (16 pages; eligible for contpte-mapping)
+
+config ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+	int
+	default 5 if ARM64_64K_PAGES	# 2M  (32 page; eligible for contpte-mapping)
+	default 7 if ARM64_16K_PAGES	# 2M  (128 pages; eligible for contpte-mapping)
+	default 4 if ARM64_4K_PAGES	# 64K (16 pages; eligible for contpte-mapping)
+
 config ARCH_MMAP_RND_BITS_MIN
 	default 14 if ARM64_64K_PAGES
 	default 16 if ARM64_16K_PAGES
Signature · 1 line
2.25.1
#37 lore raw

Re: [PATCH v1 09/10] arm64: mm: Declare support for large anonymous folios

Quoted 35 lines · On Mon, Jun 26, 2023 at 11:15 AM Ryan Roberts <[email protected]> wrote:

For the unhinted case, when THP is not permitted for the vma, don't allow anything bigger than 64K. This means we don't waste too much memory. Additionally, for 4K pages this is the contpte size, and for 16K, this is (usually) the HPA size when the uarch feature is implemented. For the hinted case, when THP is permitted for the vma, allow the contpte size for all page size configurations; 64K for 4K, 2M for 16K and 2M for 64K.

---
 arch/arm64/Kconfig | 13 +++++++++++++
 1 file changed, 13 insertions(+)
arch/arm64/Kconfig +6 -0
index 343e1e1cae10..0e91b5bc8cd9 100644
@@ -243,6 +243,7 @@ config ARM64
        select TRACE_IRQFLAGS_SUPPORT
        select TRACE_IRQFLAGS_NMI_SUPPORT
        select HAVE_SOFTIRQ_ON_OWN_STACK
+       select ARCH_SUPPORTS_LARGE_ANON_FOLIO
        help
          ARM 64-bit (AArch64) Linux support.
 
@@ -281,6 +282,18 @@ config ARM64_CONT_PMD_SHIFT
        default 5 if ARM64_16K_PAGES
        default 4
 
+config ARCH_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX
+       int
+       default 0 if ARM64_64K_PAGES    # 64K (1 page)
+       default 2 if ARM64_16K_PAGES    # 64K (4 pages; benefits from HPA where HW supports it)
+       default 4 if ARM64_4K_PAGES     # 64K (16 pages; eligible for contpte-mapping)

arch/arm64/Kconfig config ARM64_CONT_PMD_SHIFT · line 299

Quoted 10 lines
arch/arm64/Kconfig +7 -0
+
+config ARCH_LARGE_ANON_FOLIO_THP_ORDER_MAX
+       int
+       default 5 if ARM64_64K_PAGES    # 2M  (32 page; eligible for contpte-mapping)
+       default 7 if ARM64_16K_PAGES    # 2M  (128 pages; eligible for contpte-mapping)
+       default 4 if ARM64_4K_PAGES     # 64K (16 pages; eligible for contpte-mapping)
+
 config ARCH_MMAP_RND_BITS_MIN
        default 14 if ARM64_64K_PAGES
        default 16 if ARM64_16K_PAGES

Can we please just add one Kconfig for the large anon folio feature, i.e., ARCH_WANTS_PTE_ORDER, for now?

Feel free to add as many as you wish for arm specific features like HPA and contpte.

#38 lore raw

PATCH 10/10 mm: Allocate large folios for anonymous memory

In reply to Ryan Roberts

With all of the enabler patches in place, modify the anonymous memory write allocation path so that it opportunistically attempts to allocate a large folio up to max_anon_folio_order() size (This value is ultimately configured by the architecture). This reduces the number of page faults, reduces the size of (e.g. LRU) lists, and generally improves performance by batching what were per-page operations into per-(large)-folio operations.

If CONFIG_LARGE_ANON_FOLIO is not enabled (the default) then max_anon_folio_order() always returns 0, meaning we get the existing allocation behaviour.

---
 mm/memory.c | 159 +++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 144 insertions(+), 15 deletions(-)
mm/memory.c +144 -15
index a8f7e2b28d7a..d23c44cc5092 100644
@@ -3161,6 +3161,90 @@ static inline int max_anon_folio_order(struct vm_area_struct *vma)
 		return CONFIG_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX;
 }
 
+/*
+ * Returns index of first pte that is not none, or nr if all are none.
+ */
+static inline int check_ptes_none(pte_t *pte, int nr)
+{
+	int i;
+
+	for (i = 0; i < nr; i++) {
+		if (!pte_none(ptep_get(pte++)))
+			return i;
+	}
+
+	return nr;
+}
+
+static int calc_anon_folio_order_alloc(struct vm_fault *vmf, int order)
+{
+	/*
+	 * The aim here is to determine what size of folio we should allocate
+	 * for this fault. Factors include:
+	 * - Order must not be higher than `order` upon entry
+	 * - Folio must be naturally aligned within VA space
+	 * - Folio must not breach boundaries of vma
+	 * - Folio must be fully contained inside one pmd entry
+	 * - Folio must not overlap any non-none ptes
+	 *
+	 * Additionally, we do not allow order-1 since this breaks assumptions
+	 * elsewhere in the mm; THP pages must be at least order-2 (since they
+	 * store state up to the 3rd struct page subpage), and these pages must
+	 * be THP in order to correctly use pre-existing THP infrastructure such
+	 * as folio_split().
+	 *
+	 * As a consequence of relying on the THP infrastructure, if the system
+	 * does not support THP, we always fallback to order-0.
+	 *
+	 * Note that the caller may or may not choose to lock the pte. If
+	 * unlocked, the calculation should be considered an estimate that will
+	 * need to be validated under the lock.
+	 */
+
+	struct vm_area_struct *vma = vmf->vma;
+	int nr;
+	unsigned long addr;
+	pte_t *pte;
+	pte_t *first_set = NULL;
+	int ret;
+
+	if (has_transparent_hugepage()) {
+		order = min(order, PMD_SHIFT - PAGE_SHIFT);
+
+		for (; order > 1; order--) {
+			nr = 1 << order;
+			addr = ALIGN_DOWN(vmf->address, nr << PAGE_SHIFT);
+			pte = vmf->pte - ((vmf->address - addr) >> PAGE_SHIFT);
+
+			/* Check vma bounds. */
+			if (addr < vma->vm_start ||
+			    addr + (nr << PAGE_SHIFT) > vma->vm_end)
+				continue;
+
+			/* Ptes covered by order already known to be none. */
+			if (pte + nr <= first_set)
+				break;
+
+			/* Already found set pte in range covered by order. */
+			if (pte <= first_set)
+				continue;
+
+			/* Need to check if all the ptes are none. */
+			ret = check_ptes_none(pte, nr);
+			if (ret == nr)
+				break;
+
+			first_set = pte + ret;
+		}
+
+		if (order == 1)
+			order = 0;
+	} else
+		order = 0;
+
+	return order;
+}
+
 /*
  * Handle write page faults for pages that can be reused in the current vma
  *
@@ -4201,6 +4285,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	struct folio *folio;
 	vm_fault_t ret = 0;
 	pte_t entry;
+	unsigned long addr;
+	int order = uffd_wp ? 0 : max_anon_folio_order(vma);
+	int pgcount = BIT(order);
 
 	/* File mapping without ->vm_ops ? */
 	if (vma->vm_flags & VM_SHARED)
@@ -4242,24 +4329,44 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 			pte_unmap_unlock(vmf->pte, vmf->ptl);
 			return handle_userfault(vmf, VM_UFFD_MISSING);
 		}
-		goto setpte;
+		if (uffd_wp)
+			entry = pte_mkuffd_wp(entry);
+		set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
+
+		/* No need to invalidate - it was non-present before */
+		update_mmu_cache(vma, vmf->address, vmf->pte);
+		goto unlock;
 	}
 
-	/* Allocate our own private page. */
+retry:
+	/*
+	 * Estimate the folio order to allocate. We are not under the ptl here
+	 * so this estiamte needs to be re-checked later once we have the lock.
+	 */
+	vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
+	order = calc_anon_folio_order_alloc(vmf, order);
+	pte_unmap(vmf->pte);
+
+	/* Allocate our own private folio. */
 	if (unlikely(anon_vma_prepare(vma)))
 		goto oom;
-	folio = vma_alloc_zeroed_movable_folio(vma, vmf->address, 0, 0);
+	folio = try_vma_alloc_movable_folio(vma, vmf->address, order, true);
 	if (!folio)
 		goto oom;
 
+	/* We may have been granted less than we asked for. */
+	order = folio_order(folio);
+	pgcount = BIT(order);
+	addr = ALIGN_DOWN(vmf->address, pgcount << PAGE_SHIFT);
+
 	if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL))
 		goto oom_free_page;
 	folio_throttle_swaprate(folio, GFP_KERNEL);
 
 	/*
 	 * The memory barrier inside __folio_mark_uptodate makes sure that
-	 * preceding stores to the page contents become visible before
-	 * the set_pte_at() write.
+	 * preceding stores to the folio contents become visible before
+	 * the set_ptes() write.
 	 */
 	__folio_mark_uptodate(folio);
 
@@ -4268,11 +4375,31 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	if (vma->vm_flags & VM_WRITE)
 		entry = pte_mkwrite(pte_mkdirty(entry));
 
-	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
-			&vmf->ptl);
-	if (vmf_pte_changed(vmf)) {
-		update_mmu_tlb(vma, vmf->address, vmf->pte);
-		goto release;
+	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl);
+
+	/*
+	 * Ensure our estimate above is still correct; we could have raced with
+	 * another thread to service a fault in the region.
+	 */
+	if (order == 0) {
+		if (vmf_pte_changed(vmf)) {
+			update_mmu_tlb(vma, vmf->address, vmf->pte);
+			goto release;
+		}
+	} else if (check_ptes_none(vmf->pte, pgcount) != pgcount) {
+		pte_t *pte = vmf->pte + ((vmf->address - addr) >> PAGE_SHIFT);
+
+		/* If faulting pte was allocated by another, exit early. */
+		if (!pte_none(ptep_get(pte))) {
+			update_mmu_tlb(vma, vmf->address, pte);
+			goto release;
+		}
+
+		/* Else try again, with a lower order. */
+		pte_unmap_unlock(vmf->pte, vmf->ptl);
+		folio_put(folio);
+		order--;
+		goto retry;
 	}
 
 	ret = check_stable_address_space(vma->vm_mm);
@@ -4286,16 +4413,18 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 		return handle_userfault(vmf, VM_UFFD_MISSING);
 	}
 
-	inc_mm_counter(vma->vm_mm, MM_ANONPAGES);
-	folio_add_new_anon_rmap(folio, vma, vmf->address);
+	folio_ref_add(folio, pgcount - 1);
+
+	add_mm_counter(vma->vm_mm, MM_ANONPAGES, pgcount);
+	folio_add_new_anon_rmap_range(folio, &folio->page, pgcount, vma, addr);
 	folio_add_lru_vma(folio, vma);
-setpte:
+
 	if (uffd_wp)
 		entry = pte_mkuffd_wp(entry);
-	set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
+	set_ptes(vma->vm_mm, addr, vmf->pte, entry, pgcount);
 
 	/* No need to invalidate - it was non-present before */
-	update_mmu_cache(vma, vmf->address, vmf->pte);
+	update_mmu_cache_range(vma, addr, vmf->pte, pgcount);
 unlock:
 	pte_unmap_unlock(vmf->pte, vmf->ptl);
 	return ret;
Signature · 1 line
2.25.1
#39 lore raw

Re: [PATCH v1 10/10] mm: Allocate large folios for anonymous memory

Quoted 32 lines · On Mon, Jun 26, 2023 at 11:15 AM Ryan Roberts <[email protected]> wrote:

With all of the enabler patches in place, modify the anonymous memory write allocation path so that it opportunistically attempts to allocate a large folio up to max_anon_folio_order() size (This value is ultimately configured by the architecture). This reduces the number of page faults, reduces the size of (e.g. LRU) lists, and generally improves performance by batching what were per-page operations into per-(large)-folio operations.

If CONFIG_LARGE_ANON_FOLIO is not enabled (the default) then max_anon_folio_order() always returns 0, meaning we get the existing allocation behaviour.

---
 mm/memory.c | 159 +++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 144 insertions(+), 15 deletions(-)
mm/memory.c +6 -0
index a8f7e2b28d7a..d23c44cc5092 100644
@@ -3161,6 +3161,90 @@ static inline int max_anon_folio_order(struct vm_area_struct *vma)
                return CONFIG_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX;
 }
 
+/*
+ * Returns index of first pte that is not none, or nr if all are none.
+ */
+static inline int check_ptes_none(pte_t *pte, int nr)
+{
+       int i;

mm/memory.c static inline int max_anon_folio_order(struct vm_area_struct *vma) · line 3179

Quoted 10 lines
mm/memory.c +10 -0
+
+       for (i = 0; i < nr; i++) {
+               if (!pte_none(ptep_get(pte++)))
+                       return i;
+       }
+
+       return nr;
+}
+
+static int calc_anon_folio_order_alloc(struct vm_fault *vmf, int order)

As suggested previously in 03/10, we can leave this for later.

#40 lore raw

Re: [PATCH v1 10/10] mm: Allocate large folios for anonymous memory

Quoted 37 lines · Yu Zhao
Quoted 36 lines · On Mon, Jun 26, 2023 at 11:15 AM Ryan Roberts <[email protected]> wrote:

With all of the enabler patches in place, modify the anonymous memory write allocation path so that it opportunistically attempts to allocate a large folio up to max_anon_folio_order() size (This value is ultimately configured by the architecture). This reduces the number of page faults, reduces the size of (e.g. LRU) lists, and generally improves performance by batching what were per-page operations into per-(large)-folio operations.

If CONFIG_LARGE_ANON_FOLIO is not enabled (the default) then max_anon_folio_order() always returns 0, meaning we get the existing allocation behaviour.

---
 mm/memory.c | 159 +++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 144 insertions(+), 15 deletions(-)
mm/memory.c +10 -0
index a8f7e2b28d7a..d23c44cc5092 100644
@@ -3161,6 +3161,90 @@ static inline int max_anon_folio_order(struct vm_area_struct *vma)
                return CONFIG_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX;
 }
 
+/*
+ * Returns index of first pte that is not none, or nr if all are none.
+ */
+static inline int check_ptes_none(pte_t *pte, int nr)
+{
+       int i;
+
+       for (i = 0; i < nr; i++) {
+               if (!pte_none(ptep_get(pte++)))
+                       return i;
Quoted 8 lines

mm/memory.c static inline int max_anon_folio_order(struct vm_area_struct *vma) · line 3179

Quoted 6 lines
mm/memory.c +6 -0
+       }
+
+       return nr;
+}
+
+static int calc_anon_folio_order_alloc(struct vm_fault *vmf, int order)

As suggested previously in 03/10, we can leave this for later.

I disagree. This is the logic that prevents us from accidentally replacing already set PTEs, or wandering out of the VMA bounds etc. How would you catch all those corener cases without this?

#41 lore raw

Re: [PATCH v1 10/10] mm: Allocate large folios for anonymous memory

Quoted 48 lines · Ryan Roberts
Quoted 37 lines · Yu Zhao
Quoted 36 lines · Ryan Roberts

With all of the enabler patches in place, modify the anonymous memory write allocation path so that it opportunistically attempts to allocate a large folio up to max_anon_folio_order() size (This value is ultimately configured by the architecture). This reduces the number of page faults, reduces the size of (e.g. LRU) lists, and generally improves performance by batching what were per-page operations into per-(large)-folio operations.

If CONFIG_LARGE_ANON_FOLIO is not enabled (the default) then max_anon_folio_order() always returns 0, meaning we get the existing allocation behaviour.

---
 mm/memory.c | 159 +++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 144 insertions(+), 15 deletions(-)
mm/memory.c +10 -0
index a8f7e2b28d7a..d23c44cc5092 100644
@@ -3161,6 +3161,90 @@ static inline int max_anon_folio_order(struct vm_area_struct *vma)
                return CONFIG_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX;
 }
 
+/*
+ * Returns index of first pte that is not none, or nr if all are none.
+ */
+static inline int check_ptes_none(pte_t *pte, int nr)
+{
+       int i;
+
+       for (i = 0; i < nr; i++) {
+               if (!pte_none(ptep_get(pte++)))
+                       return i;
Quoted 8 lines

mm/memory.c static inline int max_anon_folio_order(struct vm_area_struct *vma) · line 3179

Quoted 6 lines
mm/memory.c +6 -0
+       }
+
+       return nr;
+}
+
+static int calc_anon_folio_order_alloc(struct vm_fault *vmf, int order)

As suggested previously in 03/10, we can leave this for later.

Quoted 3 lines

I disagree. This is the logic that prevents us from accidentally replacing already set PTEs, or wandering out of the VMA bounds etc. How would you catch all those corener cases without this?

Again, sorry for not being clear previously: we definitely need to handle alignments & overlapps. But the fallback, i.e., "for (; order > 1; order--) {" in calc_anon_folio_order_alloc() is not necessary.

For now, we just need something like

  bool is_order_suitable() {
    // check whether it fits properly
  }

Later on, we could add

  alloc_anon_folio_best_effort()
  {
    for a list of fallback orders
      is_order_suitable()
  }
#42 lore raw

Re: [PATCH v1 10/10] mm: Allocate large folios for anonymous memory

In reply to Ryan Roberts
Quoted 197 lines · On Mon, Jun 26, 2023 at 10:15 AM Ryan Roberts <[email protected]> wrote:

With all of the enabler patches in place, modify the anonymous memory write allocation path so that it opportunistically attempts to allocate a large folio up to max_anon_folio_order() size (This value is ultimately configured by the architecture). This reduces the number of page faults, reduces the size of (e.g. LRU) lists, and generally improves performance by batching what were per-page operations into per-(large)-folio operations.

If CONFIG_LARGE_ANON_FOLIO is not enabled (the default) then max_anon_folio_order() always returns 0, meaning we get the existing allocation behaviour.

---
 mm/memory.c | 159 +++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 144 insertions(+), 15 deletions(-)
mm/memory.c +127 -10
index a8f7e2b28d7a..d23c44cc5092 100644
@@ -3161,6 +3161,90 @@ static inline int max_anon_folio_order(struct vm_area_struct *vma)
                return CONFIG_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX;
 }
 
+/*
+ * Returns index of first pte that is not none, or nr if all are none.
+ */
+static inline int check_ptes_none(pte_t *pte, int nr)
+{
+       int i;
+
+       for (i = 0; i < nr; i++) {
+               if (!pte_none(ptep_get(pte++)))
+                       return i;
+       }
+
+       return nr;
+}
+
+static int calc_anon_folio_order_alloc(struct vm_fault *vmf, int order)
+{
+       /*
+        * The aim here is to determine what size of folio we should allocate
+        * for this fault. Factors include:
+        * - Order must not be higher than `order` upon entry
+        * - Folio must be naturally aligned within VA space
+        * - Folio must not breach boundaries of vma
+        * - Folio must be fully contained inside one pmd entry
+        * - Folio must not overlap any non-none ptes
+        *
+        * Additionally, we do not allow order-1 since this breaks assumptions
+        * elsewhere in the mm; THP pages must be at least order-2 (since they
+        * store state up to the 3rd struct page subpage), and these pages must
+        * be THP in order to correctly use pre-existing THP infrastructure such
+        * as folio_split().
+        *
+        * As a consequence of relying on the THP infrastructure, if the system
+        * does not support THP, we always fallback to order-0.
+        *
+        * Note that the caller may or may not choose to lock the pte. If
+        * unlocked, the calculation should be considered an estimate that will
+        * need to be validated under the lock.
+        */
+
+       struct vm_area_struct *vma = vmf->vma;
+       int nr;
+       unsigned long addr;
+       pte_t *pte;
+       pte_t *first_set = NULL;
+       int ret;
+
+       if (has_transparent_hugepage()) {
+               order = min(order, PMD_SHIFT - PAGE_SHIFT);
+
+               for (; order > 1; order--) {
+                       nr = 1 << order;
+                       addr = ALIGN_DOWN(vmf->address, nr << PAGE_SHIFT);
+                       pte = vmf->pte - ((vmf->address - addr) >> PAGE_SHIFT);
+
+                       /* Check vma bounds. */
+                       if (addr < vma->vm_start ||
+                           addr + (nr << PAGE_SHIFT) > vma->vm_end)
+                               continue;
+
+                       /* Ptes covered by order already known to be none. */
+                       if (pte + nr <= first_set)
+                               break;
+
+                       /* Already found set pte in range covered by order. */
+                       if (pte <= first_set)
+                               continue;
+
+                       /* Need to check if all the ptes are none. */
+                       ret = check_ptes_none(pte, nr);
+                       if (ret == nr)
+                               break;
+
+                       first_set = pte + ret;
+               }
+
+               if (order == 1)
+                       order = 0;
+       } else
+               order = 0;
+
+       return order;
+}
+
 /*
  * Handle write page faults for pages that can be reused in the current vma
  *
@@ -4201,6 +4285,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
        struct folio *folio;
        vm_fault_t ret = 0;
        pte_t entry;
+       unsigned long addr;
+       int order = uffd_wp ? 0 : max_anon_folio_order(vma);
+       int pgcount = BIT(order);
 
        /* File mapping without ->vm_ops ? */
        if (vma->vm_flags & VM_SHARED)
@@ -4242,24 +4329,44 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
                        pte_unmap_unlock(vmf->pte, vmf->ptl);
                        return handle_userfault(vmf, VM_UFFD_MISSING);
                }
-               goto setpte;
+               if (uffd_wp)
+                       entry = pte_mkuffd_wp(entry);
+               set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
+
+               /* No need to invalidate - it was non-present before */
+               update_mmu_cache(vma, vmf->address, vmf->pte);
+               goto unlock;
        }
 
-       /* Allocate our own private page. */
+retry:
+       /*
+        * Estimate the folio order to allocate. We are not under the ptl here
+        * so this estiamte needs to be re-checked later once we have the lock.
+        */
+       vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
+       order = calc_anon_folio_order_alloc(vmf, order);
+       pte_unmap(vmf->pte);
+
+       /* Allocate our own private folio. */
        if (unlikely(anon_vma_prepare(vma)))
                goto oom;
-       folio = vma_alloc_zeroed_movable_folio(vma, vmf->address, 0, 0);
+       folio = try_vma_alloc_movable_folio(vma, vmf->address, order, true);
        if (!folio)
                goto oom;
 
+       /* We may have been granted less than we asked for. */
+       order = folio_order(folio);
+       pgcount = BIT(order);
+       addr = ALIGN_DOWN(vmf->address, pgcount << PAGE_SHIFT);
+
        if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL))
                goto oom_free_page;
        folio_throttle_swaprate(folio, GFP_KERNEL);
 
        /*
         * The memory barrier inside __folio_mark_uptodate makes sure that
-        * preceding stores to the page contents become visible before
-        * the set_pte_at() write.
+        * preceding stores to the folio contents become visible before
+        * the set_ptes() write.
         */
        __folio_mark_uptodate(folio);
 
@@ -4268,11 +4375,31 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
        if (vma->vm_flags & VM_WRITE)
                entry = pte_mkwrite(pte_mkdirty(entry));
 
-       vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
-                       &vmf->ptl);
-       if (vmf_pte_changed(vmf)) {
-               update_mmu_tlb(vma, vmf->address, vmf->pte);
-               goto release;
+       vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl);
+
+       /*
+        * Ensure our estimate above is still correct; we could have raced with
+        * another thread to service a fault in the region.
+        */
+       if (order == 0) {
+               if (vmf_pte_changed(vmf)) {
+                       update_mmu_tlb(vma, vmf->address, vmf->pte);
+                       goto release;
+               }
+       } else if (check_ptes_none(vmf->pte, pgcount) != pgcount) {
+               pte_t *pte = vmf->pte + ((vmf->address - addr) >> PAGE_SHIFT);
+
+               /* If faulting pte was allocated by another, exit early. */

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4402

Quoted 10 lines
mm/memory.c +10 -0
+               if (!pte_none(ptep_get(pte))) {
+                       update_mmu_tlb(vma, vmf->address, pte);
+                       goto release;
+               }
+
+               /* Else try again, with a lower order. */
+               pte_unmap_unlock(vmf->pte, vmf->ptl);
+               folio_put(folio);
+               order--;
+               goto retry;

I'm not sure whether this extra fallback logic is worth it or not. Do you have any benchmark data or is it just an arbitrary design choice? If it is just an arbitrary design choice, I'd like to go with the simplest way by just exiting page fault handler, just like the order-0, IMHO.

Quoted 31 lines
mm/memory.c +7 -6
        }
 
        ret = check_stable_address_space(vma->vm_mm);
@@ -4286,16 +4413,18 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
                return handle_userfault(vmf, VM_UFFD_MISSING);
        }
 
-       inc_mm_counter(vma->vm_mm, MM_ANONPAGES);
-       folio_add_new_anon_rmap(folio, vma, vmf->address);
+       folio_ref_add(folio, pgcount - 1);
+
+       add_mm_counter(vma->vm_mm, MM_ANONPAGES, pgcount);
+       folio_add_new_anon_rmap_range(folio, &folio->page, pgcount, vma, addr);
        folio_add_lru_vma(folio, vma);
-setpte:
+
        if (uffd_wp)
                entry = pte_mkuffd_wp(entry);
-       set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
+       set_ptes(vma->vm_mm, addr, vmf->pte, entry, pgcount);
 
        /* No need to invalidate - it was non-present before */
-       update_mmu_cache(vma, vmf->address, vmf->pte);
+       update_mmu_cache_range(vma, addr, vmf->pte, pgcount);
 unlock:
        pte_unmap_unlock(vmf->pte, vmf->ptl);
        return ret;
--

2.25.1

#43 lore raw

Re: [PATCH v1 10/10] mm: Allocate large folios for anonymous memory

Quoted 206 lines · Yang Shi
Quoted 205 lines · On Mon, Jun 26, 2023 at 10:15 AM Ryan Roberts <[email protected]> wrote:

With all of the enabler patches in place, modify the anonymous memory write allocation path so that it opportunistically attempts to allocate a large folio up to max_anon_folio_order() size (This value is ultimately configured by the architecture). This reduces the number of page faults, reduces the size of (e.g. LRU) lists, and generally improves performance by batching what were per-page operations into per-(large)-folio operations.

If CONFIG_LARGE_ANON_FOLIO is not enabled (the default) then max_anon_folio_order() always returns 0, meaning we get the existing allocation behaviour.

---
 mm/memory.c | 159 +++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 144 insertions(+), 15 deletions(-)
mm/memory.c +135 -10
index a8f7e2b28d7a..d23c44cc5092 100644
@@ -3161,6 +3161,90 @@ static inline int max_anon_folio_order(struct vm_area_struct *vma)
                return CONFIG_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX;
 }
 
+/*
+ * Returns index of first pte that is not none, or nr if all are none.
+ */
+static inline int check_ptes_none(pte_t *pte, int nr)
+{
+       int i;
+
+       for (i = 0; i < nr; i++) {
+               if (!pte_none(ptep_get(pte++)))
+                       return i;
+       }
+
+       return nr;
+}
+
+static int calc_anon_folio_order_alloc(struct vm_fault *vmf, int order)
+{
+       /*
+        * The aim here is to determine what size of folio we should allocate
+        * for this fault. Factors include:
+        * - Order must not be higher than `order` upon entry
+        * - Folio must be naturally aligned within VA space
+        * - Folio must not breach boundaries of vma
+        * - Folio must be fully contained inside one pmd entry
+        * - Folio must not overlap any non-none ptes
+        *
+        * Additionally, we do not allow order-1 since this breaks assumptions
+        * elsewhere in the mm; THP pages must be at least order-2 (since they
+        * store state up to the 3rd struct page subpage), and these pages must
+        * be THP in order to correctly use pre-existing THP infrastructure such
+        * as folio_split().
+        *
+        * As a consequence of relying on the THP infrastructure, if the system
+        * does not support THP, we always fallback to order-0.
+        *
+        * Note that the caller may or may not choose to lock the pte. If
+        * unlocked, the calculation should be considered an estimate that will
+        * need to be validated under the lock.
+        */
+
+       struct vm_area_struct *vma = vmf->vma;
+       int nr;
+       unsigned long addr;
+       pte_t *pte;
+       pte_t *first_set = NULL;
+       int ret;
+
+       if (has_transparent_hugepage()) {
+               order = min(order, PMD_SHIFT - PAGE_SHIFT);
+
+               for (; order > 1; order--) {
+                       nr = 1 << order;
+                       addr = ALIGN_DOWN(vmf->address, nr << PAGE_SHIFT);
+                       pte = vmf->pte - ((vmf->address - addr) >> PAGE_SHIFT);
+
+                       /* Check vma bounds. */
+                       if (addr < vma->vm_start ||
+                           addr + (nr << PAGE_SHIFT) > vma->vm_end)
+                               continue;
+
+                       /* Ptes covered by order already known to be none. */
+                       if (pte + nr <= first_set)
+                               break;
+
+                       /* Already found set pte in range covered by order. */
+                       if (pte <= first_set)
+                               continue;
+
+                       /* Need to check if all the ptes are none. */
+                       ret = check_ptes_none(pte, nr);
+                       if (ret == nr)
+                               break;
+
+                       first_set = pte + ret;
+               }
+
+               if (order == 1)
+                       order = 0;
+       } else
+               order = 0;
+
+       return order;
+}
+
 /*
  * Handle write page faults for pages that can be reused in the current vma
  *
@@ -4201,6 +4285,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
        struct folio *folio;
        vm_fault_t ret = 0;
        pte_t entry;
+       unsigned long addr;
+       int order = uffd_wp ? 0 : max_anon_folio_order(vma);
+       int pgcount = BIT(order);
 
        /* File mapping without ->vm_ops ? */
        if (vma->vm_flags & VM_SHARED)
@@ -4242,24 +4329,44 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
                        pte_unmap_unlock(vmf->pte, vmf->ptl);
                        return handle_userfault(vmf, VM_UFFD_MISSING);
                }
-               goto setpte;
+               if (uffd_wp)
+                       entry = pte_mkuffd_wp(entry);
+               set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
+
+               /* No need to invalidate - it was non-present before */
+               update_mmu_cache(vma, vmf->address, vmf->pte);
+               goto unlock;
        }
 
-       /* Allocate our own private page. */
+retry:
+       /*
+        * Estimate the folio order to allocate. We are not under the ptl here
+        * so this estiamte needs to be re-checked later once we have the lock.
+        */
+       vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
+       order = calc_anon_folio_order_alloc(vmf, order);
+       pte_unmap(vmf->pte);
+
+       /* Allocate our own private folio. */
        if (unlikely(anon_vma_prepare(vma)))
                goto oom;
-       folio = vma_alloc_zeroed_movable_folio(vma, vmf->address, 0, 0);
+       folio = try_vma_alloc_movable_folio(vma, vmf->address, order, true);
        if (!folio)
                goto oom;
 
+       /* We may have been granted less than we asked for. */
+       order = folio_order(folio);
+       pgcount = BIT(order);
+       addr = ALIGN_DOWN(vmf->address, pgcount << PAGE_SHIFT);
+
        if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL))
                goto oom_free_page;
        folio_throttle_swaprate(folio, GFP_KERNEL);
 
        /*
         * The memory barrier inside __folio_mark_uptodate makes sure that
-        * preceding stores to the page contents become visible before
-        * the set_pte_at() write.
+        * preceding stores to the folio contents become visible before
+        * the set_ptes() write.
         */
        __folio_mark_uptodate(folio);
 
@@ -4268,11 +4375,31 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
        if (vma->vm_flags & VM_WRITE)
                entry = pte_mkwrite(pte_mkdirty(entry));
 
-       vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
-                       &vmf->ptl);
-       if (vmf_pte_changed(vmf)) {
-               update_mmu_tlb(vma, vmf->address, vmf->pte);
-               goto release;
+       vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl);
+
+       /*
+        * Ensure our estimate above is still correct; we could have raced with
+        * another thread to service a fault in the region.
+        */
+       if (order == 0) {
+               if (vmf_pte_changed(vmf)) {
+                       update_mmu_tlb(vma, vmf->address, vmf->pte);
+                       goto release;
+               }
+       } else if (check_ptes_none(vmf->pte, pgcount) != pgcount) {
+               pte_t *pte = vmf->pte + ((vmf->address - addr) >> PAGE_SHIFT);
+
+               /* If faulting pte was allocated by another, exit early. */
+               if (!pte_none(ptep_get(pte))) {
+                       update_mmu_tlb(vma, vmf->address, pte);
+                       goto release;
+               }
+
+               /* Else try again, with a lower order. */
+               pte_unmap_unlock(vmf->pte, vmf->ptl);
+               folio_put(folio);
Quoted 8 lines

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4402

Quoted 2 lines
mm/memory.c +2 -0
+               order--;
+               goto retry;

I'm not sure whether this extra fallback logic is worth it or not. Do you have any benchmark data or is it just an arbitrary design choice? If it is just an arbitrary design choice, I'd like to go with the simplest way by just exiting page fault handler, just like the order-0, IMHO.

Yes, its an arbitrary design choice. Based on Yu Zhao's feedback, I'm already reworking this so that we only try the preferred order and order-0, so no longer iterating through intermediate orders.

I think what you are suggesting is that if attempting to allocate the preferred order and we find there was a race meaning that the folio now is overlapping populated ptes (but the faulting pte is still empty), just exit and rely on the page fault being re-triggered, rather than immediately falling back to order-0?

The reason I didn't do that was I wasn't sure if the return path might have assumptions that the faulting pte is now valid if no error was returned? I guess another option is to return VM_FAULT_RETRY but then it seemed cleaner to do the retry directly here. What do you suggest?

Thanks,
Ryan

Quoted 32 lines
Quoted 32 lines
mm/memory.c +7 -6
        }
 
        ret = check_stable_address_space(vma->vm_mm);
@@ -4286,16 +4413,18 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
                return handle_userfault(vmf, VM_UFFD_MISSING);
        }
 
-       inc_mm_counter(vma->vm_mm, MM_ANONPAGES);
-       folio_add_new_anon_rmap(folio, vma, vmf->address);
+       folio_ref_add(folio, pgcount - 1);
+
+       add_mm_counter(vma->vm_mm, MM_ANONPAGES, pgcount);
+       folio_add_new_anon_rmap_range(folio, &folio->page, pgcount, vma, addr);
        folio_add_lru_vma(folio, vma);
-setpte:
+
        if (uffd_wp)
                entry = pte_mkuffd_wp(entry);
-       set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
+       set_ptes(vma->vm_mm, addr, vmf->pte, entry, pgcount);
 
        /* No need to invalidate - it was non-present before */
-       update_mmu_cache(vma, vmf->address, vmf->pte);
+       update_mmu_cache_range(vma, addr, vmf->pte, pgcount);
 unlock:
        pte_unmap_unlock(vmf->pte, vmf->ptl);
        return ret;
--

2.25.1

#44 lore raw

Re: [PATCH v1 10/10] mm: Allocate large folios for anonymous memory

Quoted 217 lines · Ryan Roberts
Quoted 206 lines · Yang Shi
Quoted 205 lines · Ryan Roberts

With all of the enabler patches in place, modify the anonymous memory write allocation path so that it opportunistically attempts to allocate a large folio up to max_anon_folio_order() size (This value is ultimately configured by the architecture). This reduces the number of page faults, reduces the size of (e.g. LRU) lists, and generally improves performance by batching what were per-page operations into per-(large)-folio operations.

If CONFIG_LARGE_ANON_FOLIO is not enabled (the default) then max_anon_folio_order() always returns 0, meaning we get the existing allocation behaviour.

---
 mm/memory.c | 159 +++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 144 insertions(+), 15 deletions(-)
mm/memory.c +135 -10
index a8f7e2b28d7a..d23c44cc5092 100644
@@ -3161,6 +3161,90 @@ static inline int max_anon_folio_order(struct vm_area_struct *vma)
                return CONFIG_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX;
 }
 
+/*
+ * Returns index of first pte that is not none, or nr if all are none.
+ */
+static inline int check_ptes_none(pte_t *pte, int nr)
+{
+       int i;
+
+       for (i = 0; i < nr; i++) {
+               if (!pte_none(ptep_get(pte++)))
+                       return i;
+       }
+
+       return nr;
+}
+
+static int calc_anon_folio_order_alloc(struct vm_fault *vmf, int order)
+{
+       /*
+        * The aim here is to determine what size of folio we should allocate
+        * for this fault. Factors include:
+        * - Order must not be higher than `order` upon entry
+        * - Folio must be naturally aligned within VA space
+        * - Folio must not breach boundaries of vma
+        * - Folio must be fully contained inside one pmd entry
+        * - Folio must not overlap any non-none ptes
+        *
+        * Additionally, we do not allow order-1 since this breaks assumptions
+        * elsewhere in the mm; THP pages must be at least order-2 (since they
+        * store state up to the 3rd struct page subpage), and these pages must
+        * be THP in order to correctly use pre-existing THP infrastructure such
+        * as folio_split().
+        *
+        * As a consequence of relying on the THP infrastructure, if the system
+        * does not support THP, we always fallback to order-0.
+        *
+        * Note that the caller may or may not choose to lock the pte. If
+        * unlocked, the calculation should be considered an estimate that will
+        * need to be validated under the lock.
+        */
+
+       struct vm_area_struct *vma = vmf->vma;
+       int nr;
+       unsigned long addr;
+       pte_t *pte;
+       pte_t *first_set = NULL;
+       int ret;
+
+       if (has_transparent_hugepage()) {
+               order = min(order, PMD_SHIFT - PAGE_SHIFT);
+
+               for (; order > 1; order--) {
+                       nr = 1 << order;
+                       addr = ALIGN_DOWN(vmf->address, nr << PAGE_SHIFT);
+                       pte = vmf->pte - ((vmf->address - addr) >> PAGE_SHIFT);
+
+                       /* Check vma bounds. */
+                       if (addr < vma->vm_start ||
+                           addr + (nr << PAGE_SHIFT) > vma->vm_end)
+                               continue;
+
+                       /* Ptes covered by order already known to be none. */
+                       if (pte + nr <= first_set)
+                               break;
+
+                       /* Already found set pte in range covered by order. */
+                       if (pte <= first_set)
+                               continue;
+
+                       /* Need to check if all the ptes are none. */
+                       ret = check_ptes_none(pte, nr);
+                       if (ret == nr)
+                               break;
+
+                       first_set = pte + ret;
+               }
+
+               if (order == 1)
+                       order = 0;
+       } else
+               order = 0;
+
+       return order;
+}
+
 /*
  * Handle write page faults for pages that can be reused in the current vma
  *
@@ -4201,6 +4285,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
        struct folio *folio;
        vm_fault_t ret = 0;
        pte_t entry;
+       unsigned long addr;
+       int order = uffd_wp ? 0 : max_anon_folio_order(vma);
+       int pgcount = BIT(order);
 
        /* File mapping without ->vm_ops ? */
        if (vma->vm_flags & VM_SHARED)
@@ -4242,24 +4329,44 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
                        pte_unmap_unlock(vmf->pte, vmf->ptl);
                        return handle_userfault(vmf, VM_UFFD_MISSING);
                }
-               goto setpte;
+               if (uffd_wp)
+                       entry = pte_mkuffd_wp(entry);
+               set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
+
+               /* No need to invalidate - it was non-present before */
+               update_mmu_cache(vma, vmf->address, vmf->pte);
+               goto unlock;
        }
 
-       /* Allocate our own private page. */
+retry:
+       /*
+        * Estimate the folio order to allocate. We are not under the ptl here
+        * so this estiamte needs to be re-checked later once we have the lock.
+        */
+       vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
+       order = calc_anon_folio_order_alloc(vmf, order);
+       pte_unmap(vmf->pte);
+
+       /* Allocate our own private folio. */
        if (unlikely(anon_vma_prepare(vma)))
                goto oom;
-       folio = vma_alloc_zeroed_movable_folio(vma, vmf->address, 0, 0);
+       folio = try_vma_alloc_movable_folio(vma, vmf->address, order, true);
        if (!folio)
                goto oom;
 
+       /* We may have been granted less than we asked for. */
+       order = folio_order(folio);
+       pgcount = BIT(order);
+       addr = ALIGN_DOWN(vmf->address, pgcount << PAGE_SHIFT);
+
        if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL))
                goto oom_free_page;
        folio_throttle_swaprate(folio, GFP_KERNEL);
 
        /*
         * The memory barrier inside __folio_mark_uptodate makes sure that
-        * preceding stores to the page contents become visible before
-        * the set_pte_at() write.
+        * preceding stores to the folio contents become visible before
+        * the set_ptes() write.
         */
        __folio_mark_uptodate(folio);
 
@@ -4268,11 +4375,31 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
        if (vma->vm_flags & VM_WRITE)
                entry = pte_mkwrite(pte_mkdirty(entry));
 
-       vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
-                       &vmf->ptl);
-       if (vmf_pte_changed(vmf)) {
-               update_mmu_tlb(vma, vmf->address, vmf->pte);
-               goto release;
+       vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl);
+
+       /*
+        * Ensure our estimate above is still correct; we could have raced with
+        * another thread to service a fault in the region.
+        */
+       if (order == 0) {
+               if (vmf_pte_changed(vmf)) {
+                       update_mmu_tlb(vma, vmf->address, vmf->pte);
+                       goto release;
+               }
+       } else if (check_ptes_none(vmf->pte, pgcount) != pgcount) {
+               pte_t *pte = vmf->pte + ((vmf->address - addr) >> PAGE_SHIFT);
+
+               /* If faulting pte was allocated by another, exit early. */
+               if (!pte_none(ptep_get(pte))) {
+                       update_mmu_tlb(vma, vmf->address, pte);
+                       goto release;
+               }
+
+               /* Else try again, with a lower order. */
+               pte_unmap_unlock(vmf->pte, vmf->ptl);
+               folio_put(folio);
Quoted 8 lines

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4402

Quoted 2 lines
mm/memory.c +2 -0
+               order--;
+               goto retry;

I'm not sure whether this extra fallback logic is worth it or not. Do you have any benchmark data or is it just an arbitrary design choice? If it is just an arbitrary design choice, I'd like to go with the simplest way by just exiting page fault handler, just like the order-0, IMHO.

Quoted 8 lines

Yes, its an arbitrary design choice. Based on Yu Zhao's feedback, I'm already reworking this so that we only try the preferred order and order-0, so no longer iterating through intermediate orders.

I think what you are suggesting is that if attempting to allocate the preferred order and we find there was a race meaning that the folio now is overlapping populated ptes (but the faulting pte is still empty), just exit and rely on the page fault being re-triggered, rather than immediately falling back to order-0?

The faulting PTE might be filled too. Yes, just exit and rely on the CPU re-trigger page fault.

Quoted 5 lines

The reason I didn't do that was I wasn't sure if the return path might have assumptions that the faulting pte is now valid if no error was returned? I guess another option is to return VM_FAULT_RETRY but then it seemed cleaner to do the retry directly here. What do you suggest?

IIRC as long as the page fault handler doesn't return any error, it is safe to rely on CPU re-trigger page fault if PTE is not installed.

VM_FAULT_RETRY means the page fault handler released mmap_lock (or per-VMA lock with per-VMA lock enabled) due to waiting for page lock. TBH I really don't want to make that semantic more complicated and overloaded. And I don't see any fundamental difference between vmf_pte_changed() for order-0 folio and overlapping PTEs installed for large folio. So I'd like to follow the same behavior.

Quoted 39 lines

Thanks,
Ryan

Quoted 32 lines
Quoted 32 lines
mm/memory.c +7 -6
        }
 
        ret = check_stable_address_space(vma->vm_mm);
@@ -4286,16 +4413,18 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
                return handle_userfault(vmf, VM_UFFD_MISSING);
        }
 
-       inc_mm_counter(vma->vm_mm, MM_ANONPAGES);
-       folio_add_new_anon_rmap(folio, vma, vmf->address);
+       folio_ref_add(folio, pgcount - 1);
+
+       add_mm_counter(vma->vm_mm, MM_ANONPAGES, pgcount);
+       folio_add_new_anon_rmap_range(folio, &folio->page, pgcount, vma, addr);
        folio_add_lru_vma(folio, vma);
-setpte:
+
        if (uffd_wp)
                entry = pte_mkuffd_wp(entry);
-       set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
+       set_ptes(vma->vm_mm, addr, vmf->pte, entry, pgcount);
 
        /* No need to invalidate - it was non-present before */
-       update_mmu_cache(vma, vmf->address, vmf->pte);
+       update_mmu_cache_range(vma, addr, vmf->pte, pgcount);
 unlock:
        pte_unmap_unlock(vmf->pte, vmf->ptl);
        return ret;
--

2.25.1

#45 lore raw
In reply to Ryan Roberts
Quoted 61 lines · On Mon, Jun 26, 2023 at 11:14 AM Ryan Roberts <[email protected]> wrote:

Hi All,

Following on from the previous RFCv2 [1], this series implements variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults:

- Since SW (the kernel) is dealing with larger chunks of memory than base pages, there are efficiency savings to be had; fewer page faults, batched PTE and RMAP manipulation, fewer items on lists, etc. In short, we reduce kernel overhead. This should benefit all architectures.
- Since we are now mapping physically contiguous chunks of memory, we can take advantage of HW TLB compression techniques. A reduction in TLB pressure speeds up kernel and user space. arm64 systems have 2 mechanisms to coalesce TLB entries; "the contiguous bit" (architectural) and HPA (uarch).

This patch set deals with the SW side of things only and based on feedback from the RFC, aims to be the most minimal initial change, upon which future incremental changes can be added. For this reason, the new behaviour is hidden behind a new Kconfig switch, CONFIG_LARGE_ANON_FOLIO, which is disabled by default. Although the code has been refactored to parameterize the desired order of the allocation, when the feature is disabled (by forcing the order to be always 0) my performance tests measure no regression. So I'm hoping this will be a suitable mechanism to allow incremental submissions to the kernel without affecting the rest of the world.

The patches are based on top of v6.4 plus Matthew Wilcox's set_ptes() series [2], which is a hard dependency. I'm not sure of Matthew's exact plans for getting that series into the kernel, but I'm hoping we can start the review process on this patch set independently. I have a branch at [3].

I've posted a separate series concerning the HW part (contpte mapping) for arm64 at [4].

Performance
-----------

Below results show 2 benchmarks; kernel compilation and speedometer 2.0 (a javascript benchmark running in Chromium). Both cases are running on Ampere Altra with 1 NUMA node enabled, Ubuntu 22.04 and XFS filesystem. Each benchmark is repeated 15 times over 5 reboots and averaged.

All improvements are relative to baseline-4k. 'anonfolio-basic' is this series. 'anonfolio' is the full patch set similar to the RFC with the additional changes to the extra 3 fault paths. The rest of the configs are described at [4].

Kernel Compilation (smaller is better):

| kernel          |   real-time |   kern-time |   user-time |
|:----------------|------------:|------------:|------------:|
| baseline-4k     |        0.0% |        0.0% |        0.0% |
| anonfolio-basic |       -5.3% |      -42.9% |       -0.6% |
| anonfolio       |       -5.4% |      -46.0% |       -0.3% |
| contpte         |       -6.8% |      -45.7% |       -2.1% |
| exefolio        |       -8.4% |      -46.4% |       -3.7% |
| baseline-16k    |       -8.7% |      -49.2% |       -3.7% |
| baseline-64k    |      -10.5% |      -66.0% |       -3.5% |

Speedometer 2.0 (bigger is better):

Quoted 8 lines
| kernel          |   runs_per_min |
|:----------------|---------------:|
| baseline-4k     |           0.0% |
| anonfolio-basic |           0.7% |
| anonfolio       |           1.2% |
| contpte         |           3.1% |
| exefolio        |           4.2% |
| baseline-16k    |           5.3% |

Thanks for pushing this forward!

Quoted 4 lines

Changes since RFCv2
-------------------

- Simplified series to bare minimum (on David Hildenbrand's advice)

My impression is that this series still includes many pieces that can be split out and discussed separately with followup series.

(I skipped 04/10 and will look at it tomorrow.)

#46 lore raw
Quoted 77 lines · Yu Zhao
Quoted 61 lines · Ryan Roberts

Hi All,

Following on from the previous RFCv2 [1], this series implements variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults:

- Since SW (the kernel) is dealing with larger chunks of memory than base pages, there are efficiency savings to be had; fewer page faults, batched PTE and RMAP manipulation, fewer items on lists, etc. In short, we reduce kernel overhead. This should benefit all architectures.
- Since we are now mapping physically contiguous chunks of memory, we can take advantage of HW TLB compression techniques. A reduction in TLB pressure speeds up kernel and user space. arm64 systems have 2 mechanisms to coalesce TLB entries; "the contiguous bit" (architectural) and HPA (uarch).

This patch set deals with the SW side of things only and based on feedback from the RFC, aims to be the most minimal initial change, upon which future incremental changes can be added. For this reason, the new behaviour is hidden behind a new Kconfig switch, CONFIG_LARGE_ANON_FOLIO, which is disabled by default. Although the code has been refactored to parameterize the desired order of the allocation, when the feature is disabled (by forcing the order to be always 0) my performance tests measure no regression. So I'm hoping this will be a suitable mechanism to allow incremental submissions to the kernel without affecting the rest of the world.

The patches are based on top of v6.4 plus Matthew Wilcox's set_ptes() series [2], which is a hard dependency. I'm not sure of Matthew's exact plans for getting that series into the kernel, but I'm hoping we can start the review process on this patch set independently. I have a branch at [3].

I've posted a separate series concerning the HW part (contpte mapping) for arm64 at [4].

Performance
-----------

Below results show 2 benchmarks; kernel compilation and speedometer 2.0 (a javascript benchmark running in Chromium). Both cases are running on Ampere Altra with 1 NUMA node enabled, Ubuntu 22.04 and XFS filesystem. Each benchmark is repeated 15 times over 5 reboots and averaged.

All improvements are relative to baseline-4k. 'anonfolio-basic' is this series. 'anonfolio' is the full patch set similar to the RFC with the additional changes to the extra 3 fault paths. The rest of the configs are described at [4].

Kernel Compilation (smaller is better):

| kernel          |   real-time |   kern-time |   user-time |
|:----------------|------------:|------------:|------------:|
| baseline-4k     |        0.0% |        0.0% |        0.0% |
| anonfolio-basic |       -5.3% |      -42.9% |       -0.6% |
| anonfolio       |       -5.4% |      -46.0% |       -0.3% |
| contpte         |       -6.8% |      -45.7% |       -2.1% |
| exefolio        |       -8.4% |      -46.4% |       -3.7% |
| baseline-16k    |       -8.7% |      -49.2% |       -3.7% |
| baseline-64k    |      -10.5% |      -66.0% |       -3.5% |

Speedometer 2.0 (bigger is better):

Quoted 8 lines
| kernel          |   runs_per_min |
|:----------------|---------------:|
| baseline-4k     |           0.0% |
| anonfolio-basic |           0.7% |
| anonfolio       |           1.2% |
| contpte         |           3.1% |
| exefolio        |           4.2% |
| baseline-16k    |           5.3% |

Thanks for pushing this forward!

Quoted 3 lines

Changes since RFCv2
-------------------

Quoted 6 lines
Quoted 1 line

- Simplified series to bare minimum (on David Hildenbrand's advice)

My impression is that this series still includes many pieces that can be split out and discussed separately with followup series.

(I skipped 04/10 and will look at it tomorrow.)

I went through the series twice. Here what I think a bare minimum series (easier to review/debug/land) would look like:
1. a new arch specific function providing a prefered order within (0, PMD_ORDER).
2. an extended anon folio alloc API taking that order (02/10, partially).
3. an updated folio_add_new_anon_rmap() covering the large() && !pmd_mappable() case (similar to 04/10).
4. s/folio_test_pmd_mappable/folio_test_large/ in page_remove_rmap() (06/10, reviewed-by provided).
5. finally, use the extended anon folio alloc API with the arch preferred order in do_anonymous_page() (10/10, partially).

The rest can be split out into separate series and move forward in parallel with probably a long list of things we need/want to do.

#47 lore raw
Quoted 97 lines · Yu Zhao
Quoted 77 lines · Yu Zhao
Quoted 61 lines · Ryan Roberts

Hi All,

Following on from the previous RFCv2 [1], this series implements variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults:

- Since SW (the kernel) is dealing with larger chunks of memory than base pages, there are efficiency savings to be had; fewer page faults, batched PTE and RMAP manipulation, fewer items on lists, etc. In short, we reduce kernel overhead. This should benefit all architectures.
- Since we are now mapping physically contiguous chunks of memory, we can take advantage of HW TLB compression techniques. A reduction in TLB pressure speeds up kernel and user space. arm64 systems have 2 mechanisms to coalesce TLB entries; "the contiguous bit" (architectural) and HPA (uarch).

This patch set deals with the SW side of things only and based on feedback from the RFC, aims to be the most minimal initial change, upon which future incremental changes can be added. For this reason, the new behaviour is hidden behind a new Kconfig switch, CONFIG_LARGE_ANON_FOLIO, which is disabled by default. Although the code has been refactored to parameterize the desired order of the allocation, when the feature is disabled (by forcing the order to be always 0) my performance tests measure no regression. So I'm hoping this will be a suitable mechanism to allow incremental submissions to the kernel without affecting the rest of the world.

The patches are based on top of v6.4 plus Matthew Wilcox's set_ptes() series [2], which is a hard dependency. I'm not sure of Matthew's exact plans for getting that series into the kernel, but I'm hoping we can start the review process on this patch set independently. I have a branch at [3].

I've posted a separate series concerning the HW part (contpte mapping) for arm64 at [4].

Performance
-----------

Below results show 2 benchmarks; kernel compilation and speedometer 2.0 (a javascript benchmark running in Chromium). Both cases are running on Ampere Altra with 1 NUMA node enabled, Ubuntu 22.04 and XFS filesystem. Each benchmark is repeated 15 times over 5 reboots and averaged.

All improvements are relative to baseline-4k. 'anonfolio-basic' is this series. 'anonfolio' is the full patch set similar to the RFC with the additional changes to the extra 3 fault paths. The rest of the configs are described at [4].

Kernel Compilation (smaller is better):

| kernel          |   real-time |   kern-time |   user-time |
|:----------------|------------:|------------:|------------:|
| baseline-4k     |        0.0% |        0.0% |        0.0% |
| anonfolio-basic |       -5.3% |      -42.9% |       -0.6% |
| anonfolio       |       -5.4% |      -46.0% |       -0.3% |
| contpte         |       -6.8% |      -45.7% |       -2.1% |
| exefolio        |       -8.4% |      -46.4% |       -3.7% |
| baseline-16k    |       -8.7% |      -49.2% |       -3.7% |
| baseline-64k    |      -10.5% |      -66.0% |       -3.5% |

Speedometer 2.0 (bigger is better):

Quoted 8 lines
| kernel          |   runs_per_min |
|:----------------|---------------:|
| baseline-4k     |           0.0% |
| anonfolio-basic |           0.7% |
| anonfolio       |           1.2% |
| contpte         |           3.1% |
| exefolio        |           4.2% |
| baseline-16k    |           5.3% |

Thanks for pushing this forward!

Quoted 3 lines

Changes since RFCv2
-------------------

Quoted 6 lines
Quoted 1 line

- Simplified series to bare minimum (on David Hildenbrand's advice)

My impression is that this series still includes many pieces that can be split out and discussed separately with followup series.

(I skipped 04/10 and will look at it tomorrow.)

I went through the series twice. Here what I think a bare minimum series (easier to review/debug/land) would look like:
1. a new arch specific function providing a prefered order within (0, PMD_ORDER).
2. an extended anon folio alloc API taking that order (02/10, partially).
3. an updated folio_add_new_anon_rmap() covering the large() && !pmd_mappable() case (similar to 04/10).
4. s/folio_test_pmd_mappable/folio_test_large/ in page_remove_rmap() (06/10, reviewed-by provided).
5. finally, use the extended anon folio alloc API with the arch preferred order in do_anonymous_page() (10/10, partially).

Quoted 2 lines

The rest can be split out into separate series and move forward in parallel with probably a long list of things we need/want to do.

Thanks for the fadt review - I really appreciate it!

I've responded to many of your comments. I'd appreciate if we can close those points then I will work up a v2.

Thanks,
Ryan

#48 lore raw
Quoted 82 lines · Ryan Roberts
Quoted 80 lines · Yu Zhao
Quoted 79 lines · Yu Zhao
Quoted 61 lines · Ryan Roberts

Hi All,

Following on from the previous RFCv2 [1], this series implements variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults:

- Since SW (the kernel) is dealing with larger chunks of memory than base pages, there are efficiency savings to be had; fewer page faults, batched PTE and RMAP manipulation, fewer items on lists, etc. In short, we reduce kernel overhead. This should benefit all architectures.
- Since we are now mapping physically contiguous chunks of memory, we can take advantage of HW TLB compression techniques. A reduction in TLB pressure speeds up kernel and user space. arm64 systems have 2 mechanisms to coalesce TLB entries; "the contiguous bit" (architectural) and HPA (uarch).

This patch set deals with the SW side of things only and based on feedback from the RFC, aims to be the most minimal initial change, upon which future incremental changes can be added. For this reason, the new behaviour is hidden behind a new Kconfig switch, CONFIG_LARGE_ANON_FOLIO, which is disabled by default. Although the code has been refactored to parameterize the desired order of the allocation, when the feature is disabled (by forcing the order to be always 0) my performance tests measure no regression. So I'm hoping this will be a suitable mechanism to allow incremental submissions to the kernel without affecting the rest of the world.

The patches are based on top of v6.4 plus Matthew Wilcox's set_ptes() series [2], which is a hard dependency. I'm not sure of Matthew's exact plans for getting that series into the kernel, but I'm hoping we can start the review process on this patch set independently. I have a branch at [3].

I've posted a separate series concerning the HW part (contpte mapping) for arm64 at [4].

Performance
-----------

Below results show 2 benchmarks; kernel compilation and speedometer 2.0 (a javascript benchmark running in Chromium). Both cases are running on Ampere Altra with 1 NUMA node enabled, Ubuntu 22.04 and XFS filesystem. Each benchmark is repeated 15 times over 5 reboots and averaged.

All improvements are relative to baseline-4k. 'anonfolio-basic' is this series. 'anonfolio' is the full patch set similar to the RFC with the additional changes to the extra 3 fault paths. The rest of the configs are described at [4].

Kernel Compilation (smaller is better):

| kernel          |   real-time |   kern-time |   user-time |
|:----------------|------------:|------------:|------------:|
| baseline-4k     |        0.0% |        0.0% |        0.0% |
| anonfolio-basic |       -5.3% |      -42.9% |       -0.6% |
| anonfolio       |       -5.4% |      -46.0% |       -0.3% |
| contpte         |       -6.8% |      -45.7% |       -2.1% |
| exefolio        |       -8.4% |      -46.4% |       -3.7% |
| baseline-16k    |       -8.7% |      -49.2% |       -3.7% |
| baseline-64k    |      -10.5% |      -66.0% |       -3.5% |

Speedometer 2.0 (bigger is better):

Quoted 8 lines
| kernel          |   runs_per_min |
|:----------------|---------------:|
| baseline-4k     |           0.0% |
| anonfolio-basic |           0.7% |
| anonfolio       |           1.2% |
| contpte         |           3.1% |
| exefolio        |           4.2% |
| baseline-16k    |           5.3% |

Thanks for pushing this forward!

Quoted 4 lines

Changes since RFCv2
-------------------

- Simplified series to bare minimum (on David Hildenbrand's advice)

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

My impression is that this series still includes many pieces that can be split out and discussed separately with followup series.

(I skipped 04/10 and will look at it tomorrow.)

I went through the series twice. Here what I think a bare minimum series (easier to review/debug/land) would look like:

===

Quoted 9 lines
Quoted 9 lines

1. a new arch specific function providing a prefered order within (0, PMD_ORDER).
2. an extended anon folio alloc API taking that order (02/10, partially).
3. an updated folio_add_new_anon_rmap() covering the large() && !pmd_mappable() case (similar to 04/10).
4. s/folio_test_pmd_mappable/folio_test_large/ in page_remove_rmap() (06/10, reviewed-by provided).
5. finally, use the extended anon folio alloc API with the arch preferred order in do_anonymous_page() (10/10, partially).

===

Quoted 7 lines
Quoted 2 lines

The rest can be split out into separate series and move forward in parallel with probably a long list of things we need/want to do.

Thanks for the fadt review - I really appreciate it!

I've responded to many of your comments. I'd appreciate if we can close those points then I will work up a v2.

Thanks!

Based on the latest discussion here [1], my original list above can be optionally reduced to 4 patches: item 2 can be quashed into item 5.

Also please make sure we have only one global (apply to all archs) Kconfig option, and it should be added in item 5:

if TRANSPARENT_HUGEPAGE
config FLEXIBLE/VARIABLE_THP # or whatever name you see fit
end if

(How many new Kconfig options added within arch/arm64/ is not a concern of MM.)

And please make sure it's disabled by default, because we are still missing many important functions, e.g., I don't think we can mlock() when large() && !pmd_mappable(), see mlock_pte_range() and mlock_vma_folio(). We can fix it along with many things later, but we need to present a plan and a schedule now. Otherwise, there would be pushback if we try to land the series without supporting mlock().

Do you or Fengwei plan to take on it? (I personally don't.) If not, I'll try to find someone from our team to look at it. (It'd be more scalable if we have a coordinated group of people individually solving different problems.)

[1] https://lore.kernel.org/r/[email protected]/

#49 lore raw

Hi Yu,

Quoted 8 lines · Yu Zhao

And please make sure it's disabled by default, because we are still missing many important functions, e.g., I don't think we can mlock() when large() && !pmd_mappable(), see mlock_pte_range() and mlock_vma_folio(). We can fix it along with many things later, but we need to present a plan and a schedule now. Otherwise, there would be pushback if we try to land the series without supporting mlock().

Do you or Fengwei plan to take on it? (I personally don't.) If not,

Do you mean the mlock() with large folio? Yes. I can work on it. Thanks.

Regards
Yin, Fengwei

Quoted 3 lines

I'll try to find someone from our team to look at it. (It'd be more scalable if we have a coordinated group of people individually solving different problems.)

#50 lore raw
Quoted 11 lines · On Wed, Jun 28, 2023 at 5:59 PM Yin Fengwei <[email protected]> wrote:

Hi Yu,

Quoted 7 lines · Yu Zhao

And please make sure it's disabled by default, because we are still missing many important functions, e.g., I don't think we can mlock() when large() && !pmd_mappable(), see mlock_pte_range() and mlock_vma_folio(). We can fix it along with many things later, but we need to present a plan and a schedule now. Otherwise, there would be pushback if we try to land the series without supporting mlock().

Quoted 2 lines
Quoted 1 line

Do you or Fengwei plan to take on it? (I personally don't.) If not,

Do you mean the mlock() with large folio? Yes. I can work on it. Thanks.

Great. Thanks!

Other places that have the similar problem but are probably easier to fix than the mlock() case:
* madvise_cold_or_pageout_pte_range()
* shrink_folio_list()

#51 lore raw
Quoted 12 lines · Yu Zhao
Quoted 11 lines · On Wed, Jun 28, 2023 at 5:59 PM Yin Fengwei <[email protected]> wrote:

Hi Yu,

Quoted 7 lines · Yu Zhao

And please make sure it's disabled by default, because we are still missing many important functions, e.g., I don't think we can mlock() when large() && !pmd_mappable(), see mlock_pte_range() and mlock_vma_folio(). We can fix it along with many things later, but we need to present a plan and a schedule now. Otherwise, there would be pushback if we try to land the series without supporting mlock().

Quoted 8 lines
Quoted 2 lines
Quoted 1 line

Do you or Fengwei plan to take on it? (I personally don't.) If not,

Do you mean the mlock() with large folio? Yes. I can work on it. Thanks.

Great. Thanks!

Other places that have the similar problem but are probably easier to fix than the mlock() case:
* madvise_cold_or_pageout_pte_range()

This one was on my radar. :).

Regards
Yin, Fengwei

Quoted 1 line

* shrink_folio_list()

#52 lore raw
In reply to Yu Zhao
Quoted 118 lines · Yu Zhao
Quoted 82 lines · Ryan Roberts
Quoted 80 lines · Yu Zhao
Quoted 79 lines · Yu Zhao
Quoted 61 lines · Ryan Roberts

Hi All,

Following on from the previous RFCv2 [1], this series implements variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults:

- Since SW (the kernel) is dealing with larger chunks of memory than base pages, there are efficiency savings to be had; fewer page faults, batched PTE and RMAP manipulation, fewer items on lists, etc. In short, we reduce kernel overhead. This should benefit all architectures.
- Since we are now mapping physically contiguous chunks of memory, we can take advantage of HW TLB compression techniques. A reduction in TLB pressure speeds up kernel and user space. arm64 systems have 2 mechanisms to coalesce TLB entries; "the contiguous bit" (architectural) and HPA (uarch).

This patch set deals with the SW side of things only and based on feedback from the RFC, aims to be the most minimal initial change, upon which future incremental changes can be added. For this reason, the new behaviour is hidden behind a new Kconfig switch, CONFIG_LARGE_ANON_FOLIO, which is disabled by default. Although the code has been refactored to parameterize the desired order of the allocation, when the feature is disabled (by forcing the order to be always 0) my performance tests measure no regression. So I'm hoping this will be a suitable mechanism to allow incremental submissions to the kernel without affecting the rest of the world.

The patches are based on top of v6.4 plus Matthew Wilcox's set_ptes() series [2], which is a hard dependency. I'm not sure of Matthew's exact plans for getting that series into the kernel, but I'm hoping we can start the review process on this patch set independently. I have a branch at [3].

I've posted a separate series concerning the HW part (contpte mapping) for arm64 at [4].

Performance
-----------

Below results show 2 benchmarks; kernel compilation and speedometer 2.0 (a javascript benchmark running in Chromium). Both cases are running on Ampere Altra with 1 NUMA node enabled, Ubuntu 22.04 and XFS filesystem. Each benchmark is repeated 15 times over 5 reboots and averaged.

All improvements are relative to baseline-4k. 'anonfolio-basic' is this series. 'anonfolio' is the full patch set similar to the RFC with the additional changes to the extra 3 fault paths. The rest of the configs are described at [4].

Kernel Compilation (smaller is better):

| kernel          |   real-time |   kern-time |   user-time |
|:----------------|------------:|------------:|------------:|
| baseline-4k     |        0.0% |        0.0% |        0.0% |
| anonfolio-basic |       -5.3% |      -42.9% |       -0.6% |
| anonfolio       |       -5.4% |      -46.0% |       -0.3% |
| contpte         |       -6.8% |      -45.7% |       -2.1% |
| exefolio        |       -8.4% |      -46.4% |       -3.7% |
| baseline-16k    |       -8.7% |      -49.2% |       -3.7% |
| baseline-64k    |      -10.5% |      -66.0% |       -3.5% |

Speedometer 2.0 (bigger is better):

Quoted 8 lines
| kernel          |   runs_per_min |
|:----------------|---------------:|
| baseline-4k     |           0.0% |
| anonfolio-basic |           0.7% |
| anonfolio       |           1.2% |
| contpte         |           3.1% |
| exefolio        |           4.2% |
| baseline-16k    |           5.3% |

Thanks for pushing this forward!

Quoted 4 lines

Changes since RFCv2
-------------------

- Simplified series to bare minimum (on David Hildenbrand's advice)

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

My impression is that this series still includes many pieces that can be split out and discussed separately with followup series.

(I skipped 04/10 and will look at it tomorrow.)

I went through the series twice. Here what I think a bare minimum series (easier to review/debug/land) would look like:

===

Quoted 9 lines
Quoted 9 lines

1. a new arch specific function providing a prefered order within (0, PMD_ORDER).
2. an extended anon folio alloc API taking that order (02/10, partially).
3. an updated folio_add_new_anon_rmap() covering the large() && !pmd_mappable() case (similar to 04/10).
4. s/folio_test_pmd_mappable/folio_test_large/ in page_remove_rmap() (06/10, reviewed-by provided).
5. finally, use the extended anon folio alloc API with the arch preferred order in do_anonymous_page() (10/10, partially).

===

Quoted 7 lines
Quoted 2 lines

The rest can be split out into separate series and move forward in parallel with probably a long list of things we need/want to do.

Thanks for the fadt review - I really appreciate it!

I've responded to many of your comments. I'd appreciate if we can close those points then I will work up a v2.

Thanks!

Based on the latest discussion here [1], my original list above can be optionally reduced to 4 patches: item 2 can be quashed into item 5.

Quoted 6 lines

Also please make sure we have only one global (apply to all archs) Kconfig option, and it should be added in item 5:

if TRANSPARENT_HUGEPAGE
config FLEXIBLE/VARIABLE_THP # or whatever name you see fit
end if

Naming is always the hardest part. I've been calling it LARGE_ANON_FOLIO up until now. But I think you are right that we should show that it is related to THP, so I'll go with FLEXIBLE_THP for v2, and let people shout if they hate it.

If we are not letting the arch declare that it supports FLEXIBLE_THP, then I think we need the default version of arch_wants_pte_order() to return a value higher than 0 (which is what I have it returning at the moment). Because otherwise, for an arch that hasn't defined its own version of arch_wants_pte_order(), FLEXIBLE_THP on vs off will give the same result. So I propose to set the default to ilog2(SZ_64K >> PAGE_SHIFT). Shout if you have any concerns.

Quoted 4 lines

(How many new Kconfig options added within arch/arm64/ is not a concern of MM.)

And please make sure it's disabled by default,

Done

because we are still

Quoted 5 lines

missing many important functions, e.g., I don't think we can mlock() when large() && !pmd_mappable(), see mlock_pte_range() and mlock_vma_folio(). We can fix it along with many things later, but we need to present a plan and a schedule now. Otherwise, there would be pushback if we try to land the series without supporting mlock().

There are other areas that I'm aware off. I'll put together a table and send it out once I have v2 out the door (hopefully tomorrow or Monday). Hopefully we can work together to fill it in and figure out who can do what? I'm certainly planning to continue to push this work forwards beyond this initial patch set.

Thanks,
Ryan

Quoted 7 lines

Do you or Fengwei plan to take on it? (I personally don't.) If not, I'll try to find someone from our team to look at it. (It'd be more scalable if we have a coordinated group of people individually solving different problems.)

[1] https://lore.kernel.org/r/[email protected]/

#53 lore raw
In reply to Yu Zhao
Quoted 98 lines · Yu Zhao
Quoted 77 lines · Yu Zhao
Quoted 61 lines · Ryan Roberts

Hi All,

Following on from the previous RFCv2 [1], this series implements variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults:

- Since SW (the kernel) is dealing with larger chunks of memory than base pages, there are efficiency savings to be had; fewer page faults, batched PTE and RMAP manipulation, fewer items on lists, etc. In short, we reduce kernel overhead. This should benefit all architectures.
- Since we are now mapping physically contiguous chunks of memory, we can take advantage of HW TLB compression techniques. A reduction in TLB pressure speeds up kernel and user space. arm64 systems have 2 mechanisms to coalesce TLB entries; "the contiguous bit" (architectural) and HPA (uarch).

This patch set deals with the SW side of things only and based on feedback from the RFC, aims to be the most minimal initial change, upon which future incremental changes can be added. For this reason, the new behaviour is hidden behind a new Kconfig switch, CONFIG_LARGE_ANON_FOLIO, which is disabled by default. Although the code has been refactored to parameterize the desired order of the allocation, when the feature is disabled (by forcing the order to be always 0) my performance tests measure no regression. So I'm hoping this will be a suitable mechanism to allow incremental submissions to the kernel without affecting the rest of the world.

The patches are based on top of v6.4 plus Matthew Wilcox's set_ptes() series [2], which is a hard dependency. I'm not sure of Matthew's exact plans for getting that series into the kernel, but I'm hoping we can start the review process on this patch set independently. I have a branch at [3].

I've posted a separate series concerning the HW part (contpte mapping) for arm64 at [4].

Performance
-----------

Below results show 2 benchmarks; kernel compilation and speedometer 2.0 (a javascript benchmark running in Chromium). Both cases are running on Ampere Altra with 1 NUMA node enabled, Ubuntu 22.04 and XFS filesystem. Each benchmark is repeated 15 times over 5 reboots and averaged.

All improvements are relative to baseline-4k. 'anonfolio-basic' is this series. 'anonfolio' is the full patch set similar to the RFC with the additional changes to the extra 3 fault paths. The rest of the configs are described at [4].

Kernel Compilation (smaller is better):

| kernel          |   real-time |   kern-time |   user-time |
|:----------------|------------:|------------:|------------:|
| baseline-4k     |        0.0% |        0.0% |        0.0% |
| anonfolio-basic |       -5.3% |      -42.9% |       -0.6% |
| anonfolio       |       -5.4% |      -46.0% |       -0.3% |
| contpte         |       -6.8% |      -45.7% |       -2.1% |
| exefolio        |       -8.4% |      -46.4% |       -3.7% |
| baseline-16k    |       -8.7% |      -49.2% |       -3.7% |
| baseline-64k    |      -10.5% |      -66.0% |       -3.5% |

Speedometer 2.0 (bigger is better):

Quoted 8 lines
| kernel          |   runs_per_min |
|:----------------|---------------:|
| baseline-4k     |           0.0% |
| anonfolio-basic |           0.7% |
| anonfolio       |           1.2% |
| contpte         |           3.1% |
| exefolio        |           4.2% |
| baseline-16k    |           5.3% |

Thanks for pushing this forward!

Quoted 3 lines

Changes since RFCv2
-------------------

Quoted 6 lines
Quoted 1 line

- Simplified series to bare minimum (on David Hildenbrand's advice)

My impression is that this series still includes many pieces that can be split out and discussed separately with followup series.

(I skipped 04/10 and will look at it tomorrow.)

I went through the series twice. Here what I think a bare minimum series (easier to review/debug/land) would look like:
1. a new arch specific function providing a prefered order within (0, PMD_ORDER).
2. an extended anon folio alloc API taking that order (02/10, partially).
3. an updated folio_add_new_anon_rmap() covering the large() && !pmd_mappable() case (similar to 04/10).
4. s/folio_test_pmd_mappable/folio_test_large/ in page_remove_rmap() (06/10, reviewed-by provided).
5. finally, use the extended anon folio alloc API with the arch preferred order in do_anonymous_page() (10/10, partially).

Quoted 2 lines

The rest can be split out into separate series and move forward in parallel with probably a long list of things we need/want to do.

Yeah, the suggestion makes sense to me. And I'd like to go with the simplest way unless there is strong justification for extra optimization for the time being IMHO.

Quoted 1 line