Skip to contentSkip to conversation

mm: export folio_pte_batch as a couple of modules might need it

PATCH In review Barry Song 5 participants 20 messages Feb 27, 2024 – Feb 28, 2024

Subsystems (2)
lore
Share & tools
lore

Barry Song

author
#0 lore raw

PATCH 1/1 mm: export folio_pte_batch as a couple of modules might need it

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>

---
-v1:
at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

 mm/internal.h | 13 +++++++++++++
 mm/memory.c   | 11 +----------
 2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
 	return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
 }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY		((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY		((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+		pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+		bool *any_writable);
+
 void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
 						int nr_throttled);
 static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +1 -10
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
 	set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
 }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY		((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY		((__force fpb_t)BIT(1))
-
 static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
 {
 	if (flags & FPB_IGNORE_DIRTY)
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
  * If "any_writable" is set, it will indicate if any other PTE besides the
  * first (given) PTE is writable.
  */
-static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
+int folio_pte_batch(struct folio *folio, unsigned long addr,
 		pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
 		bool *any_writable)
 {
Signature · 1 line
2.34.1
author
#1 lore raw
Quoted 10 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>

Hi David, Ryan,

Sorry, I realize I just made a mistake and your tags should be both Suggested-by. Please feel
free to review the patch and give comments. I will fix the tags together with addressing your
review comments in v2.

Quoted 66 lines

---
-v1:
at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

 mm/internal.h | 13 +++++++++++++
 mm/memory.c   | 11 +----------
 2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
        return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
 }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY               ((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY          ((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+               pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+               bool *any_writable);
+
 void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
                                                int nr_throttled);
 static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +1 -11
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
        set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
 }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY               ((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY          ((__force fpb_t)BIT(1))
-
 static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
 {
        if (flags & FPB_IGNORE_DIRTY)
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
  * If "any_writable" is set, it will indicate if any other PTE besides the
  * first (given) PTE is writable.
  */
-static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
+int folio_pte_batch(struct folio *folio, unsigned long addr,
                pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
                bool *any_writable)
 {
--
Quoted 2 lines

2.34.1

Thanks
Barry

#2 lore raw
Quoted 12 lines · Barry Song
Quoted 10 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Quoted 7 lines

Hi David, Ryan,

Sorry, I realize I just made a mistake and your tags should be both Suggested-by. Please feel
free to review the patch and give comments. I will fix the tags together with addressing your
review comments in v2.

Don't worry about it. Don't feel you need to update it on my account.

Quoted 72 lines
Quoted 67 lines

---
-v1:
at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

 mm/internal.h | 13 +++++++++++++
 mm/memory.c   | 11 +----------
 2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
        return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
 }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY               ((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY          ((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+               pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+               bool *any_writable);
+
 void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
                                                int nr_throttled);
 static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +1 -11
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
        set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
 }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY               ((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY          ((__force fpb_t)BIT(1))
-
 static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
 {
        if (flags & FPB_IGNORE_DIRTY)
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
  * If "any_writable" is set, it will indicate if any other PTE besides the
  * first (given) PTE is writable.
  */
-static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
+int folio_pte_batch(struct folio *folio, unsigned long addr,
                pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
                bool *any_writable)
 {
--
Quoted 2 lines

2.34.1

Thanks
Barry

#3 lore raw
In reply to Barry Song
Quoted 11 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>
Quoted 5 lines

---
-v1:
at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

I think this will also ultimately be useful for mprotect too, though I haven't looked at it properly yet.

Quoted 50 lines
 mm/internal.h | 13 +++++++++++++
 mm/memory.c   | 11 +----------
 2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
 	return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
 }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY		((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY		((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+		pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+		bool *any_writable);
+
 void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
 						int nr_throttled);
 static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +0 -9
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
 	set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
 }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY		((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY		((__force fpb_t)BIT(1))
-
 static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
 {
 	if (flags & FPB_IGNORE_DIRTY)

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 975

Quoted 3 lines
mm/memory.c +0 -0
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
  * If "any_writable" is set, it will indicate if any other PTE besides the
  * first (given) PTE is writable.
  */

David was talking in Lance's patch thread, about improving the docs for this function now that its exported. Might be worth syncing on that.

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 976

Quoted 2 lines
mm/memory.c +1 -1
-static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
+int folio_pte_batch(struct folio *folio, unsigned long addr,

fork() is very performance sensitive. Is there a risk we are regressing performance by making this out-of-line? Although its in the same compilation unit so the compiler may well inline it anyway?

Either way, perhaps we are better off making it inline in the header? That would avoid needing to rerun David's micro-benchmarks for fork() and munmap().

Thanks,
Ryan

Quoted 3 lines
mm/memory.c +0 -0
 		pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
 		bool *any_writable)
 {
#4 lore raw
Quoted 18 lines · Ryan Roberts
Quoted 11 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>
Quoted 5 lines

---
-v1:
at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

Quoted 3 lines

I think this will also ultimately be useful for mprotect too, though I haven't looked at it properly yet.

Yes, I think we briefly discussed that.

Quoted 48 lines
Quoted 48 lines
  mm/internal.h | 13 +++++++++++++
  mm/memory.c   | 11 +----------
  2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
  	return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
  }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY		((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY		((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+		pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+		bool *any_writable);
+
  void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
  						int nr_throttled);
  static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +0 -9
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
  	set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
  }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY		((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY		((__force fpb_t)BIT(1))
-
  static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
  {
Quoted 8 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 975

Quoted 5 lines
mm/memory.c +0 -0
  	if (flags & FPB_IGNORE_DIRTY)
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
   * If "any_writable" is set, it will indicate if any other PTE besides the
   * first (given) PTE is writable.
   */

David was talking in Lance's patch thread, about improving the docs for this function now that its exported. Might be worth syncing on that.

Here is my take:

---
 mm/memory.c | 22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)
mm/memory.c +18 -4
index d0b855a1837a8..098356b8805ae 100644
@@ -971,16 +971,28 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
 	return pte_wrprotect(pte_mkold(pte));
 }
 
-/*
+/**
+ * folio_pte_batch - detect a PTE batch for a large folio
+ * @folio: The large folio to detect a PTE batch for.
+ * @addr: The user virtual address the first page is mapped at.
+ * @start_ptep: Page table pointer for the first entry.
+ * @pte: Page table entry for the first page.
+ * @max_nr: The maximum number of table entries to consider.
+ * @flags: Flags to modify the PTE batch semantics.
+ * @any_writable: Optional pointer to indicate whether any entry except the
+ *		  first one is writable.
+ *
  * Detect a PTE batch: consecutive (present) PTEs that map consecutive
- * pages of the same folio.
+ * pages of the same large folio.
  *
  * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN,
  * the accessed bit, writable bit, dirty bit (with FPB_IGNORE_DIRTY) and
  * soft-dirty bit (with FPB_IGNORE_SOFT_DIRTY).
  *
- * If "any_writable" is set, it will indicate if any other PTE besides the
- * first (given) PTE is writable.
+ * start_ptep must map any page of the folio. max_nr must be at least one and
+ * must be limited by the caller so scanning cannot exceed a single page table.
+ *
+ * Return: the number of table entries in the batch.
  */
 static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
 		pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
@@ -996,6 +1008,8 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
 		*any_writable = false;
 
 	VM_WARN_ON_FOLIO(!pte_present(pte), folio);
+	VM_WARN_ON_FOLIO(!folio_test_large(folio) || max_nr < 1, folio);
+	VM_WARN_ON_FOLIO(page_folio(pfn_to_page(pte_pfn(pte))) != folio, folio);
 
 	nr = pte_batch_hint(start_ptep, pte);
 	expected_pte = __pte_batch_clear_ignored(pte_advance_pfn(pte, nr), flags);
Signature · 1 line
2.43.2
Quoted 7 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 976

Quoted 3 lines
mm/memory.c +1 -1
-static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
+int folio_pte_batch(struct folio *folio, unsigned long addr,

fork() is very performance sensitive. Is there a risk we are regressing performance by making this out-of-line? Although its in the same compilation unit so the compiler may well inline it anyway?

Easy to verify by looking at the generated asm I guess?

Quoted 3 lines

Either way, perhaps we are better off making it inline in the header? That would avoid needing to rerun David's micro-benchmarks for fork() and munmap().

That way, the compiler can most certainly better optimize it also outside of mm/memory.c

Signature · 3 lines
Cheers,

David / dhildenb
#5 lore raw
Quoted 117 lines · David Hildenbrand
Quoted 18 lines · Ryan Roberts
Quoted 11 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>
Quoted 5 lines

---
  -v1:
  at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

Quoted 3 lines

I think this will also ultimately be useful for mprotect too, though I haven't looked at it properly yet.

Yes, I think we briefly discussed that.

Quoted 48 lines
Quoted 48 lines

  mm/internal.h | 13 +++++++++++++
  mm/memory.c   | 11 +----------

  2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +0 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
       return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
   }

  +/* Flags for folio_pte_batch(). */

mm/internal.h +12 -0
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY        ((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY        ((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+        pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+        bool *any_writable);
+

  void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio, int nr_throttled);
  static inline void acct_reclaim_writeback(struct folio *folio)

mm/memory.c +0 -0
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct
 vm_area_struct *dst_vma,
       set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
   }

  -/* Flags for folio_pte_batch(). */

mm/memory.c +0 -8
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY        ((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY        ((__force fpb_t)BIT(1))
-

  static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) {
      if (flags & FPB_IGNORE_DIRTY)

Quoted 8 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, · line 976

Quoted 5 lines
mm/memory.c +0 -0
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte,
 fpb_t flags)
    * If "any_writable" is set, it will indicate if any other PTE besides the
    * first (given) PTE is writable.
    */

David was talking in Lance's patch thread, about improving the docs for this function now that its exported. Might be worth syncing on that.

Here is my take:

---
 mm/memory.c | 22 ++++++++++++++++++----

 1 file changed, 18 insertions(+), 4 deletions(-)
mm/memory.c +0 -0
index d0b855a1837a8..098356b8805ae 100644
@@ -971,16 +971,28 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte,
 fpb_t flags)
      return pte_wrprotect(pte_mkold(pte));
  }

mm/memory.c +11 -1
-/*
+/**
+ * folio_pte_batch - detect a PTE batch for a large folio
+ * @folio: The large folio to detect a PTE batch for.
+ * @addr: The user virtual address the first page is mapped at.
+ * @start_ptep: Page table pointer for the first entry.
+ * @pte: Page table entry for the first page.
+ * @max_nr: The maximum number of table entries to consider.
+ * @flags: Flags to modify the PTE batch semantics.
+ * @any_writable: Optional pointer to indicate whether any entry except the
+ *          first one is writable.
+ *

  * Detect a PTE batch: consecutive (present) PTEs that map consecutive
- * pages of the same folio.
+ * pages of the same large folio.
  *
  * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN,
  * the accessed bit, writable bit, dirty bit (with FPB_IGNORE_DIRTY) and
  * soft-dirty bit (with FPB_IGNORE_SOFT_DIRTY).

Quoted 8 lines

  *
- * If "any_writable" is set, it will indicate if any other PTE besides the
- * first (given) PTE is writable.
+ * start_ptep must map any page of the folio. max_nr must be at least one and
+ * must be limited by the caller so scanning cannot exceed a single page table.
+ *
+ * Return: the number of table entries in the batch.
  */

LGTM!

Quoted 12 lines

 static inline int folio_pte_batch(struct folio *folio, unsigned long addr, pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,

mm/memory.c +0 -0
@@ -996,6 +1008,8 @@ static inline int folio_pte_batch(struct folio *folio,
 unsigned long addr,
          *any_writable = false;


     VM_WARN_ON_FOLIO(!pte_present(pte), folio);
+    VM_WARN_ON_FOLIO(!folio_test_large(folio) || max_nr < 1, folio); +    VM_WARN_ON_FOLIO(page_folio(pfn_to_page(pte_pfn(pte))) != folio, folio);
     nr = pte_batch_hint(start_ptep, pte);
     expected_pte = __pte_batch_clear_ignored(pte_advance_pfn(pte, nr), flags);

author
#6 lore raw
In reply to David Hildenbrand
Quoted 141 lines · David Hildenbrand
Quoted 18 lines · Ryan Roberts
Quoted 11 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>
Quoted 5 lines

---
-v1:
at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

Quoted 3 lines

I think this will also ultimately be useful for mprotect too, though I haven't looked at it properly yet.

Yes, I think we briefly discussed that.

Quoted 48 lines
Quoted 48 lines
  mm/internal.h | 13 +++++++++++++
  mm/memory.c   | 11 +----------
  2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
     return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
  }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+            pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+            bool *any_writable);
+
  void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
                                             int nr_throttled);
  static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +0 -9
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
     set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
  }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
-
  static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
  {
Quoted 8 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 975

Quoted 5 lines
mm/memory.c +0 -0
     if (flags & FPB_IGNORE_DIRTY)
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
   * If "any_writable" is set, it will indicate if any other PTE besides the
   * first (given) PTE is writable.
   */

David was talking in Lance's patch thread, about improving the docs for this function now that its exported. Might be worth syncing on that.

Here is my take:

---
  mm/memory.c | 22 ++++++++++++++++++----
  1 file changed, 18 insertions(+), 4 deletions(-)
mm/memory.c +18 -5
index d0b855a1837a8..098356b8805ae 100644
@@ -971,16 +971,28 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
        return pte_wrprotect(pte_mkold(pte));
  }
 
-/*
+/**
+ * folio_pte_batch - detect a PTE batch for a large folio
+ * @folio: The large folio to detect a PTE batch for.
+ * @addr: The user virtual address the first page is mapped at.
+ * @start_ptep: Page table pointer for the first entry.
+ * @pte: Page table entry for the first page.
+ * @max_nr: The maximum number of table entries to consider.
+ * @flags: Flags to modify the PTE batch semantics.
+ * @any_writable: Optional pointer to indicate whether any entry except the
+ *               first one is writable.
+ *
   * Detect a PTE batch: consecutive (present) PTEs that map consecutive
- * pages of the same folio.
+ * pages of the same large folio.
   *
   * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN,
   * the accessed bit, writable bit, dirty bit (with FPB_IGNORE_DIRTY) and
   * soft-dirty bit (with FPB_IGNORE_SOFT_DIRTY).
   *
- * If "any_writable" is set, it will indicate if any other PTE besides the
- * first (given) PTE is writable.
+ * start_ptep must map any page of the folio. max_nr must be at least one and
+ * must be limited by the caller so scanning cannot exceed a single page table.
+ *
+ * Return: the number of table entries in the batch.
   */
  static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
                pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
@@ -996,6 +1008,8 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
                *any_writable = false;
 
        VM_WARN_ON_FOLIO(!pte_present(pte), folio);
+       VM_WARN_ON_FOLIO(!folio_test_large(folio) || max_nr < 1, folio);
+       VM_WARN_ON_FOLIO(page_folio(pfn_to_page(pte_pfn(pte))) != folio, folio);
 
        nr = pte_batch_hint(start_ptep, pte);
        expected_pte = __pte_batch_clear_ignored(pte_advance_pfn(pte, nr), flags);
--

2.43.2

Quoted 1 line
Quoted 8 lines
Quoted 6 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 976

Quoted 2 lines
mm/memory.c +1 -1
-static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
+int folio_pte_batch(struct folio *folio, unsigned long addr,

fork() is very performance sensitive. Is there a risk we are regressing performance by making this out-of-line? Although its in the same compilation unit so the compiler may well inline it anyway?

Easy to verify by looking at the generated asm I guess?

my aarch64-linux-gnu-gcc didn't inline it

$ aarch64-linux-gnu-gcc --version
aarch64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
$ nm -S -s vmlinux.a | grep  folio_pte_batch
0000000000003818 0000000000000204 T folio_pte_batch
Quoted 4 lines
Quoted 4 lines

Either way, perhaps we are better off making it inline in the header? That would avoid needing to rerun David's micro-benchmarks for fork() and munmap().

actually tried this before trying extern, the problem is that we have to add others into internal.h, for example __pte_batch_clear_ignored, which seems not API. are we comfortable to move that one to internal.h too?

Quoted 7 lines

That way, the compiler can most certainly better optimize it also outside of mm/memory.c

--
Cheers,

David / dhildenb

Thanks
Barry

#7 lore raw
Quoted 149 lines · Barry Song
Quoted 148 lines · David Hildenbrand
Quoted 18 lines · Ryan Roberts
Quoted 11 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>
Quoted 5 lines

---
-v1:
at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

Quoted 3 lines

I think this will also ultimately be useful for mprotect too, though I haven't looked at it properly yet.

Yes, I think we briefly discussed that.

Quoted 48 lines
Quoted 48 lines
   mm/internal.h | 13 +++++++++++++
   mm/memory.c   | 11 +----------
   2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
      return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
   }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+            pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+            bool *any_writable);
+
   void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
                                              int nr_throttled);
   static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +0 -9
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
      set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
   }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
-
   static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
   {
Quoted 8 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 975

Quoted 5 lines
mm/memory.c +0 -0
      if (flags & FPB_IGNORE_DIRTY)
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
    * If "any_writable" is set, it will indicate if any other PTE besides the
    * first (given) PTE is writable.
    */

David was talking in Lance's patch thread, about improving the docs for this function now that its exported. Might be worth syncing on that.

Here is my take:

---
   mm/memory.c | 22 ++++++++++++++++++----
   1 file changed, 18 insertions(+), 4 deletions(-)
mm/memory.c +18 -5
index d0b855a1837a8..098356b8805ae 100644
@@ -971,16 +971,28 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
         return pte_wrprotect(pte_mkold(pte));
   }
 
-/*
+/**
+ * folio_pte_batch - detect a PTE batch for a large folio
+ * @folio: The large folio to detect a PTE batch for.
+ * @addr: The user virtual address the first page is mapped at.
+ * @start_ptep: Page table pointer for the first entry.
+ * @pte: Page table entry for the first page.
+ * @max_nr: The maximum number of table entries to consider.
+ * @flags: Flags to modify the PTE batch semantics.
+ * @any_writable: Optional pointer to indicate whether any entry except the
+ *               first one is writable.
+ *
    * Detect a PTE batch: consecutive (present) PTEs that map consecutive
- * pages of the same folio.
+ * pages of the same large folio.
    *
    * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN,
    * the accessed bit, writable bit, dirty bit (with FPB_IGNORE_DIRTY) and
    * soft-dirty bit (with FPB_IGNORE_SOFT_DIRTY).
    *
- * If "any_writable" is set, it will indicate if any other PTE besides the
- * first (given) PTE is writable.
+ * start_ptep must map any page of the folio. max_nr must be at least one and
+ * must be limited by the caller so scanning cannot exceed a single page table.
+ *
+ * Return: the number of table entries in the batch.
    */
   static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
                 pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
@@ -996,6 +1008,8 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
                 *any_writable = false;
 
         VM_WARN_ON_FOLIO(!pte_present(pte), folio);
+       VM_WARN_ON_FOLIO(!folio_test_large(folio) || max_nr < 1, folio);
+       VM_WARN_ON_FOLIO(page_folio(pfn_to_page(pte_pfn(pte))) != folio, folio);
 
         nr = pte_batch_hint(start_ptep, pte);
         expected_pte = __pte_batch_clear_ignored(pte_advance_pfn(pte, nr), flags);
--

2.43.2

Quoted 7 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 976

Quoted 3 lines
mm/memory.c +1 -1
-static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
+int folio_pte_batch(struct folio *folio, unsigned long addr,

fork() is very performance sensitive. Is there a risk we are regressing performance by making this out-of-line? Although its in the same compilation unit so the compiler may well inline it anyway?

Quoted 3 lines
Quoted 1 line

Easy to verify by looking at the generated asm I guess?

my aarch64-linux-gnu-gcc didn't inline it

I think on x86-64 it would inline it with "gcc (GCC) 13.2.1 20231205 (Red Hat 13.2.1-6)"

Quoted 8 lines
$ aarch64-linux-gnu-gcc --version
aarch64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
$ nm -S -s vmlinux.a | grep  folio_pte_batch
0000000000003818 0000000000000204 T folio_pte_batch

As it's only used on the folio_test_large() "slower" paths, likely optimizing out the "writable" check (and possibly the flags) might not be that important.

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Either way, perhaps we are better off making it inline in the header? That would avoid needing to rerun David's micro-benchmarks for fork() and munmap().

actually tried this before trying extern, the problem is that we have to add others into internal.h, for example __pte_batch_clear_ignored, which seems not API. are we comfortable to move that one to internal.h too?

Yes, that shouldn't stop us.

Signature · 3 lines
Cheers,

David / dhildenb
#8 lore raw
In reply to David Hildenbrand
Quoted 128 lines · David Hildenbrand
Quoted 18 lines · Ryan Roberts
Quoted 11 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>
Quoted 5 lines

---
-v1:
at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

Quoted 3 lines

I think this will also ultimately be useful for mprotect too, though I haven't looked at it properly yet.

Yes, I think we briefly discussed that.

Quoted 48 lines
Quoted 48 lines
  mm/internal.h | 13 +++++++++++++
  mm/memory.c   | 11 +----------
  2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
     return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
  }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+            pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+            bool *any_writable);
+
  void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
                                             int nr_throttled);
  static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +0 -9
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
     set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
  }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
-
  static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
  {
Quoted 8 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 975

Quoted 5 lines
mm/memory.c +0 -0
     if (flags & FPB_IGNORE_DIRTY)
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
   * If "any_writable" is set, it will indicate if any other PTE besides the
   * first (given) PTE is writable.
   */

David was talking in Lance's patch thread, about improving the docs for this function now that its exported. Might be worth syncing on that.

Here is my take:

---
  mm/memory.c | 22 ++++++++++++++++++----
  1 file changed, 18 insertions(+), 4 deletions(-)
mm/memory.c +16 -4
index d0b855a1837a8..098356b8805ae 100644
@@ -971,16 +971,28 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
        return pte_wrprotect(pte_mkold(pte));
  }
 
-/*
+/**
+ * folio_pte_batch - detect a PTE batch for a large folio
+ * @folio: The large folio to detect a PTE batch for.
+ * @addr: The user virtual address the first page is mapped at.
+ * @start_ptep: Page table pointer for the first entry.
+ * @pte: Page table entry for the first page.
+ * @max_nr: The maximum number of table entries to consider.
+ * @flags: Flags to modify the PTE batch semantics.
+ * @any_writable: Optional pointer to indicate whether any entry except the
+ *               first one is writable.
+ *
   * Detect a PTE batch: consecutive (present) PTEs that map consecutive
- * pages of the same folio.
+ * pages of the same large folio.
   *
   * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN,
   * the accessed bit, writable bit, dirty bit (with FPB_IGNORE_DIRTY) and
   * soft-dirty bit (with FPB_IGNORE_SOFT_DIRTY).
   *
- * If "any_writable" is set, it will indicate if any other PTE besides the
- * first (given) PTE is writable.
+ * start_ptep must map any page of the folio. max_nr must be at least one and
+ * must be limited by the caller so scanning cannot exceed a single page table.
+ *
+ * Return: the number of table entries in the batch.
   */
  static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
                pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,

mm/memory.c static inline int folio_pte_batch(struct folio *folio, unsigned long addr, · line 1012

Quoted 5 lines
mm/memory.c +2 -0
@@ -996,6 +1008,8 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
                *any_writable = false;
 
        VM_WARN_ON_FOLIO(!pte_present(pte), folio);
+       VM_WARN_ON_FOLIO(!folio_test_large(folio) || max_nr < 1, folio);
+       VM_WARN_ON_FOLIO(page_folio(pfn_to_page(pte_pfn(pte))) != folio, folio);

Nit:
IIUC, the pte that maps to the first page.
- VM_WARN_ON_FOLIO(page_folio(pfn_to_page(pte_pfn(pte))) !=
folio, folio);
+ VM_WARN_ON_FOLIO(pte_pfn(pte) != folio_pfn(folio), folio);

Quoted 27 lines
mm/memory.c +0 -1
        nr = pte_batch_hint(start_ptep, pte);
        expected_pte = __pte_batch_clear_ignored(pte_advance_pfn(pte, nr), flags);
--

2.43.2

Quoted 7 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 976

Quoted 3 lines
mm/memory.c +1 -1
-static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
+int folio_pte_batch(struct folio *folio, unsigned long addr,

fork() is very performance sensitive. Is there a risk we are regressing performance by making this out-of-line? Although its in the same compilation unit so the compiler may well inline it anyway?

Easy to verify by looking at the generated asm I guess?

Quoted 3 lines

Either way, perhaps we are better off making it inline in the header? That would avoid needing to rerun David's micro-benchmarks for fork() and munmap().

That way, the compiler can most certainly better optimize it also outside of mm/memory.c

--
Cheers,

David / dhildenb

#9 lore raw
Quoted 131 lines · Lance Yang
Quoted 130 lines · David Hildenbrand
Quoted 18 lines · Ryan Roberts
Quoted 11 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>
Quoted 5 lines

---
-v1:
at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

Quoted 3 lines

I think this will also ultimately be useful for mprotect too, though I haven't looked at it properly yet.

Yes, I think we briefly discussed that.

Quoted 48 lines
Quoted 48 lines
   mm/internal.h | 13 +++++++++++++
   mm/memory.c   | 11 +----------
   2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
      return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
   }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+            pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+            bool *any_writable);
+
   void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
                                              int nr_throttled);
   static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +0 -9
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
      set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
   }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
-
   static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
   {
Quoted 8 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 975

Quoted 5 lines
mm/memory.c +0 -0
      if (flags & FPB_IGNORE_DIRTY)
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
    * If "any_writable" is set, it will indicate if any other PTE besides the
    * first (given) PTE is writable.
    */

David was talking in Lance's patch thread, about improving the docs for this function now that its exported. Might be worth syncing on that.

Here is my take:

---
   mm/memory.c | 22 ++++++++++++++++++----
   1 file changed, 18 insertions(+), 4 deletions(-)
mm/memory.c +16 -4
index d0b855a1837a8..098356b8805ae 100644
@@ -971,16 +971,28 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
         return pte_wrprotect(pte_mkold(pte));
   }
 
-/*
+/**
+ * folio_pte_batch - detect a PTE batch for a large folio
+ * @folio: The large folio to detect a PTE batch for.
+ * @addr: The user virtual address the first page is mapped at.
+ * @start_ptep: Page table pointer for the first entry.
+ * @pte: Page table entry for the first page.
+ * @max_nr: The maximum number of table entries to consider.
+ * @flags: Flags to modify the PTE batch semantics.
+ * @any_writable: Optional pointer to indicate whether any entry except the
+ *               first one is writable.
+ *
    * Detect a PTE batch: consecutive (present) PTEs that map consecutive
- * pages of the same folio.
+ * pages of the same large folio.
    *
    * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN,
    * the accessed bit, writable bit, dirty bit (with FPB_IGNORE_DIRTY) and
    * soft-dirty bit (with FPB_IGNORE_SOFT_DIRTY).
    *
- * If "any_writable" is set, it will indicate if any other PTE besides the
- * first (given) PTE is writable.
+ * start_ptep must map any page of the folio. max_nr must be at least one and
+ * must be limited by the caller so scanning cannot exceed a single page table.
+ *
+ * Return: the number of table entries in the batch.
    */
   static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
                 pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
@@ -996,6 +1008,8 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
                 *any_writable = false;
 
Quoted 9 lines

mm/memory.c static inline int folio_pte_batch(struct folio *folio, unsigned long addr, · line 1012

Quoted 3 lines
mm/memory.c +2 -0
         VM_WARN_ON_FOLIO(!pte_present(pte), folio);
+       VM_WARN_ON_FOLIO(!folio_test_large(folio) || max_nr < 1, folio);
+       VM_WARN_ON_FOLIO(page_folio(pfn_to_page(pte_pfn(pte))) != folio, folio);

Nit:
IIUC, the pte that maps to the first page.
- VM_WARN_ON_FOLIO(page_folio(pfn_to_page(pte_pfn(pte))) !=
folio, folio);
+ VM_WARN_ON_FOLIO(pte_pfn(pte) != folio_pfn(folio), folio);

That would only work if the PTE would map the very first subpage of the folio, not any subpage?

Signature · 3 lines
Cheers,

David / dhildenb
#10 lore raw
Quoted 143 lines · David Hildenbrand
Quoted 131 lines · Lance Yang
Quoted 130 lines · David Hildenbrand
Quoted 18 lines · Ryan Roberts
Quoted 11 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>
Quoted 5 lines

---
-v1:
at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

Quoted 3 lines

I think this will also ultimately be useful for mprotect too, though I haven't looked at it properly yet.

Yes, I think we briefly discussed that.

Quoted 48 lines
Quoted 48 lines
   mm/internal.h | 13 +++++++++++++
   mm/memory.c   | 11 +----------
   2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
      return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
   }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+            pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+            bool *any_writable);
+
   void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
                                              int nr_throttled);
   static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +0 -9
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
      set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
   }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
-
   static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
   {
Quoted 8 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 975

Quoted 5 lines
mm/memory.c +0 -0
      if (flags & FPB_IGNORE_DIRTY)
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
    * If "any_writable" is set, it will indicate if any other PTE besides the
    * first (given) PTE is writable.
    */

David was talking in Lance's patch thread, about improving the docs for this function now that its exported. Might be worth syncing on that.

Here is my take:

---
   mm/memory.c | 22 ++++++++++++++++++----
   1 file changed, 18 insertions(+), 4 deletions(-)
mm/memory.c +16 -4
index d0b855a1837a8..098356b8805ae 100644
@@ -971,16 +971,28 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
         return pte_wrprotect(pte_mkold(pte));
   }
 
-/*
+/**
+ * folio_pte_batch - detect a PTE batch for a large folio
+ * @folio: The large folio to detect a PTE batch for.
+ * @addr: The user virtual address the first page is mapped at.
+ * @start_ptep: Page table pointer for the first entry.
+ * @pte: Page table entry for the first page.
+ * @max_nr: The maximum number of table entries to consider.
+ * @flags: Flags to modify the PTE batch semantics.
+ * @any_writable: Optional pointer to indicate whether any entry except the
+ *               first one is writable.
+ *
    * Detect a PTE batch: consecutive (present) PTEs that map consecutive
- * pages of the same folio.
+ * pages of the same large folio.
    *
    * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN,
    * the accessed bit, writable bit, dirty bit (with FPB_IGNORE_DIRTY) and
    * soft-dirty bit (with FPB_IGNORE_SOFT_DIRTY).
    *
- * If "any_writable" is set, it will indicate if any other PTE besides the
- * first (given) PTE is writable.
+ * start_ptep must map any page of the folio. max_nr must be at least one and
+ * must be limited by the caller so scanning cannot exceed a single page table.
+ *
+ * Return: the number of table entries in the batch.
    */
   static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
                 pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
@@ -996,6 +1008,8 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
                 *any_writable = false;
 
Quoted 9 lines

mm/memory.c static inline int folio_pte_batch(struct folio *folio, unsigned long addr, · line 1012

Quoted 3 lines
mm/memory.c +2 -0
         VM_WARN_ON_FOLIO(!pte_present(pte), folio);
+       VM_WARN_ON_FOLIO(!folio_test_large(folio) || max_nr < 1, folio);
+       VM_WARN_ON_FOLIO(page_folio(pfn_to_page(pte_pfn(pte))) != folio, folio);

Nit:
IIUC, the pte that maps to the first page.
- VM_WARN_ON_FOLIO(page_folio(pfn_to_page(pte_pfn(pte))) !=
folio, folio);
+ VM_WARN_ON_FOLIO(pte_pfn(pte) != folio_pfn(folio), folio);

Quoted 2 lines

That would only work if the PTE would map the very first subpage of the folio, not any subpage?

You're right. I got it.

Quoted 6 lines

--
Cheers,

David / dhildenb

#11 lore raw
In reply to David Hildenbrand
Quoted 95 lines · David Hildenbrand
Quoted 18 lines · Ryan Roberts
Quoted 11 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>
Quoted 5 lines

---
-v1:
at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

Quoted 3 lines

I think this will also ultimately be useful for mprotect too, though I haven't looked at it properly yet.

Yes, I think we briefly discussed that.

Quoted 48 lines
Quoted 48 lines
  mm/internal.h | 13 +++++++++++++
  mm/memory.c   | 11 +----------
  2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
     return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
  }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+            pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+            bool *any_writable);
+
  void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
                                             int nr_throttled);
  static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +0 -9
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
     set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
  }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
-
  static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
  {
Quoted 8 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 975

Quoted 5 lines
mm/memory.c +0 -0
     if (flags & FPB_IGNORE_DIRTY)
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
   * If "any_writable" is set, it will indicate if any other PTE besides the
   * first (given) PTE is writable.
   */

David was talking in Lance's patch thread, about improving the docs for this function now that its exported. Might be worth syncing on that.

Here is my take:

---
  mm/memory.c | 22 ++++++++++++++++++----
  1 file changed, 18 insertions(+), 4 deletions(-)
mm/memory.c +0 -0
index d0b855a1837a8..098356b8805ae 100644

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 979

Quoted 10 lines
mm/memory.c +6 -1
@@ -971,16 +971,28 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
        return pte_wrprotect(pte_mkold(pte));
  }
 
-/*
+/**
+ * folio_pte_batch - detect a PTE batch for a large folio
+ * @folio: The large folio to detect a PTE batch for.
+ * @addr: The user virtual address the first page is mapped at.
+ * @start_ptep: Page table pointer for the first entry.
+ * @pte: Page table entry for the first page.

Nit:

- * @pte: Page table entry for the first page.
+ * @pte: Page table entry for the first page that must be the first subpage of
+ * the folio excluding arm64 for now.

IIUC, pte_batch_hint is always 1 excluding arm64 for now. I'm not sure if this modification will be helpful?

Thanks,
Lance

Quoted 56 lines
mm/memory.c +12 -4
+ * @max_nr: The maximum number of table entries to consider.
+ * @flags: Flags to modify the PTE batch semantics.
+ * @any_writable: Optional pointer to indicate whether any entry except the
+ *               first one is writable.
+ *
   * Detect a PTE batch: consecutive (present) PTEs that map consecutive
- * pages of the same folio.
+ * pages of the same large folio.
   *
   * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN,
   * the accessed bit, writable bit, dirty bit (with FPB_IGNORE_DIRTY) and
   * soft-dirty bit (with FPB_IGNORE_SOFT_DIRTY).
   *
- * If "any_writable" is set, it will indicate if any other PTE besides the
- * first (given) PTE is writable.
+ * start_ptep must map any page of the folio. max_nr must be at least one and
+ * must be limited by the caller so scanning cannot exceed a single page table.
+ *
+ * Return: the number of table entries in the batch.
   */
  static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
                pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
@@ -996,6 +1008,8 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
                *any_writable = false;
 
        VM_WARN_ON_FOLIO(!pte_present(pte), folio);
+       VM_WARN_ON_FOLIO(!folio_test_large(folio) || max_nr < 1, folio);
+       VM_WARN_ON_FOLIO(page_folio(pfn_to_page(pte_pfn(pte))) != folio, folio);
 
        nr = pte_batch_hint(start_ptep, pte);
        expected_pte = __pte_batch_clear_ignored(pte_advance_pfn(pte, nr), flags);
--

2.43.2

Quoted 7 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 976

Quoted 3 lines
mm/memory.c +1 -1
-static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
+int folio_pte_batch(struct folio *folio, unsigned long addr,

fork() is very performance sensitive. Is there a risk we are regressing performance by making this out-of-line? Although its in the same compilation unit so the compiler may well inline it anyway?

Easy to verify by looking at the generated asm I guess?

Quoted 3 lines

Either way, perhaps we are better off making it inline in the header? That would avoid needing to rerun David's micro-benchmarks for fork() and munmap().

That way, the compiler can most certainly better optimize it also outside of mm/memory.c

--
Cheers,

David / dhildenb

#12 lore raw
Quoted 109 lines · Lance Yang
Quoted 95 lines · David Hildenbrand
Quoted 18 lines · Ryan Roberts
Quoted 11 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>
Quoted 5 lines

---
-v1:
at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

Quoted 3 lines

I think this will also ultimately be useful for mprotect too, though I haven't looked at it properly yet.

Yes, I think we briefly discussed that.

Quoted 48 lines
Quoted 48 lines
   mm/internal.h | 13 +++++++++++++
   mm/memory.c   | 11 +----------
   2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
      return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
   }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+            pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+            bool *any_writable);
+
   void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
                                              int nr_throttled);
   static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +0 -9
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
      set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
   }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
-
   static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
   {
Quoted 8 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 975

Quoted 5 lines
mm/memory.c +0 -0
      if (flags & FPB_IGNORE_DIRTY)
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
    * If "any_writable" is set, it will indicate if any other PTE besides the
    * first (given) PTE is writable.
    */

David was talking in Lance's patch thread, about improving the docs for this function now that its exported. Might be worth syncing on that.

Here is my take:

---
   mm/memory.c | 22 ++++++++++++++++++----
   1 file changed, 18 insertions(+), 4 deletions(-)
mm/memory.c +0 -0
index d0b855a1837a8..098356b8805ae 100644

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 979

Quoted 10 lines
mm/memory.c +6 -1
@@ -971,16 +971,28 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
         return pte_wrprotect(pte_mkold(pte));
   }
 
-/*
+/**
+ * folio_pte_batch - detect a PTE batch for a large folio
+ * @folio: The large folio to detect a PTE batch for.
+ * @addr: The user virtual address the first page is mapped at.
+ * @start_ptep: Page table pointer for the first entry.
+ * @pte: Page table entry for the first page.

Nit:

Quoted 6 lines

- * @pte: Page table entry for the first page.
+ * @pte: Page table entry for the first page that must be the first subpage of
+ * the folio excluding arm64 for now.

IIUC, pte_batch_hint is always 1 excluding arm64 for now. I'm not sure if this modification will be helpful?

IIRC, Ryan made sure that this also works when passing another subpage, after when cont-pte is set. Otherwise this would already be broken for fork/zap.

So I don't think this comment would actually be correct.

Signature · 3 lines
Cheers,

David / dhildenb
#13 lore raw
Quoted 117 lines · David Hildenbrand
Quoted 116 lines · Lance Yang
Quoted 98 lines · David Hildenbrand
Quoted 18 lines · Ryan Roberts
Quoted 11 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>
Quoted 5 lines

---
   -v1:
   at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

Quoted 3 lines

I think this will also ultimately be useful for mprotect too, though I haven't looked at it properly yet.

Yes, I think we briefly discussed that.

Quoted 50 lines
Quoted 50 lines

   mm/internal.h | 13 +++++++++++++
   mm/memory.c   | 11 +----------

   2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
       return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
    }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+            pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+            bool *any_writable);
+
    void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
                                               int nr_throttled);
    static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +0 -9
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct
 vm_area_struct *dst_vma,
       set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
    }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
-
    static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
    {

      if (flags & FPB_IGNORE_DIRTY)

Quoted 8 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t · line 976

Quoted 5 lines
mm/memory.c +0 -0
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t
 pte, fpb_t flags)
     * If "any_writable" is set, it will indicate if any other PTE besides the
     * first (given) PTE is writable.
     */

David was talking in Lance's patch thread, about improving the docs for this function now that its exported. Might be worth syncing on that.

Here is my take:

---
   mm/memory.c | 22 ++++++++++++++++++----

   1 file changed, 18 insertions(+), 4 deletions(-)
mm/memory.c +0 -0
index d0b855a1837a8..098356b8805ae 100644
@@ -971,16 +971,28 @@ static inline pte_t __pte_batch_clear_ignored(pte_t
 pte, fpb_t flags)

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t · line 980

Quoted 10 lines
mm/memory.c +6 -1
          return pte_wrprotect(pte_mkold(pte));
    }
 
-/*
+/**
+ * folio_pte_batch - detect a PTE batch for a large folio
+ * @folio: The large folio to detect a PTE batch for.
+ * @addr: The user virtual address the first page is mapped at.
+ * @start_ptep: Page table pointer for the first entry.
+ * @pte: Page table entry for the first page.

Nit:

- * @pte: Page table entry for the first page.
+ * @pte: Page table entry for the first page that must be the first subpage of
+ *               the folio excluding arm64 for now.

Quoted 7 lines
Quoted 2 lines

IIUC, pte_batch_hint is always 1 excluding arm64 for now. I'm not sure if this modification will be helpful?

IIRC, Ryan made sure that this also works when passing another subpage, after when cont-pte is set. Otherwise this would already be broken for fork/zap.

So I don't think this comment would actually be correct.

Indeed, the spec for the function is exactly the same for arm64 as for other arches. It's just that arm64 can accelerate the implementation by skipping forward to the next contpte boundary when the current pte is part of a contpte block.

There is no requirement for pte (or addr or start_ptep) to point to the first subpage of a folio - they can point to any subpage.

pte, addr and start_ptep must all refer to the same entry, but I think that's clear from the existing text.

#14 lore raw

Thanks, Ryan, Barry, David!

Best,
Lance

Quoted 139 lines · Ryan Roberts
Quoted 117 lines · David Hildenbrand
Quoted 116 lines · Lance Yang
Quoted 98 lines · David Hildenbrand
Quoted 18 lines · Ryan Roberts
Quoted 11 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>
Quoted 5 lines

---
-v1:
at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

Quoted 3 lines

I think this will also ultimately be useful for mprotect too, though I haven't looked at it properly yet.

Yes, I think we briefly discussed that.

Quoted 50 lines
Quoted 50 lines
   mm/internal.h | 13 +++++++++++++
   mm/memory.c   | 11 +----------
   2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
      return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
   }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+            pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+            bool *any_writable);
+
   void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
                                              int nr_throttled);
   static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +0 -9
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct
 vm_area_struct *dst_vma,
      set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
   }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
-
   static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
   {

if (flags & FPB_IGNORE_DIRTY)

Quoted 8 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t · line 976

Quoted 5 lines
mm/memory.c +0 -0
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t
 pte, fpb_t flags)
    * If "any_writable" is set, it will indicate if any other PTE besides the
    * first (given) PTE is writable.
    */

David was talking in Lance's patch thread, about improving the docs for this function now that its exported. Might be worth syncing on that.

Here is my take:

---
   mm/memory.c | 22 ++++++++++++++++++----
   1 file changed, 18 insertions(+), 4 deletions(-)
mm/memory.c +0 -0
index d0b855a1837a8..098356b8805ae 100644
@@ -971,16 +971,28 @@ static inline pte_t __pte_batch_clear_ignored(pte_t
 pte, fpb_t flags)

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t · line 980

Quoted 10 lines
mm/memory.c +6 -1
         return pte_wrprotect(pte_mkold(pte));
   }
 
-/*
+/**
+ * folio_pte_batch - detect a PTE batch for a large folio
+ * @folio: The large folio to detect a PTE batch for.
+ * @addr: The user virtual address the first page is mapped at.
+ * @start_ptep: Page table pointer for the first entry.
+ * @pte: Page table entry for the first page.

Nit:

- * @pte: Page table entry for the first page.
+ * @pte: Page table entry for the first page that must be the first subpage of
+ * the folio excluding arm64 for now.

Quoted 7 lines
Quoted 2 lines

IIUC, pte_batch_hint is always 1 excluding arm64 for now. I'm not sure if this modification will be helpful?

IIRC, Ryan made sure that this also works when passing another subpage, after when cont-pte is set. Otherwise this would already be broken for fork/zap.

So I don't think this comment would actually be correct.

Indeed, the spec for the function is exactly the same for arm64 as for other arches. It's just that arm64 can accelerate the implementation by skipping forward to the next contpte boundary when the current pte is part of a contpte block.

There is no requirement for pte (or addr or start_ptep) to point to the first subpage of a folio - they can point to any subpage.

pte, addr and start_ptep must all refer to the same entry, but I think that's clear from the existing text.

author
#15 lore raw
In reply to Lance Yang
Quoted 110 lines · Lance Yang
Quoted 95 lines · David Hildenbrand
Quoted 18 lines · Ryan Roberts
Quoted 11 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>
Quoted 5 lines

---
-v1:
at least two jobs madv_free and madv_pageout depend on it. To avoid conflicts and dependencies, after discussing with Lance, we prefer this one can land earlier.

Quoted 3 lines

I think this will also ultimately be useful for mprotect too, though I haven't looked at it properly yet.

Yes, I think we briefly discussed that.

Quoted 48 lines
Quoted 48 lines
  mm/internal.h | 13 +++++++++++++
  mm/memory.c   | 11 +----------
  2 files changed, 14 insertions(+), 10 deletions(-)
mm/internal.h +13 -0
index 13b59d384845..8e2bc304f671 100644
@@ -83,6 +83,19 @@ static inline void *folio_raw_mapping(struct folio *folio)
     return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
  }
 
+/* Flags for folio_pte_batch(). */
+typedef int __bitwise fpb_t;
+
+/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
+#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
+
+/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
+#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
+
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+            pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+            bool *any_writable);
+
  void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
                                             int nr_throttled);
  static inline void acct_reclaim_writeback(struct folio *folio)
mm/memory.c +0 -9
index 1c45b6a42a1b..319b3be05e75 100644
@@ -953,15 +953,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
     set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
  }
 
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */
-#define FPB_IGNORE_DIRTY            ((__force fpb_t)BIT(0))
-
-/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */
-#define FPB_IGNORE_SOFT_DIRTY               ((__force fpb_t)BIT(1))
-
  static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
  {
Quoted 8 lines

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 975

Quoted 5 lines
mm/memory.c +0 -0
     if (flags & FPB_IGNORE_DIRTY)
@@ -982,7 +973,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
   * If "any_writable" is set, it will indicate if any other PTE besides the
   * first (given) PTE is writable.
   */

David was talking in Lance's patch thread, about improving the docs for this function now that its exported. Might be worth syncing on that.

Here is my take:

---
  mm/memory.c | 22 ++++++++++++++++++----
  1 file changed, 18 insertions(+), 4 deletions(-)
mm/memory.c +0 -0
index d0b855a1837a8..098356b8805ae 100644

mm/memory.c static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags) · line 979

Quoted 10 lines
mm/memory.c +6 -1
@@ -971,16 +971,28 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
        return pte_wrprotect(pte_mkold(pte));
  }
 
-/*
+/**
+ * folio_pte_batch - detect a PTE batch for a large folio
+ * @folio: The large folio to detect a PTE batch for.
+ * @addr: The user virtual address the first page is mapped at.
+ * @start_ptep: Page table pointer for the first entry.
+ * @pte: Page table entry for the first page.

Nit:

Quoted 6 lines

- * @pte: Page table entry for the first page.
+ * @pte: Page table entry for the first page that must be the first subpage of
+ * the folio excluding arm64 for now.

IIUC, pte_batch_hint is always 1 excluding arm64 for now. I'm not sure if this modification will be helpful?

I don't understand how this will be different for arm64 and others. It seems pte_batch_hint with one value > 1 only helps move the PTE pointer faster to finish the call.

Thanks
Barry

#16 lore raw
In reply to Barry Song
Quoted 2 lines · Barry Song

From: Barry Song <[email protected]>

Nit: "mm: make folio_pte_batch available outside of mm/memory.c"

Quoted 2 lines

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial

s/physcial/physical/

Quoted 2 lines

addresses. Let's export it for others to use.

Nit: "Let's make it available in mm/internal.h"

(talking about exporting and modules sounds like EXPORT_SYMBOL)

Quoted 5 lines
Cc Lance Yang <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Yin Fengwei <[email protected]>
Signature · 3 lines
Cheers,

David / dhildenb
#17 lore raw
In reply to Barry Song
Quoted 5 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

It doesn't look exported to me in the patch (and that's a good thing!).

But even for making it non-static you probably want to include that in the series actually making use of it.

mm/internal.h static inline void *folio_raw_mapping(struct folio *folio) · line 97

Quoted 3 lines
mm/internal.h +3 -0
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+		pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+		bool *any_writable);

no need for the extern here.

author
#18 lore raw
Quoted 4 lines · Christoph Hellwig
Quoted 2 lines · Barry Song

From: Barry Song <[email protected]>

Quoted 8 lines
Quoted 3 lines

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

It doesn't look exported to me in the patch (and that's a good thing!).

But even for making it non-static you probably want to include that in the series actually making use of it.

at least two parallel jobs[1][2] (maybe more ) need it right now. Getting this one pulled in early
will help build a common base for them and avoid duplicates&conflicts in them.

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 6 lines

mm/internal.h static inline void *folio_raw_mapping(struct folio *folio) · line 97

Quoted 4 lines
mm/internal.h +3 -0
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+             pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+             bool *any_writable);

no need for the extern here.

Yes. this has been moved to internal.h as "static inline" in v2:
https://lore.kernel.org/linux-mm/[email protected]/

Thanks
Barry

#19 lore raw
Quoted 10 lines · Barry Song
Quoted 8 lines · Christoph Hellwig
Quoted 5 lines · Barry Song

From: Barry Song <[email protected]>

madvise and some others might need folio_pte_batch to check if a range of PTEs are completely mapped to a large folio with contiguous physcial addresses. Let's export it for others to use.

Quoted 6 lines
Quoted 4 lines

It doesn't look exported to me in the patch (and that's a good thing!).

But even for making it non-static you probably want to include that in the series actually making use of it.

at least two parallel jobs[1][2] (maybe more ) need it right now.

+1

After kernel support for anonymous multi-size THP, PTE-mapped THP will no longer be the exception. IMO, folio_pte_batch() is widely useful for checking whether we're mapping all subpages of the large folio or not.

Thanks,
Lance

Quoted 17 lines

Getting this one pulled in early
will help build a common base for them and avoid duplicates&conflicts in them.

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 6 lines

mm/internal.h static inline void *folio_raw_mapping(struct folio *folio) · line 97

Quoted 4 lines
mm/internal.h +3 -0
+extern int folio_pte_batch(struct folio *folio, unsigned long addr,
+             pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags,
+             bool *any_writable);

no need for the extern here.

Yes. this has been moved to internal.h as "static inline" in v2:
https://lore.kernel.org/linux-mm/[email protected]/

Thanks
Barry