Introduce new huge_ptep_get_access_flags() interface

Now we use huge_ptep_get() to get the pte value of a hugetlb page, however it will only return one specific pte value for the CONT-PTE or CONT-PMD size hugetlb on ARM64 system, which can contain seravel continuous pte or pmd entries with same page table attributes. And it will not take into account the subpages' dirty or young bits of a CONT-PTE/PMD size hugetlb page.

So the huge_ptep_get() is inconsistent with huge_ptep_get_and_clear(), which already takes account the dirty or young bits for any subpages in this CONT-PTE/PMD size hugetlb [1]. Meanwhile we can miss dirty or young flags statistics for hugetlb pages with current huge_ptep_get(), such as the gather_hugetlb_stats() function.

Thus introduce a new huge_ptep_get_access_flags() interface and define an ARM64 specific implementation, that will take into account any subpages' dirty or young bits for CONT-PTE/PMD size hugetlb page, for those functions that want to check the dirty and young flags of a hugetlb page.

[1]
https://lore.kernel.org/linux-mm/85bd80b4-b4fd-0d3f-a2e5-149559f2f387@oracl
e.com/

Signed-off-by Baolin Wang <[email protected]>

---
 arch/arm64/include/asm/hugetlb.h |  2 ++
 arch/arm64/mm/hugetlbpage.c      | 24 ++++++++++++++++++++++++
 include/asm-generic/hugetlb.h    |  7 +++++++
 3 files changed, 33 insertions(+)

(quoted diff) +0 -0

b/arch/arm64/include/asm/hugetlb.h index 616b2ca..a473544 100644

arch/arm64/mm/hugetlbpage.c +12 -0

+{
+	int ncontig, i;
+	size_t pgsize;
+	pte_t orig_pte = ptep_get(ptep);
+
+	if (!pte_cont(orig_pte))
+		return orig_pte;
+
+	ncontig = num_contig_ptes(sz, &pgsize);
+
+	for (i = 0; i < ncontig; i++, ptep++) {
+		pte_t pte = ptep_get(ptep);

arch/arm64/mm/hugetlbpage.c +1 -0

+
 /*
  * Changing some bits of contiguous entries requires us to follow a
  * Break-Before-Make approach, breaking the whole contiguous set

include/asm-generic/hugetlb.h +0 -0

index a57d667..bb77fb0 100644

include/asm-generic/hugetlb.h +3 -0

+}
+#endif
+
 #ifndef __HAVE_ARCH_GIGANTIC_PAGE_RUNTIME_SUPPORTED
 static inline bool gigantic_page_runtime_supported(void)
 {

Hi,

As Mike pointed out [1], the huge_ptep_get() will only return one specific pte value for the CONT-PTE or CONT-PMD size hugetlb on ARM64 system, which will not take into account the subpages' dirty or young bits of a CONT-PTE/PMD size hugetlb page. That will make us miss dirty or young flags of a CONT-PTE/PMD size hugetlb page for those functions that want to check the dirty or young flags of a hugetlb page. For example, the gather_hugetlb_stats() will get inaccurate dirty hugetlb page statistics, and the DAMON for hugetlb monitoring will also get inaccurate access statistics.

To fix this issue, one approach is that we can define an ARM64 specific huge_ptep_get() implementation, which will take into account any subpages' dirty or young bits.

However we should add a new parameter for ARM64 specific huge_ptep_get() to check how many continuous PTEs or PMDs in this CONT-PTE/PMD size hugetlb, that means we should convert all the places using huge_ptep_get(), meanwhile most places using huge_ptep_get() did not care about the dirty or young flags at all.

So instead of changing the prototype of huge_ptep_get(), this patch set introduces a new huge_ptep_get_access_flags() interface and define an ARM64 specific implementation, that will take into account any subpages' dirty or young bits for CONT-PTE/PMD size hugetlb page. And we can only change to use huge_ptep_get_access_flags() for those functions that care about the dirty or young flags of a hugetlb page.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Baolin Wang (3):
arm64/hugetlb: Introduce new huge_ptep_get_access_flags() interface fs/proc/task_mmu: Change to use huge_ptep_get_access_flags() mm/damon/vaddr: Change to use huge_ptep_get_access_flags()

 arch/arm64/include/asm/hugetlb.h |  2 ++
 arch/arm64/mm/hugetlbpage.c      | 24 ++++++++++++++++++++++++
 fs/proc/task_mmu.c               |  3 ++-
 include/asm-generic/hugetlb.h    |  7 +++++++
 mm/damon/vaddr.c                 |  5 +++--
 5 files changed, 38 insertions(+), 3 deletions(-)

Signature · 1 line

1.8.3.1

As Mike pointed out [1], the huge_ptep_get() will only return one specific pte value for the CONT-PTE or CONT-PMD size hugetlb on ARM64 system, which will not take into account the subpages' dirty or young bits of a CONT-PTE/PMD size hugetlb page. That will make us miss dirty or young flags of a CONT-PTE/PMD size hugetlb page for those functions that want to check the dirty or young flags of a hugetlb page. For example, the gather_hugetlb_stats() will get inaccurate dirty hugetlb page statistics, and the DAMON for hugetlb monitoring will also get inaccurate access statistics.

To fix this issue, one approach is that we can define an ARM64 specific huge_ptep_get() implementation, which will take into account any subpages' dirty or young bits. However we should add a new parameter for ARM64 specific huge_ptep_get() to check how many continuous PTEs or PMDs in this CONT-PTE/PMD size hugetlb, that means we should convert all the places using huge_ptep_get(), meanwhile most places using huge_ptep_get() did not care about the dirty or young flags at all.

So instead of changing the prototype of huge_ptep_get(), this patch set introduces a new huge_ptep_get_access_flags() interface and define an ARM64 specific implementation, that will take into account any subpages' dirty or young bits for CONT-PTE/PMD size hugetlb page. And we can only change to use huge_ptep_get_access_flags() for those functions that care about the dirty or young flags of a hugetlb page.

Quoted 34 lines · On 2022年5月8日星期日 CST 下午4:58:52 Baolin Wang wrote:

Now we use huge_ptep_get() to get the pte value of a hugetlb page, however it will only return one specific pte value for the CONT-PTE or CONT-PMD size hugetlb on ARM64 system, which can contain seravel continuous pte or pmd entries with same page table attributes. And it will not take into account the subpages' dirty or young bits of a CONT-PTE/PMD size hugetlb page.

So the huge_ptep_get() is inconsistent with huge_ptep_get_and_clear(), which already takes account the dirty or young bits for any subpages in this CONT-PTE/PMD size hugetlb [1]. Meanwhile we can miss dirty or young flags statistics for hugetlb pages with current huge_ptep_get(), such as the gather_hugetlb_stats() function.

Thus introduce a new huge_ptep_get_access_flags() interface and define an ARM64 specific implementation, that will take into account any subpages' dirty or young bits for CONT-PTE/PMD size hugetlb page, for those functions that want to check the dirty and young flags of a hugetlb page.

[1]
https://lore.kernel.org/linux-mm/85bd80b4-b4fd-0d3f-a2e5-149559f2f387@oracl
e.com/

Signed-off-by Baolin Wang <[email protected]>

---
  arch/arm64/include/asm/hugetlb.h |  2 ++
  arch/arm64/mm/hugetlbpage.c      | 24 ++++++++++++++++++++++++
  include/asm-generic/hugetlb.h    |  7 +++++++
  3 files changed, 33 insertions(+)

(quoted diff) +0 -0

b/arch/arm64/include/asm/hugetlb.h index 616b2ca..a473544 100644

arch/arm64/include/asm/hugetlb.h extern pte_t huge_ptep_clear_flush(struct vm_area_struct · line 50

Quoted 7 lines

arch/arm64/include/asm/hugetlb.h +2 -0

@@ -44,6 +44,8 @@ extern pte_t huge_ptep_clear_flush(struct vm_area_struct
 *vma, #define __HAVE_ARCH_HUGE_PTE_CLEAR
  extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr,
  			   pte_t *ptep, unsigned long sz);
+#define __HAVE_ARCH_HUGE_PTEP_GET_ACCESS_FLAGS
+extern pte_t huge_ptep_get_access_flags(pte_t *ptep, unsigned long sz);
  extern void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr,
  				 pte_t *ptep, pte_t pte, unsigned long

sz);

Quoted 9 lines

arch/arm64/include/asm/hugetlb.h +0 -0

  #define set_huge_swap_pte_at set_huge_swap_pte_at

arch/arm64/mm/hugetlbpage.c +0 -0

index ca8e65c..ce39699 100644
@@ -158,6 +158,30 @@ static inline int num_contig_ptes(unsigned long size,
 size_t *pgsize) return contig_ptes;
  }
 

arch/arm64/mm/hugetlbpage.c static inline int num_contig_ptes(unsigned long size, size_t *pgsize) · line 161

Quoted 1 line

arch/arm64/mm/hugetlbpage.c +1 -0

+pte_t huge_ptep_get_access_flags(pte_t *ptep, unsigned long sz)

The function name looks to me that it returns access flags of PTE.

Quoted 13 lines

arch/arm64/mm/hugetlbpage.c +12 -0

+{
+	int ncontig, i;
+	size_t pgsize;
+	pte_t orig_pte = ptep_get(ptep);
+
+	if (!pte_cont(orig_pte))
+		return orig_pte;
+
+	ncontig = num_contig_ptes(sz, &pgsize);
+
+	for (i = 0; i < ncontig; i++, ptep++) {
+		pte_t pte = ptep_get(ptep);

arch/arm64/mm/hugetlbpage.c static inline int num_contig_ptes(unsigned long size, size_t *pgsize) · line 183

Quoted 10 lines

arch/arm64/mm/hugetlbpage.c +10 -0

+
+		if (pte_dirty(pte))
+			orig_pte = pte_mkdirty(orig_pte);
+
+		if (pte_young(pte))
+			orig_pte = pte_mkyoung(orig_pte);
+	}
+
+	return orig_pte;
+}

Not sure whether it's worthy being changed to:

         bool dirty = false, young = false;

         for (i = 0; i < ncontig; i++, ptep++) {
                 pte_t pte = ptep_get(ptep);

                 if (pte_dirty(pte))
                         dirty = true;

                 if (pte_young(pte))
                         young = true;

                 if (dirty && young)
                         break;
         }

         if (dirty)
                 orig_pte = pte_mkdirty(orig_pte);

         if (young)
                 orig_pte = pte_mkyoung(orit_pte);

         return orig_pte;

Quoted 12 lines

arch/arm64/mm/hugetlbpage.c +1 -0

+
  /*
   * Changing some bits of contiguous entries requires us to follow a
   * Break-Before-Make approach, breaking the whole contiguous set

include/asm-generic/hugetlb.h +0 -0

index a57d667..bb77fb0 100644
@@ -150,6 +150,13 @@ static inline pte_t huge_ptep_get(pte_t *ptep)
  }
  #endif
 

include/asm-generic/hugetlb.h static inline pte_t huge_ptep_get(pte_t *ptep) · line 156

Quoted 4 lines

include/asm-generic/hugetlb.h +3 -0

+#ifndef __HAVE_ARCH_HUGE_PTEP_GET_ACCESS_FLAGS
+static inline pte_t huge_ptep_get_access_flags(pte_t *ptep, unsigned long
 sz) +{
+	return ptep_get(ptep);

Should be:
return huge_ptep_get(ptep) ?

Quoted 11 lines · Baolin Wang

Hi,

As Mike pointed out [1], the huge_ptep_get() will only return one specific pte value for the CONT-PTE or CONT-PMD size hugetlb on ARM64 system, which will not take into account the subpages' dirty or young bits of a CONT-PTE/PMD size hugetlb page. That will make us miss dirty or young flags of a CONT-PTE/PMD size hugetlb page for those functions that want to check the dirty or young flags of a hugetlb page. For example, the gather_hugetlb_stats() will get inaccurate dirty hugetlb page statistics, and the DAMON for hugetlb monitoring will also get inaccurate access statistics.

Quoted 2 lines

To fix this issue, one approach is that we can define an ARM64 specific huge_ptep_get() implementation, which will take into account any subpages' dirty or young bits.

IIUC, we could get the page size by page_size(pte_page(pte)). So, how about the following implementation of huge_ptep_get()? Does this work for you?

pte_t huge_ptep_get(pte_t *ptep)

{
	int ncontig, i;
	size_t pgsize;
	pte_t orig_pte = ptep_get(ptep);

	if (!pte_present(orig_pte) || !pte_cont(orig_pte))
		return orig_pte;

	ncontig = num_contig_ptes(page_size(pte_page(orig_pte)), &pgsize);

	for (i = 0; i < ncontig; i++, ptep++) {
		pte_t pte = ptep_get(ptep);

		if (pte_dirty(pte))

			orig_pte = pte_mkdirty(orig_pte);

		if (pte_young(pte))
			orig_pte = pte_mkyoung(orig_pte);
	}

	return orig_pte;
}

Quoted 16 lines · Baolin Wang

As Mike pointed out [1], the huge_ptep_get() will only return one specific pte value for the CONT-PTE or CONT-PMD size hugetlb on ARM64 system, which will not take into account the subpages' dirty or young bits of a CONT-PTE/PMD size hugetlb page. That will make us miss dirty or young flags of a CONT-PTE/PMD size hugetlb page for those functions that want to check the dirty or young flags of a hugetlb page. For example, the gather_hugetlb_stats() will get inaccurate dirty hugetlb page statistics, and the DAMON for hugetlb monitoring will also get inaccurate access statistics.

To fix this issue, one approach is that we can define an ARM64 specific huge_ptep_get() implementation, which will take into account any subpages' dirty or young bits. However we should add a new parameter for ARM64 specific huge_ptep_get() to check how many continuous PTEs or PMDs in this CONT-PTE/PMD size hugetlb, that means we should convert all the places using huge_ptep_get(), meanwhile most places using huge_ptep_get() did not care about the dirty or young flags at all.

Quoted 5 lines

So instead of changing the prototype of huge_ptep_get(), this patch set introduces a new huge_ptep_get_access_flags() interface and define an ARM64 specific implementation, that will take into account any subpages' dirty or young bits for CONT-PTE/PMD size hugetlb page. And we can only change to use huge_ptep_get_access_flags() for those functions that care about the dirty or young flags of a hugetlb page.

I question whether this is the right approach. I understand that different hardware implementations have different requirements here, but at least one that I'm aware of (AMD Zen 2/3) requires that all PTEs that are part of a contig PTE must have identical A/D bits. Now, you could say that's irrelevant because it's x86 and we don't currently support contPTE on x86, but I wouldn't be surprised to see that other hardware has the same requirement.

So what if we make that a Linux requirement? Setting a contPTE dirty or accessed becomes a bit more expensive (although still one/two cachelines, so not really much more expensive than a single write). Then there's no need to change the "get" side of things because they're always identical.

It does mean that we can't take advantage of hardware setting A/D bits, unless hardware can be persuaded to behave this way. I don't have any ARM specs in front of me to check.

Quoted 16 lines · Baolin Wang

As Mike pointed out [1], the huge_ptep_get() will only return one specific pte value for the CONT-PTE or CONT-PMD size hugetlb on ARM64 system, which will not take into account the subpages' dirty or young bits of a CONT-PTE/PMD size hugetlb page. That will make us miss dirty or young flags of a CONT-PTE/PMD size hugetlb page for those functions that want to check the dirty or young flags of a hugetlb page. For example, the gather_hugetlb_stats() will get inaccurate dirty hugetlb page statistics, and the DAMON for hugetlb monitoring will also get inaccurate access statistics.

To fix this issue, one approach is that we can define an ARM64 specific huge_ptep_get() implementation, which will take into account any subpages' dirty or young bits. However we should add a new parameter for ARM64 specific huge_ptep_get() to check how many continuous PTEs or PMDs in this CONT-PTE/PMD size hugetlb, that means we should convert all the places using huge_ptep_get(), meanwhile most places using huge_ptep_get() did not care about the dirty or young flags at all.

Quoted 5 lines

So instead of changing the prototype of huge_ptep_get(), this patch set introduces a new huge_ptep_get_access_flags() interface and define an ARM64 specific implementation, that will take into account any subpages' dirty or young bits for CONT-PTE/PMD size hugetlb page. And we can only change to use huge_ptep_get_access_flags() for those functions that care about the dirty or young flags of a hugetlb page.

I question whether this is the right approach. I understand that different hardware implementations have different requirements here, but at least one that I'm aware of (AMD Zen 2/3) requires that all PTEs that are part of a contig PTE must have identical A/D bits. Now, you could say that's irrelevant because it's x86 and we don't currently support contPTE on x86, but I wouldn't be surprised to see that other hardware has the same requirement.

So what if we make that a Linux requirement? Setting a contPTE dirty or accessed becomes a bit more expensive (although still one/two cachelines, so not really much more expensive than a single write). Then there's no need to change the "get" side of things because they're always identical.

It does mean that we can't take advantage of hardware setting A/D bits, unless hardware can be persuaded to behave this way. I don't have any ARM specs in front of me to check.

Quoted 11 lines

Quoted 11 lines

arch/arm64/mm/hugetlbpage.c +0 -0

index ca8e65c..ce39699 100644
@@ -158,6 +158,30 @@ static inline int num_contig_ptes(unsigned long
 size,
 size_t *pgsize) return contig_ptes;
 
  }
 

Quoted 3 lines

arch/arm64/mm/hugetlbpage.c static inline int num_contig_ptes(unsigned long size, size_t *pgsize) · line 161

Quoted 1 line

arch/arm64/mm/hugetlbpage.c +1 -0

+pte_t huge_ptep_get_access_flags(pte_t *ptep, unsigned long sz)

The function name looks to me that it returns access flags of PTE.

Yes, not a good name. That's why this is a RFC patch set to get more suggestion :)

Maybe huge_ptep_get_with_access_flags()? or do you have some better idea?

Quoted 10 lines

Quoted 10 lines

include/asm-generic/hugetlb.h +0 -0

index a57d667..bb77fb0 100644
@@ -150,6 +150,13 @@ static inline pte_t huge_ptep_get(pte_t *ptep)
 
  }
  #endif
 

Quoted 8 lines

include/asm-generic/hugetlb.h static inline pte_t huge_ptep_get(pte_t *ptep) · line 156

Quoted 5 lines

include/asm-generic/hugetlb.h +2 -0

+#ifndef __HAVE_ARCH_HUGE_PTEP_GET_ACCESS_FLAGS
+static inline pte_t huge_ptep_get_access_flags(pte_t *ptep, unsigned

long
sz) +{

include/asm-generic/hugetlb.h +1 -0

+	return ptep_get(ptep);

Should be:
return huge_ptep_get(ptep) ?

I don't think so. If no ARCH-specific definition, the huge_ptep_get_access_flags() implementation should be same as huge_ptep_get(). Thanks for your comments.

#ifndef __HAVE_ARCH_HUGE_PTEP_GET
static inline pte_t huge_ptep_get(pte_t *ptep)

{
         return ptep_get(ptep);
}

#endif

----------------8<---------------

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

arch/arm64/mm/hugetlbpage.c +0 -0

index ca8e65c..ce39699 100644
@@ -158,6 +158,30 @@ static inline int num_contig_ptes(unsigned long
 size,
 size_t *pgsize) return contig_ptes;
 
   }
 

Quoted 8 lines

Quoted 3 lines

arch/arm64/mm/hugetlbpage.c static inline int num_contig_ptes(unsigned long size, size_t *pgsize) · line 161

Quoted 1 line

arch/arm64/mm/hugetlbpage.c +1 -0

+pte_t huge_ptep_get_access_flags(pte_t *ptep, unsigned long sz)

The function name looks to me that it returns access flags of PTE.

Yes, not a good name. That's why this is a RFC patch set to get more suggestion :)

Maybe huge_ptep_get_with_access_flags()? or do you have some better idea?

I don't have either. "Naming is hard". :)

Quoted 19 lines

Quoted 18 lines

Quoted 14 lines

include/asm-generic/hugetlb.h +2 -0

index a57d667..bb77fb0 100644
@@ -150,6 +150,13 @@ static inline pte_t huge_ptep_get(pte_t *ptep)
 
   }
   #endif
 
+#ifndef __HAVE_ARCH_HUGE_PTEP_GET_ACCESS_FLAGS
+static inline pte_t huge_ptep_get_access_flags(pte_t *ptep, unsigned

long
sz) +{

include/asm-generic/hugetlb.h static inline pte_t huge_ptep_get(pte_t *ptep) · line 156

Quoted 1 line

include/asm-generic/hugetlb.h +1 -0

+	return ptep_get(ptep);

Should be:
return huge_ptep_get(ptep) ?

Quoted 3 lines

I don't think so. If no ARCH-specific definition, the huge_ptep_get_access_flags() implementation should be same as huge_ptep_get(). Thanks for your comments.

If no __HAVE_ARCH_HUGE_PTEP_GET, huge_ptep_get() is same as ptep_get().

Or it's not possible no __HAVE_ARCH_HUGE_PTEP_GET_ACCESS_FLAGS but with __HAVE_ARCH_HUGE_PTEP_GET?