Skip to contentSkip to conversation

optimize for powerpc _PAGE_NUMA

PATCH In review Liu Ping Fan 3 participants 13 messages Dec 4, 2013 – Dec 12, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover optimize for powerpc _PAGE_NUMA

I saw Aneesh had implemented Numa faults on ppc64. Most of them have been reviewed-by. So I just diff mine and his to form some trivial modification.

Based on Aneesh's series "[PATCH -V2 0/5] powerpc: mm: Numa faults support for ppc64"

Liu Ping Fan (3):
powerpc: mm: make _PAGE_NUMA take effect
mm: export numa_migrate_prep()
powerpc: mm: optimize for the correctly placed page

 arch/powerpc/mm/hash_utils_64.c | 32 ++++++++++++++++++++++++++++++++
 include/linux/mm.h              |  2 ++
 2 files changed, 34 insertions(+)
Signature · 1 line
1.8.1.4
#1 lore raw

PATCH 1/3 powerpc: mm: make _PAGE_NUMA take effect

To enable the do_numa_page(), we should not fix _PAGE_NUMA in hash_page(), so bail out for the case of pte_numa().

---
 arch/powerpc/mm/hash_utils_64.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/mm/hash_utils_64.c +1 -1
index fb176e9..9bf1195 100644
@@ -1033,7 +1033,7 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
 
 	/* Get PTE and page size from page tables */
 	ptep = find_linux_pte_or_hugepte(pgdir, ea, &hugeshift);
-	if (ptep == NULL || !pte_present(*ptep)) {
+	if (ptep == NULL || !pte_present(*ptep) || pte_numa(*ptep)) {
 		DBG_LOW(" no PTE !\n");
 		rc = 1;
 		goto bail;
Signature · 1 line
1.8.1.4
#2 lore raw

Re: [PATCH 1/3] powerpc: mm: make _PAGE_NUMA take effect

Quoted 13 lines · Liu Ping Fan

To enable the do_numa_page(), we should not fix _PAGE_NUMA in hash_page(), so bail out for the case of pte_numa().

---
 arch/powerpc/mm/hash_utils_64.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/mm/hash_utils_64.c +0 -0
index fb176e9..9bf1195 100644

arch/powerpc/mm/hash_utils_64.c int hash_page(unsigned long ea, unsigned long access, unsigned long trap) · line 1039

Quoted 8 lines
arch/powerpc/mm/hash_utils_64.c +1 -1
@@ -1033,7 +1033,7 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
 
 	/* Get PTE and page size from page tables */
 	ptep = find_linux_pte_or_hugepte(pgdir, ea, &hugeshift);
-	if (ptep == NULL || !pte_present(*ptep)) {
+	if (ptep == NULL || !pte_present(*ptep) || pte_numa(*ptep)) {
 		DBG_LOW(" no PTE !\n");
 		rc = 1;
 		goto bail;

why ? , All the hash routines do check for _PAGE_PRESENT via access variable.

-aneesh

#3 lore raw

Re: [PATCH 1/3] powerpc: mm: make _PAGE_NUMA take effect

Quoted 4 lines · Aneesh Kumar K.V
Quoted 2 lines · Liu Ping Fan

To enable the do_numa_page(), we should not fix _PAGE_NUMA in hash_page(), so bail out for the case of pte_numa().

For some reason I don't have 2/3 and 3/3 in my mbox (though I do have them on patchwork) so I'll reply to this one.

Overall, your statement that this is a faster path needs to be backed up with numbers.

The code is complicated enough as it-is, such additional mess in the low level hashing code requires a good justification, and also a demonstration that it doesn't add overhead to the normal hash path.

Cheers,
Ben.

#4 lore raw

Re: [PATCH 1/3] powerpc: mm: make _PAGE_NUMA take effect

Quoted 9 lines · Benjamin Herrenschmidt
Quoted 4 lines · Aneesh Kumar K.V
Quoted 2 lines · Liu Ping Fan

To enable the do_numa_page(), we should not fix _PAGE_NUMA in hash_page(), so bail out for the case of pte_numa().

For some reason I don't have 2/3 and 3/3 in my mbox (though I do have them on patchwork) so I'll reply to this one.

Quoted 7 lines

Overall, your statement that this is a faster path needs to be backed up with numbers.

The code is complicated enough as it-is, such additional mess in the low level hashing code requires a good justification, and also a demonstration that it doesn't add overhead to the normal hash path.

For the test, is it ok to have an user application to copy page where all page are PG_mlocked?

Thanks and regards,
Pingfan

#5 lore raw

Re: [PATCH 1/3] powerpc: mm: make _PAGE_NUMA take effect

Quoted 14 lines · Liu ping fan
Quoted 12 lines · Benjamin Herrenschmidt
Quoted 4 lines · Aneesh Kumar K.V
Quoted 2 lines · Liu Ping Fan

To enable the do_numa_page(), we should not fix _PAGE_NUMA in hash_page(), so bail out for the case of pte_numa().

For some reason I don't have 2/3 and 3/3 in my mbox (though I do have them on patchwork) so I'll reply to this one.

Overall, your statement that this is a faster path needs to be backed up with numbers.

Quoted 6 lines
Quoted 4 lines

The code is complicated enough as it-is, such additional mess in the low level hashing code requires a good justification, and also a demonstration that it doesn't add overhead to the normal hash path.

For the test, is it ok to have an user application to copy page where all page are PG_mlocked?

If that specific scenario is relevant in practice, then yes, though also demonstrate the lack of regression with some more normal path such as a kernel compile.

Cheers,
Ben.

#6 lore raw

Re: [PATCH 1/3] powerpc: mm: make _PAGE_NUMA take effect

In reply to Aneesh Kumar K.V
Quoted 16 lines · Aneesh Kumar K.V
Quoted 14 lines · Liu Ping Fan

To enable the do_numa_page(), we should not fix _PAGE_NUMA in hash_page(), so bail out for the case of pte_numa().

---
 arch/powerpc/mm/hash_utils_64.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/mm/hash_utils_64.c +0 -0
index fb176e9..9bf1195 100644
@@ -1033,7 +1033,7 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
 
Quoted 11 lines

arch/powerpc/mm/hash_utils_64.c int hash_page(unsigned long ea, unsigned long access, unsigned long trap) · line 1039

Quoted 7 lines
arch/powerpc/mm/hash_utils_64.c +1 -1
      /* Get PTE and page size from page tables */
      ptep = find_linux_pte_or_hugepte(pgdir, ea, &hugeshift);
-     if (ptep == NULL || !pte_present(*ptep)) {
+     if (ptep == NULL || !pte_present(*ptep) || pte_numa(*ptep)) {
              DBG_LOW(" no PTE !\n");
              rc = 1;
              goto bail;

why ? , All the hash routines do check for _PAGE_PRESENT via access variable.

Going through __hash_page_4K(4k on 4k HW), I do not find such check. Am I wrong? Or I will send out a patch to fix that.

Thanks and regards,
Pingfan

Quoted 2 lines

-aneesh

#7 lore raw

Re: [PATCH 1/3] powerpc: mm: make _PAGE_NUMA take effect

Quoted 5 lines · Liu ping fan
Quoted 3 lines

why ? , All the hash routines do check for _PAGE_PRESENT via access variable.

Going through __hash_page_4K(4k on 4k HW), I do not find such check. Am I wrong? Or I will send out a patch to fix that.

We pass a bitmask of flags to check which are tested by doing an "andc" of the PTE on that mask and checking if anything is left...

Ben.

#8 lore raw

Re: [PATCH 1/3] powerpc: mm: make _PAGE_NUMA take effect

Quoted 10 lines · Benjamin Herrenschmidt
Quoted 5 lines · Liu ping fan
Quoted 3 lines

why ? , All the hash routines do check for _PAGE_PRESENT via access variable.

Going through __hash_page_4K(4k on 4k HW), I do not find such check. Am I wrong? Or I will send out a patch to fix that.

We pass a bitmask of flags to check which are tested by doing an "andc" of the PTE on that mask and checking if anything is left...

Oh, see it, thank you very much

Regards,
Pingfan

#9 lore raw

PATCH 2/3 mm: export numa_migrate_prep()

In reply to Liu Ping Fan

powerpc will use it in fast path.

---
 include/linux/mm.h | 2 ++
 1 file changed, 2 insertions(+)
include/linux/mm.h +2 -0
index 5ab0e22..420fb77 100644
@@ -1092,6 +1092,8 @@ extern unsigned long change_protection(struct vm_area_struct *vma, unsigned long
 extern int mprotect_fixup(struct vm_area_struct *vma,
 			  struct vm_area_struct **pprev, unsigned long start,
 			  unsigned long end, unsigned long newflags);
+extern int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
+				unsigned long addr, int page_nid);
 
 /*
  * doesn't attempt to fault and will return short.
Signature · 1 line
1.8.1.4
#10 lore raw

PATCH 3/3 powerpc: mm: optimize for the correctly placed page

In reply to Liu Ping Fan

The period check of _PAGE_NUMA can probably trigger the check on the correctly placed page. For this case, we can just insert hpte and do fast exception return.

---
 arch/powerpc/mm/hash_utils_64.c | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)
arch/powerpc/mm/hash_utils_64.c +33 -1
index 9bf1195..735678c 100644
@@ -965,6 +965,10 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
 	const struct cpumask *tmp;
 	int rc, user_region = 0, local = 0;
 	int psize, ssize;
+	pte_t old, new;
+	struct vm_area_struct *vma;
+	int page_nid, target_nid;
+	struct page *test_page;
 
 	DBG_LOW("hash_page(ea=%016lx, access=%lx, trap=%lx\n",
 		ea, access, trap);
@@ -1033,12 +1037,40 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
 
 	/* Get PTE and page size from page tables */
 	ptep = find_linux_pte_or_hugepte(pgdir, ea, &hugeshift);
-	if (ptep == NULL || !pte_present(*ptep) || pte_numa(*ptep)) {
+	if (ptep == NULL || !pte_present(*ptep)) {
 		DBG_LOW(" no PTE !\n");
 		rc = 1;
 		goto bail;
 	}
 
+	old = pte_val(*ptep);
+	if (pte_numa(old)) {
+		/* If fail to lock, let do_page_fault() to handle it */
+		if (down_read_trylock(&mm->mmap_sem)) {
+			vma = find_vma(mm, ea);
+			up_read(&mm->mmap_sem);
+			test_page = pte_page(old);
+			page_nid = page_to_nid(test_page);
+			target_nid = numa_migrate_prep(test_page, vma, ea,
+						page_nid);
+			if (target_nid < 0) {
+				new = pte_mknonnuma(old);
+				/* If ptep is modified under us,
+				 * just retry the access
+				 */
+				if (unlikely(cmpxchg(ptep, old, new) != old)) {
+					put_page(test_page);
+					return 0;
+				}
+				put_page(test_page);
+			}
+		} else {
+				put_page(test_page);
+				rc = 1;
+				goto bail;
+		}
+	}
+
 	/* Add _PAGE_PRESENT to the required access perm */
 	access |= _PAGE_PRESENT;
 
Signature · 1 line
1.8.1.4
#11 lore raw

Re: [PATCH 3/3] powerpc: mm: optimize for the correctly placed page

Quoted 3 lines · Liu Ping Fan

The period check of _PAGE_NUMA can probably trigger the check on the correctly placed page. For this case, we can just insert hpte and do fast exception return.

I still don't understand why we need to handle numa faults in hash page ? Are you trying to optimize the code path ? If so can you explain the benefits ? Some numbers showing it is helping ?

Quoted 26 lines
---
 arch/powerpc/mm/hash_utils_64.c | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)
arch/powerpc/mm/hash_utils_64.c +4 -1
index 9bf1195..735678c 100644
@@ -965,6 +965,10 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
 	const struct cpumask *tmp;
 	int rc, user_region = 0, local = 0;
 	int psize, ssize;
+	pte_t old, new;
+	struct vm_area_struct *vma;
+	int page_nid, target_nid;
+	struct page *test_page;
 
 	DBG_LOW("hash_page(ea=%016lx, access=%lx, trap=%lx\n",
 		ea, access, trap);
@@ -1033,12 +1037,40 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
 
 	/* Get PTE and page size from page tables */
 	ptep = find_linux_pte_or_hugepte(pgdir, ea, &hugeshift);
-	if (ptep == NULL || !pte_present(*ptep) || pte_numa(*ptep)) {

arch/powerpc/mm/hash_utils_64.c int hash_page(unsigned long ea, unsigned long access, unsigned long trap) · line 1049

Quoted 10 lines
arch/powerpc/mm/hash_utils_64.c +5 -0
+	if (ptep == NULL || !pte_present(*ptep)) {
 		DBG_LOW(" no PTE !\n");
 		rc = 1;
 		goto bail;
 	}
 
+	old = pte_val(*ptep);
+	if (pte_numa(old)) {
+		/* If fail to lock, let do_page_fault() to handle it */
+		if (down_read_trylock(&mm->mmap_sem)) {

hmm is that something we want to do in hash_page ?

Quoted 17 lines
arch/powerpc/mm/hash_utils_64.c +17 -0
+			vma = find_vma(mm, ea);
+			up_read(&mm->mmap_sem);
+			test_page = pte_page(old);
+			page_nid = page_to_nid(test_page);
+			target_nid = numa_migrate_prep(test_page, vma, ea,
+						page_nid);
+			if (target_nid < 0) {
+				new = pte_mknonnuma(old);
+				/* If ptep is modified under us,
+				 * just retry the access
+				 */
+				if (unlikely(cmpxchg(ptep, old, new) != old)) {
+					put_page(test_page);
+					return 0;
+				}
+				put_page(test_page);
+			}

arch/powerpc/mm/hash_utils_64.c int hash_page(unsigned long ea, unsigned long access, unsigned long trap) · line 1076

Quoted 10 lines
arch/powerpc/mm/hash_utils_64.c +7 -0
+		} else {
+				put_page(test_page);
+				rc = 1;
+				goto bail;
+		}
+	}
+
 	/* Add _PAGE_PRESENT to the required access perm */
 	access |= _PAGE_PRESENT;
 

-aneesh

#12 lore raw

Re: [PATCH 3/3] powerpc: mm: optimize for the correctly placed page

Quoted 10 lines · Aneesh Kumar K.V
Quoted 3 lines · Liu Ping Fan

The period check of _PAGE_NUMA can probably trigger the check on the correctly placed page. For this case, we can just insert hpte and do fast exception return.

I still don't understand why we need to handle numa faults in hash page ? Are you trying to optimize the code path ? If so can you explain the benefits ? Some numbers showing it is helping ?

When return from hash_page(), we will take fast_exc_return_irq, while from do_page_fault(), we take ret_from_except.
As the "fast" implies that there are more complicated logic to sync the interrupt states in ret_from_except, which cost much. Do you think so?

Quoted 32 lines
Quoted 32 lines
---
 arch/powerpc/mm/hash_utils_64.c | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)
arch/powerpc/mm/hash_utils_64.c +5 -1
index 9bf1195..735678c 100644
@@ -965,6 +965,10 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
      const struct cpumask *tmp;
      int rc, user_region = 0, local = 0;
      int psize, ssize;
+     pte_t old, new;
+     struct vm_area_struct *vma;
+     int page_nid, target_nid;
+     struct page *test_page;
 
      DBG_LOW("hash_page(ea=%016lx, access=%lx, trap=%lx\n",
              ea, access, trap);
@@ -1033,12 +1037,40 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
 
      /* Get PTE and page size from page tables */
      ptep = find_linux_pte_or_hugepte(pgdir, ea, &hugeshift);
-     if (ptep == NULL || !pte_present(*ptep) || pte_numa(*ptep)) {
+     if (ptep == NULL || !pte_present(*ptep)) {
              DBG_LOW(" no PTE !\n");
              rc = 1;
              goto bail;
      }
 
Quoted 7 lines

arch/powerpc/mm/hash_utils_64.c int hash_page(unsigned long ea, unsigned long access, unsigned long trap) · line 1049

Quoted 4 lines
arch/powerpc/mm/hash_utils_64.c +4 -0
+     old = pte_val(*ptep);
+     if (pte_numa(old)) {
+             /* If fail to lock, let do_page_fault() to handle it */
+             if (down_read_trylock(&mm->mmap_sem)) {

hmm is that something we want to do in hash_page ?

Yes, the function has no relation with hash. But I think it depends on whether it is worth to optimize or not.

Thanks and regards,
Pingfan

Quoted 30 lines
Quoted 17 lines
arch/powerpc/mm/hash_utils_64.c +17 -0
+                     vma = find_vma(mm, ea);
+                     up_read(&mm->mmap_sem);
+                     test_page = pte_page(old);
+                     page_nid = page_to_nid(test_page);
+                     target_nid = numa_migrate_prep(test_page, vma, ea,
+                                             page_nid);
+                     if (target_nid < 0) {
+                             new = pte_mknonnuma(old);
+                             /* If ptep is modified under us,
+                              * just retry the access
+                              */
+                             if (unlikely(cmpxchg(ptep, old, new) != old)) {
+                                     put_page(test_page);
+                                     return 0;
+                             }
+                             put_page(test_page);
+                     }

arch/powerpc/mm/hash_utils_64.c int hash_page(unsigned long ea, unsigned long access, unsigned long trap) · line 1076

Quoted 10 lines
arch/powerpc/mm/hash_utils_64.c +7 -0
+             } else {
+                             put_page(test_page);
+                             rc = 1;
+                             goto bail;
+             }
+     }
+
      /* Add _PAGE_PRESENT to the required access perm */
      access |= _PAGE_PRESENT;
 

-aneesh