Numabalancing preserve write fix
Subsystems (6)
v2 cover Numabalancing preserve write fix
This patch series address an issue w.r.t THP migration and autonuma preserve write feature. migrate_misplaced_transhuge_page() cannot deal with concurrent modification of the page. It does a page copy without following the migration pte sequence. IIUC, this was done to keep the migration simpler and at the time of implemenation we didn't had THP page cache which would have required a more elaborate migration scheme. That means thp autonuma migration expect the protnone with saved write to be done such that both kernel and user cannot update
the page content. This patch series enables archs like ppc64 to do that. We are good with the hash translation mode with the current code, because we never create a hardware page table entry for a protnone pte.
Changes from V1:
* Update the patch so that it apply cleanly to upstream.
* Add acked-by from Michael Neuling
Aneesh Kumar K.V (2):
mm/autonuma: Let architecture override how the write bit should be
stashed in a protnone pte.
powerpc/mm/autonuma: Switch ppc64 to its own implementeation of saved
write
arch/powerpc/include/asm/book3s/64/mmu-hash.h | 3 +++ arch/powerpc/include/asm/book3s/64/pgtable.h | 32 +++++++++++++++++++++++++-- include/asm-generic/pgtable.h | 16 ++++++++++++++ mm/huge_memory.c | 4 ++-- mm/memory.c | 2 +- mm/mprotect.c | 4 ++-- 6 files changed, 54 insertions(+), 7 deletions(-)
Signature · 7 lines
2.7.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 1/2 mm/autonuma: Let architecture override how the write bit should be stashed in a protnone pte.
Autonuma preserves the write permission across numa fault to avoid taking a writefault after a numa fault (Commit: b191f9b106ea " mm: numa: preserve PTE write permissions across a NUMA hinting fault"). Architecture can implement protnone in different ways and some may choose to implement that by clearing Read/ Write/Exec bit of pte. Setting the write bit on such pte can result in wrong behaviour. Fix this up by allowing arch to override how to save the write bit on a protnone pte.
--- include/asm-generic/pgtable.h | 16 ++++++++++++++++ mm/huge_memory.c | 4 ++-- mm/memory.c | 2 +- mm/mprotect.c | 4 ++-- 4 files changed, 21 insertions(+), 5 deletions(-)
include/asm-generic/pgtable.h +16 -0
@@ -192,6 +192,22 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres }#endif +#ifndef pte_savedwrite +#define pte_savedwrite pte_write +#endif + +#ifndef pte_mk_savedwrite +#define pte_mk_savedwrite pte_mkwrite +#endif + +#ifndef pmd_savedwrite +#define pmd_savedwrite pmd_write +#endif + +#ifndef pmd_mk_savedwrite +#define pmd_mk_savedwrite pmd_mkwrite +#endif +#ifndef __HAVE_ARCH_PMDP_SET_WRPROTECT#ifdef CONFIG_TRANSPARENT_HUGEPAGE static inline void pmdp_set_wrprotect(struct mm_struct *mm,
mm/huge_memory.c +2 -2
@@ -1300,7 +1300,7 @@ int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd) goto out; clear_pmdnuma:BUG_ON (!PageLocked (page)); - was_writable =pmd_write (pmd); + was_writable =pmd_savedwrite (pmd); pmd =pmd_modify (pmd, vma->vm_page_prot); pmd =pmd_mkyoung (pmd);if (was_writable)
@@ -1555,7 +1555,7 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, entry =pmdp_huge_get_and_clear_notify (mm, addr, pmd); entry =pmd_modify (entry, newprot);if (preserve_write) - entry =pmd_mkwrite (entry); + entry =pmd_mk_savedwrite (entry); ret =HPAGE_PMD_NR ;set_pmd_at (mm, addr, pmd, entry); BUG_ON(vma_is_anonymous(vma) && !preserve_write &&
mm/memory.c +1 -1
@@ -3388,7 +3388,7 @@ static int do_numa_page(struct vm_fault *vmf)int target_nid;bool migrated = false;pte_t pte; -bool was_writable =pte_write (vmf->orig_pte); +bool was_writable =pte_savedwrite (vmf->orig_pte);int flags =0 ;/*
mm/mprotect.c +2 -2
@@ -113,13 +113,13 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, ptent =ptep_modify_prot_start (mm, addr, pte); ptent =pte_modify (ptent, newprot);if (preserve_write) - ptent =pte_mkwrite (ptent); + ptent =pte_mk_savedwrite (ptent);/* Avoid taking write faults for known dirty pages */ if (dirty_accountable &&pte_dirty (ptent) && (pte_soft_dirty (ptent) || !(vma->vm_flags &VM_SOFTDIRTY ))) { - ptent =pte_mkwrite (ptent); + ptent =pte_mk_savedwrite (ptent); }ptep_modify_prot_commit (mm, addr, pte, ptent); pages++;
Signature · 7 lines
2.7.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH V2 1/2] mm/autonuma: Let architecture override how the write bit should be stashed in a protnone pte.
Quoted 7 lines · Aneesh Kumar K.V
Autonuma preserves the write permission across numa fault to avoid taking a writefault after a numa fault (Commit:
b191f9b106ea" mm: numa: preserve PTE write permissions across a NUMA hinting fault"). Architecture can implement protnone in different ways and some may choose to implement that by clearing Read/ Write/Exec bit of pte. Setting the write bit on such pte can result in wrong behaviour. Fix this up by allowing arch to override how to save the write bit on a protnone pte.
This is pretty obviously a nop on arches that don't implement the new hooks, but it'd still be good to get an ack from someone in mm land before I merge it.
cheers
Quoted 93 lines
Acked-by Michael Neuling <[email protected]>Signed-off-by Aneesh Kumar K.V <[email protected]>--- include/asm-generic/pgtable.h | 16 ++++++++++++++++ mm/huge_memory.c | 4 ++-- mm/memory.c | 2 +- mm/mprotect.c | 4 ++-- 4 files changed, 21 insertions(+), 5 deletions(-)include/asm-generic/pgtable.h +16
-0@@ -192,6 +192,22 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres }#endif +#ifndef pte_savedwrite +#define pte_savedwrite pte_write +#endif + +#ifndef pte_mk_savedwrite +#define pte_mk_savedwrite pte_mkwrite +#endif + +#ifndef pmd_savedwrite +#define pmd_savedwrite pmd_write +#endif + +#ifndef pmd_mk_savedwrite +#define pmd_mk_savedwrite pmd_mkwrite +#endif +#ifndef __HAVE_ARCH_PMDP_SET_WRPROTECT#ifdef CONFIG_TRANSPARENT_HUGEPAGE static inline void pmdp_set_wrprotect(struct mm_struct *mm,mm/huge_memory.c +2
-2@@ -1300,7 +1300,7 @@ int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd) goto out; clear_pmdnuma:BUG_ON (!PageLocked (page)); - was_writable =pmd_write (pmd); + was_writable =pmd_savedwrite (pmd); pmd =pmd_modify (pmd, vma->vm_page_prot); pmd =pmd_mkyoung (pmd);if (was_writable)@@ -1555,7 +1555,7 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, entry =pmdp_huge_get_and_clear_notify (mm, addr, pmd); entry =pmd_modify (entry, newprot);if (preserve_write) - entry =pmd_mkwrite (entry); + entry =pmd_mk_savedwrite (entry); ret =HPAGE_PMD_NR ;set_pmd_at (mm, addr, pmd, entry); BUG_ON(vma_is_anonymous(vma) && !preserve_write &&mm/memory.c +1
-1@@ -3388,7 +3388,7 @@ static int do_numa_page(struct vm_fault *vmf)int target_nid;bool migrated = false;pte_t pte; -bool was_writable =pte_write (vmf->orig_pte); +bool was_writable =pte_savedwrite (vmf->orig_pte);int flags =0 ;/* mm/mprotect.c +2
-2@@ -113,13 +113,13 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, ptent =ptep_modify_prot_start (mm, addr, pte); ptent =pte_modify (ptent, newprot);if (preserve_write) - ptent =pte_mkwrite (ptent); + ptent =pte_mk_savedwrite (ptent);/* Avoid taking write faults for known dirty pages */ if (dirty_accountable &&pte_dirty (ptent) && (pte_soft_dirty (ptent) || !(vma->vm_flags &VM_SOFTDIRTY ))) { - ptent =pte_mkwrite (ptent); + ptent =pte_mk_savedwrite (ptent); }ptep_modify_prot_commit (mm, addr, pte, ptent); pages++;Signature · 1 line
2.7.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH V2 1/2] mm/autonuma: Let architecture override how the write bit should be stashed in a protnone pte.
Quoted 9 lines · Michael Ellerman
Quoted 7 lines · Aneesh Kumar K.V
Autonuma preserves the write permission across numa fault to avoid taking a writefault after a numa fault (Commit:
b191f9b106ea" mm: numa: preserve PTE write permissions across a NUMA hinting fault"). Architecture can implement protnone in different ways and some may choose to implement that by clearing Read/ Write/Exec bit of pte. Setting the write bit on such pte can result in wrong behaviour. Fix this up by allowing arch to override how to save the write bit on a protnone pte.
Quoted 3 lines
This is pretty obviously a nop on arches that don't implement the new hooks, but it'd still be good to get an ack from someone in mm land before I merge it.
To get it apply cleanly you may need
http://ozlabs.org/~akpm/mmots/broken-out/mm-autonuma-dont-use-set_pte_at-when-updating-protnone-ptes.patch
http://ozlabs.org/~akpm/mmots/broken-out/mm-autonuma-dont-use-set_pte_at-when-updating-protnone-ptes-fix.patch
They are strictly not needed after the saved write patch. But I didn't request to drop them, because the patch helps us to get closer to the goal of no ste_pte_at() call on present ptes.
-aneesh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH V2 1/2] mm/autonuma: Let architecture override how the write bit should be stashed in a protnone pte.
Quoted 10 lines · Aneesh Kumar K.V
Quoted 9 lines · Michael Ellerman
Quoted 7 lines · Aneesh Kumar K.V
Autonuma preserves the write permission across numa fault to avoid taking a writefault after a numa fault (Commit:
b191f9b106ea" mm: numa: preserve PTE write permissions across a NUMA hinting fault"). Architecture can implement protnone in different ways and some may choose to implement that by clearing Read/ Write/Exec bit of pte. Setting the write bit on such pte can result in wrong behaviour. Fix this up by allowing arch to override how to save the write bit on a protnone pte.
Quoted 8 lines
Quoted 3 lines
This is pretty obviously a nop on arches that don't implement the new hooks, but it'd still be good to get an ack from someone in mm land before I merge it.
To get it apply cleanly you may need
http://ozlabs.org/~akpm/mmots/broken-out/mm-autonuma-dont-use-set_pte_at-when-updating-protnone-ptes.patch
http://ozlabs.org/~akpm/mmots/broken-out/mm-autonuma-dont-use-set_pte_at-when-updating-protnone-ptes-fix.patch
Ah OK, I missed those.
In that case these two should probably go via Andrew's tree.
cheers
Andrew Morton
maintainerRe: [PATCH V2 1/2] mm/autonuma: Let architecture override how the write bit should be stashed in a protnone pte.
Quoted 17 lines · On Tue, 14 Feb 2017 21:59:23 +1100 Michael Ellerman <[email protected]>…
Quoted 15 lines · Aneesh Kumar K.V
Quoted 12 lines · Michael Ellerman
Quoted 7 lines · Aneesh Kumar K.V
Autonuma preserves the write permission across numa fault to avoid taking a writefault after a numa fault (Commit:
b191f9b106ea" mm: numa: preserve PTE write permissions across a NUMA hinting fault"). Architecture can implement protnone in different ways and some may choose to implement that by clearing Read/ Write/Exec bit of pte. Setting the write bit on such pte can result in wrong behaviour. Fix this up by allowing arch to override how to save the write bit on a protnone pte.This is pretty obviously a nop on arches that don't implement the new hooks, but it'd still be good to get an ack from someone in mm land before I merge it.
Quoted 7 lines
Quoted 3 lines
To get it apply cleanly you may need
http://ozlabs.org/~akpm/mmots/broken-out/mm-autonuma-dont-use-set_pte_at-when-updating-protnone-ptes.patch
http://ozlabs.org/~akpm/mmots/broken-out/mm-autonuma-dont-use-set_pte_at-when-updating-protnone-ptes-fix.patchAh OK, I missed those.
In that case these two should probably go via Andrew's tree.
Done. But mm-autonuma-dont-use-set_pte_at-when-updating-protnone-ptes.patch is on hold because Aneesh saw a testing issue, so these two are also on hold.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 2/2 powerpc/mm/autonuma: Switch ppc64 to its own implementeation of saved write
In reply to Aneesh Kumar K.VWith this our protnone becomes a present pte with READ/WRITE/EXEC bit cleared. By default we also set _PAGE_PRIVILEGED on such pte. This is now used to help us identify a protnone pte that as saved write bit. For such pte, we will clear the _PAGE_PRIVILEGED bit. The pte still remain non-accessible from both user and kernel.
--- arch/powerpc/include/asm/book3s/64/mmu-hash.h | 3 +++ arch/powerpc/include/asm/book3s/64/pgtable.h | 32 +++++++++++++++++++++++++-- 2 files changed, 33 insertions(+), 2 deletions(-)
arch/powerpc/include/asm/book3s/64/mmu-hash.h +3 -0
@@ -16,6 +16,9 @@#include <asm/page.h> #include <asm/bug.h> +#ifndef __ASSEMBLY__ +#include <linux/mmdebug.h> +#endif /* * This is necessary to get the definition of PGTABLE_RANGE which we * need for various slices related matters. Note that this isn't the
arch/powerpc/include/asm/book3s/64/pgtable.h +30 -2
@@ -441,8 +441,8 @@ static inline pte_t pte_clear_soft_dirty(pte_t pte)*/ static inline int pte_protnone (pte_t pte) { -return (pte_raw (pte) &cpu_to_be64 (_PAGE_PRESENT | _PAGE_PRIVILEGED)) == -cpu_to_be64 (_PAGE_PRESENT | _PAGE_PRIVILEGED); +return (pte_raw (pte) &cpu_to_be64 (_PAGE_PRESENT | _PAGE_RWX)) == +cpu_to_be64 (_PAGE_PRESENT); }#endif /* CONFIG_NUMA_BALANCING */
@@ -512,6 +512,32 @@ static inline pte_t pte_mkhuge(pte_t pte)return pte; } +#define pte_mk_savedwrite pte_mk_savedwrite +static inline pte_t pte_mk_savedwrite (pte_t pte) +{ +/* +* Used by Autonuma subsystem to preserve the write bit +* while marking the pte PROT_NONE. Only allow this +* on PROT_NONE pte +*/ +VM_BUG_ON ((pte_raw (pte) &cpu_to_be64 (_PAGE_PRESENT | _PAGE_RWX | _PAGE_PRIVILEGED)) != +cpu_to_be64 (_PAGE_PRESENT | _PAGE_PRIVILEGED)); +return __pte (pte_val (pte) & ~_PAGE_PRIVILEGED); +} + +#define pte_savedwrite pte_savedwrite +static inline bool pte_savedwrite (pte_t pte) +{ +/* +* Saved write ptes are prot none ptes that doesn't have +* privileged bit sit. We mark prot none as one which has +* present and pviliged bit set and RWX cleared. To mark +* protnone which used to have _PAGE_WRITE set we clear +* the privileged bit. +*/ +return !(pte_raw (pte) &cpu_to_be64 (_PAGE_RWX | _PAGE_PRIVILEGED)); +} +static inline pte_t pte_modify (pte_t pte,pgprot_t newprot) {/* FIXME!! check whether this need to be a conditional */
@@ -873,6 +899,7 @@ static inline pte_t *pmdp_ptep(pmd_t *pmd)#define pmd_mkclean (pmd) pte_pmd(pte_mkclean(pmd_pte(pmd)))#define pmd_mkyoung (pmd) pte_pmd(pte_mkyoung(pmd_pte(pmd)))#define pmd_mkwrite (pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd))) +#define pmd_mk_savedwrite (pmd) pte_pmd(pte_mk_savedwrite(pmd_pte(pmd)))#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY #define pmd_soft_dirty (pmd) pte_soft_dirty(pmd_pte(pmd))
@@ -889,6 +916,7 @@ static inline int pmd_protnone(pmd_t pmd)#define __HAVE_ARCH_PMD_WRITE#define pmd_write (pmd) pte_write(pmd_pte(pmd)) +#define pmd_savedwrite (pmd) pte_savedwrite(pmd_pte(pmd))#ifdef CONFIG_TRANSPARENT_HUGEPAGE extern pmd_t pfn_pmd (unsigned long pfn,pgprot_t pgprot);
Signature · 7 lines
2.7.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Michael Ellerman
reviewerRe: [PATCH V2 2/2] powerpc/mm/autonuma: Switch ppc64 to its own implementeation of saved write
arch/powerpc/include/asm/book3s/64/mmu-hash.h line 21
Quoted 11 lines · Aneesh Kumar K.V
arch/powerpc/include/asm/book3s/64/mmu-hash.h +3
-0@@ -16,6 +16,9 @@#include <asm/page.h> #include <asm/bug.h> +#ifndef __ASSEMBLY__ +#include <linux/mmdebug.h> +#endif
I assume that's for the VM_BUG_ON() you add below. But if so wouldn't the #include be better placed in book3s/64/pgtable.h also?
Quoted 11 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +3
-0@@ -512,6 +512,32 @@ static inline pte_t pte_mkhuge(pte_t pte)return pte; } +#define pte_mk_savedwrite pte_mk_savedwrite +static inline pte_t pte_mk_savedwrite (pte_t pte) +{
arch/powerpc/include/asm/book3s/64/pgtable.h static inline pte_t pte_mkhuge(pte_t pte) · line 527
Quoted 10 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +10
-0+/* +* Used by Autonuma subsystem to preserve the write bit +* while marking the pte PROT_NONE. Only allow this +* on PROT_NONE pte +*/ +VM_BUG_ON ((pte_raw (pte) &cpu_to_be64 (_PAGE_PRESENT | _PAGE_RWX | _PAGE_PRIVILEGED)) != +cpu_to_be64 (_PAGE_PRESENT | _PAGE_PRIVILEGED)); +return __pte (pte_val (pte) & ~_PAGE_PRIVILEGED); +} +
cheers
Re: [PATCH V2 2/2] powerpc/mm/autonuma: Switch ppc64 to its own implementeation of saved write
Quoted 9 lines · Michael Ellerman
Quoted 8 lines · Aneesh Kumar K.V
arch/powerpc/include/asm/book3s/64/mmu-hash.h +0
-0@@ -16,6 +16,9 @@#include <asm/page.h> #include <asm/bug.h>
Quoted 6 lines
arch/powerpc/include/asm/book3s/64/mmu-hash.h line 21
Quoted 3 lines
arch/powerpc/include/asm/book3s/64/mmu-hash.h +3
-0+#ifndef __ASSEMBLY__ +#include <linux/mmdebug.h> +#endif I assume that's for the
VM_BUG_ON()you add below. But if so wouldn't the #include be better placed inbook3s/64/pgtable.halso?
mmu-hash.h has got a hack that is explained below
#ifndef __ASSEMBLY__
#include <linux/mmdebug.h>
#endif
/*
* This is necessary to get the definition of PGTABLE_RANGE which we
* need for various slices related matters. Note that this isn't the
* complete pgtable.h but only a portion of it.
*/
#include <asm/book3s/64/pgtable.h>
This is the only place where we do that book3s/64/pgtable.h include this way. Everybody should include asm/pgable.h which picks the righ version based on different config option.
#
Quoted 9 lines
Quoted 9 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +0
-0@@ -512,6 +512,32 @@ static inline pte_t pte_mkhuge(pte_t pte)return pte; }
Quoted 16 lines
arch/powerpc/include/asm/book3s/64/pgtable.h static inline pte_t pte_mkhuge(pte_t pte) · line 527
Quoted 13 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +13
-0+#define pte_mk_savedwrite pte_mk_savedwrite +static inline pte_t pte_mk_savedwrite (pte_t pte) +{ +/* +* Used by Autonuma subsystem to preserve the write bit +* while marking the pte PROT_NONE. Only allow this +* on PROT_NONE pte +*/ +VM_BUG_ON ((pte_raw (pte) &cpu_to_be64 (_PAGE_PRESENT | _PAGE_RWX | _PAGE_PRIVILEGED)) != +cpu_to_be64 (_PAGE_PRESENT | _PAGE_PRIVILEGED)); +return __pte (pte_val (pte) & ~_PAGE_PRIVILEGED); +} +cheers
-aneesh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Michael Ellerman
reviewerRe: [PATCH V2 2/2] powerpc/mm/autonuma: Switch ppc64 to its own implementeation of saved write
Quoted 30 lines · Aneesh Kumar K.V
Quoted 9 lines · Michael Ellerman
Quoted 8 lines · Aneesh Kumar K.V
arch/powerpc/include/asm/book3s/64/mmu-hash.h +0
-0@@ -16,6 +16,9 @@#include <asm/page.h> #include <asm/bug.h> Quoted 6 lines
arch/powerpc/include/asm/book3s/64/mmu-hash.h line 21
Quoted 3 lines
arch/powerpc/include/asm/book3s/64/mmu-hash.h +3
-0+#ifndef __ASSEMBLY__ +#include <linux/mmdebug.h> +#endif I assume that's for the
VM_BUG_ON()you add below. But if so wouldn't the #include be better placed inbook3s/64/pgtable.halso?
mmu-hash.hhas got a hack that is explained below#ifndef
__ASSEMBLY__
#include <linux/mmdebug.h>
#endif
/*
* This is necessary to get the definition ofPGTABLE_RANGEwhich we
* need for various slices related matters. Note that this isn't the
* completepgtable.hbut only a portion of it.
*/
#include <asm/book3s/64/pgtable.h>
Quoted 3 lines
This is the only place where we do that
book3s/64/pgtable.hinclude this way. Everybody should includeasm/pgable.hwhich picks the righ version based on different config option.
I don't understand how that is related.
If you're adding a VM_BUG_ON() in book3s/64/pgtable.h, why isn't the include of mmdebug.h in that file also?
cheers