mm/huge_memory: don't mark refcounted pages special in vmf_insert_folio_*()
Subsystems (4)
cover mm/huge_memory: don't mark refcounted pages special in vmf_insert_folio_*()
Based on Linus' master.
While working on improving vm_normal_page() and friends, I stumbled over this issues: refcounted "normal" pages must not be marked using pmd_special() / pud_special().
Fortunately, so far there doesn't seem to be serious damage.
This is only compile-tested so far. Still looking for an easy way to test PMD/PUD mappings with DAX. Any tests I can easily run?
David Hildenbrand (2):
mm/huge_memory: don't mark refcounted pages special in
vmf_insert_folio_pmd()
mm/huge_memory: don't mark refcounted pages special in
vmf_insert_folio_pud()
include/linux/mm.h | 15 ++++++++++ mm/huge_memory.c | 72 +++++++++++++++++++++++++++++++++++----------- 2 files changed, 70 insertions(+), 17 deletions(-)
base-commit: a9dfb7db96f7bc1f30feae673aab7fdbfbc94e9c
Signature · 1 line
2.49.0
PATCH 1/2 mm/huge_memory: don't mark refcounted pages special in vmf_insert_folio_pmd()
Marking PMDs that map a "normal" refcounted folios as special is against our rules documented for vm_normal_page().
Fortunately, there are not that many pmd_special() check that can be mislead, and most vm_normal_page_pmd()/vm_normal_folio_pmd() users that would get this wrong right now are rather harmless: e.g., none so far bases decisions whether to grab a folio reference on that decision.
Well, and GUP-fast will fallback to GUP-slow. All in all, so far no big implications as it seems.
Getting this right will get more important as we use folio_normal_page_pmd() in more places.
Fix it by just inlining the relevant code, making the whole pmd_none() handling cleaner. We can now use folio_mk_pmd().
While at it, make sure that a pmd that is not-none is actually present before comparing PFNs.
--- mm/huge_memory.c | 39 ++++++++++++++++++++++++++++++++------- 1 file changed, 32 insertions(+), 7 deletions(-)
mm/huge_memory.c +32 -7
@@ -1474,9 +1474,10 @@ vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vmf, struct folio *folio,struct vm_area_struct *vma = vmf->vma;unsigned long addr = vmf->address &PMD_MASK ;struct mm_struct *mm = vma->vm_mm; +pmd_t *pmd = vmf->pmd;spinlock_t *ptl;pgtable_t pgtable =NULL ; -int error; +pmd_t entry;if (addr < vma->vm_start || addr >= vma->vm_end)return VM_FAULT_SIGBUS ;
@@ -1490,17 +1491,41 @@ vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vmf, struct folio *folio,return VM_FAULT_OOM ; } - ptl =pmd_lock (mm, vmf->pmd); -if (pmd_none (*vmf->pmd)) { + ptl =pmd_lock (mm, pmd); +if (pmd_none (*pmd)) {folio_get (folio);folio_add_file_rmap_pmd (folio, &folio->page, vma);add_mm_counter (mm,mm_counter_file (folio),HPAGE_PMD_NR ); + + entry =folio_mk_pmd (folio, vma->vm_page_prot); +if (write) { + entry =pmd_mkyoung (pmd_mkdirty (entry)); + entry =maybe_pmd_mkwrite (entry, vma); + } +set_pmd_at (mm, addr, pmd, entry); +update_mmu_cache_pmd (vma, addr, pmd); + +if (pgtable) { +pgtable_trans_huge_deposit (mm, pmd, pgtable); +mm_inc_nr_ptes (mm); + pgtable =NULL ; + } + }else if (pmd_present (*pmd) && write) { +/* +* We only allow for upgrading write permissions if the +* same folio is already mapped. +*/ +if (pmd_pfn (*pmd) ==folio_pfn (folio)) { + entry =pmd_mkyoung (*pmd); + entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); +if (pmdp_set_access_flags (vma, addr, pmd, entry,1 )) +update_mmu_cache_pmd (vma, addr, pmd); + }else { +WARN_ON_ONCE (!is_huge_zero_pmd (*pmd)); + } } - error =insert_pfn_pmd (vma, addr, vmf->pmd, -pfn_to_pfn_t (folio_pfn (folio)), vma->vm_page_prot, - write, pgtable);spin_unlock (ptl); -if (error && pgtable) +if (pgtable)pte_free (mm, pgtable);return VM_FAULT_NOPAGE ;
Signature · 1 line
2.49.0
Re: [PATCH v1 1/2] mm/huge_memory: don't mark refcounted pages special in vmf_insert_folio_pmd()
Quoted 14 lines · David Hildenbrand
Marking PMDs that map a "normal" refcounted folios as special is against our rules documented for
vm_normal_page().Fortunately, there are not that many
pmd_special()check that can be mislead, and most vm_normal_page_pmd()/vm_normal_folio_pmd() users that would get this wrong right now are rather harmless: e.g., none so far bases decisions whether to grab a folio reference on that decision.Well, and GUP-fast will fallback to GUP-slow. All in all, so far no big implications as it seems.
Getting this right will get more important as we use
folio_normal_page_pmd()in more places.
Quoted 8 lines
Fix it by just inlining the relevant code, making the whole
pmd_none()handling cleaner. We can now usefolio_mk_pmd().While at it, make sure that a pmd that is not-none is actually present before comparing PFNs.
Fixes 6c88f72691f8 ("mm/huge_memory: add vmf_insert_folio_pmd()")Signed-off-by David Hildenbrand <[email protected]>
Hi David,
Quoted 48 lines
--- mm/huge_memory.c | 39 ++++++++++++++++++++++++++++++++------- 1 file changed, 32 insertions(+), 7 deletions(-)mm/huge_memory.c +21
-3@@ -1474,9 +1474,10 @@ vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vmf, struct folio *folio,struct vm_area_struct *vma = vmf->vma;unsigned long addr = vmf->address &PMD_MASK ;struct mm_struct *mm = vma->vm_mm; +pmd_t *pmd = vmf->pmd;spinlock_t *ptl;pgtable_t pgtable =NULL ; -int error; +pmd_t entry;if (addr < vma->vm_start || addr >= vma->vm_end)return VM_FAULT_SIGBUS ;@@ -1490,17 +1491,41 @@ vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vmf, struct folio *folio,return VM_FAULT_OOM ; } - ptl =pmd_lock (mm, vmf->pmd); -if (pmd_none (*vmf->pmd)) { + ptl =pmd_lock (mm, pmd); +if (pmd_none (*pmd)) {folio_get (folio);folio_add_file_rmap_pmd (folio, &folio->page, vma);add_mm_counter (mm,mm_counter_file (folio),HPAGE_PMD_NR ); + + entry =folio_mk_pmd (folio, vma->vm_page_prot); +if (write) { + entry =pmd_mkyoung (pmd_mkdirty (entry)); + entry =maybe_pmd_mkwrite (entry, vma); + } +set_pmd_at (mm, addr, pmd, entry); +update_mmu_cache_pmd (vma, addr, pmd); + +if (pgtable) { +pgtable_trans_huge_deposit (mm, pmd, pgtable); +mm_inc_nr_ptes (mm); + pgtable =NULL ; + } + }else if (pmd_present (*pmd) && write) { +/* +* We only allow for upgrading write permissions if the
mm/huge_memory.c vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vmf, struct folio *folio, · line 1525
Quoted 10 lines
mm/huge_memory.c +10
-0+* same folio is already mapped. +*/ +if (pmd_pfn (*pmd) ==folio_pfn (folio)) { + entry =pmd_mkyoung (*pmd); + entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); +if (pmdp_set_access_flags (vma, addr, pmd, entry,1 )) +update_mmu_cache_pmd (vma, addr, pmd); + }else { +WARN_ON_ONCE (!is_huge_zero_pmd (*pmd)); + }
So, this is pretty much insert_pfn_pmd without pmd_mkdevmap/pmd_mkspecial(). I guess vmf_inser_folio_pmd() doesn't have to be concerned with devmaps either, right?
Looks good to me, just a nit: would it not be better to pass a boolean to insert_pfn_pmd() that lets it know whether it "can" create a devmap/special entries?
Signature · 2 lines
Oscar Salvador SUSE Labs
Re: [PATCH v1 1/2] mm/huge_memory: don't mark refcounted pages special in vmf_insert_folio_pmd()
Quoted 89 lines · Oscar Salvador
Quoted 14 lines · David Hildenbrand
Marking PMDs that map a "normal" refcounted folios as special is against our rules documented for
vm_normal_page().Fortunately, there are not that many
pmd_special()check that can be mislead, and most vm_normal_page_pmd()/vm_normal_folio_pmd() users that would get this wrong right now are rather harmless: e.g., none so far bases decisions whether to grab a folio reference on that decision.Well, and GUP-fast will fallback to GUP-slow. All in all, so far no big implications as it seems.
Getting this right will get more important as we use
folio_normal_page_pmd()in more places.Quoted 8 lines
Fix it by just inlining the relevant code, making the whole
pmd_none()handling cleaner. We can now usefolio_mk_pmd().While at it, make sure that a pmd that is not-none is actually present before comparing PFNs.
Fixes 6c88f72691f8 ("mm/huge_memory: add vmf_insert_folio_pmd()")Signed-off-by David Hildenbrand <[email protected]>Hi David,
Quoted 48 lines
--- mm/huge_memory.c | 39 ++++++++++++++++++++++++++++++++------- 1 file changed, 32 insertions(+), 7 deletions(-)mm/huge_memory.c +21
-3@@ -1474,9 +1474,10 @@ vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vmf, struct folio *folio,struct vm_area_struct *vma = vmf->vma;unsigned long addr = vmf->address &PMD_MASK ;struct mm_struct *mm = vma->vm_mm; +pmd_t *pmd = vmf->pmd;spinlock_t *ptl;pgtable_t pgtable =NULL ; -int error; +pmd_t entry;if (addr < vma->vm_start || addr >= vma->vm_end)return VM_FAULT_SIGBUS ;@@ -1490,17 +1491,41 @@ vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vmf, struct folio *folio,return VM_FAULT_OOM ; } - ptl =pmd_lock (mm, vmf->pmd); -if (pmd_none (*vmf->pmd)) { + ptl =pmd_lock (mm, pmd); +if (pmd_none (*pmd)) {folio_get (folio);folio_add_file_rmap_pmd (folio, &folio->page, vma);add_mm_counter (mm,mm_counter_file (folio),HPAGE_PMD_NR ); + + entry =folio_mk_pmd (folio, vma->vm_page_prot); +if (write) { + entry =pmd_mkyoung (pmd_mkdirty (entry)); + entry =maybe_pmd_mkwrite (entry, vma); + } +set_pmd_at (mm, addr, pmd, entry); +update_mmu_cache_pmd (vma, addr, pmd); + +if (pgtable) { +pgtable_trans_huge_deposit (mm, pmd, pgtable); +mm_inc_nr_ptes (mm); + pgtable =NULL ; + } + }else if (pmd_present (*pmd) && write) { +/* +* We only allow for upgrading write permissions if the mm/huge_memory.c vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vmf, struct folio *folio, · line 1525
Quoted 10 lines
mm/huge_memory.c +10
-0+* same folio is already mapped. +*/ +if (pmd_pfn (*pmd) ==folio_pfn (folio)) { + entry =pmd_mkyoung (*pmd); + entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); +if (pmdp_set_access_flags (vma, addr, pmd, entry,1 )) +update_mmu_cache_pmd (vma, addr, pmd); + }else { +WARN_ON_ONCE (!is_huge_zero_pmd (*pmd)); + }So, this is pretty much
insert_pfn_pmdwithout pmd_mkdevmap/pmd_mkspecial(). I guessvmf_inser_folio_pmd()doesn't have to be concerned with devmaps either, right?
Quoted 3 lines
Looks good to me, just a nit: would it not be better to pass a boolean to
insert_pfn_pmd()that lets it know whether it "can" create a devmap/special entries?
See my reply to Dan.
Yet another boolean, yuck. Passing the folio and the pfn, yuck.
(I have a strong opinion here ;) )
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v1 1/2] mm/huge_memory: don't mark refcounted pages special in vmf_insert_folio_pmd()
Quoted 5 lines · David Hildenbrand
See my reply to Dan.
Yet another boolean, yuck. Passing the folio and the pfn, yuck.
(I have a strong opinion here ;) )
ok, I see it was already considered. No more questions then ;-)
Signature · 2 lines
Oscar Salvador SUSE Labs
Re: [PATCH v1 1/2] mm/huge_memory: don't mark refcounted pages special in vmf_insert_folio_pmd()
Quoted 8 lines · Oscar Salvador
Quoted 5 lines · David Hildenbrand
See my reply to Dan.
Yet another boolean, yuck. Passing the folio and the pfn, yuck.
(I have a strong opinion here ;) )
ok, I see it was already considered. No more questions then ;-)
Yeah, the first version I had did exactly that (second boolean), and I concluded that we are simply not "inserting PFNs".
Especially the second patch is much clearer on the pud_none() handling (similar to insert_page_into_pte_locked() where we only have exactly one such check)
I'll have some follow-up patches planned that will add more pfn-specific and folio-specific sanity checks.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v1 1/2] mm/huge_memory: don't mark refcounted pages special in vmf_insert_folio_pmd()
In reply to Oscar SalvadorQuoted 8 lines · Oscar Salvador
Quoted 5 lines · David Hildenbrand
See my reply to Dan.
Yet another boolean, yuck. Passing the folio and the pfn, yuck.
(I have a strong opinion here ;) )
ok, I see it was already considered. No more questions then ;-)
Okay, change of plans, let me try to unify both functions in a clean way :)
I'll only have to find a clean way to make the other stuff I have planned with these functions work ...
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v1 1/2] mm/huge_memory: don't mark refcounted pages special in vmf_insert_folio_pmd()
In reply to David HildenbrandQuoted 14 lines · David Hildenbrand
Marking PMDs that map a "normal" refcounted folios as special is against our rules documented for
vm_normal_page().Fortunately, there are not that many
pmd_special()check that can be mislead, and most vm_normal_page_pmd()/vm_normal_folio_pmd() users that would get this wrong right now are rather harmless: e.g., none so far bases decisions whether to grab a folio reference on that decision.Well, and GUP-fast will fallback to GUP-slow. All in all, so far no big implications as it seems.
Getting this right will get more important as we use
folio_normal_page_pmd()in more places.
Quoted 8 lines
Fix it by just inlining the relevant code, making the whole
pmd_none()handling cleaner. We can now usefolio_mk_pmd().While at it, make sure that a pmd that is not-none is actually present before comparing PFNs.
Fixes 6c88f72691f8 ("mm/huge_memory: add vmf_insert_folio_pmd()")Signed-off-by David Hildenbrand <[email protected]>
Signature · 2 lines
Oscar Salvador SUSE Labs
PATCH 2/2 mm/huge_memory: don't mark refcounted pages special in vmf_insert_folio_pud()
In reply to David HildenbrandMarking PUDs that map a "normal" refcounted folios as special is against our rules documented for vm_normal_page().
Fortunately, there are not that many pud_special() check that can be mislead and are right now rather harmless: e.g., none so far bases decisions whether to grab a folio reference on that decision.
Well, and GUP-fast will fallback to GUP-slow. All in all, so far no big implications as it seems.
Getting this right will get more important as we introduce folio_normal_page_pud() and start using it in more place where we currently special-case based on other VMA flags.
Fix it by just inlining the relevant code, making the whole pud_none() handling cleaner.
Add folio_mk_pud() to mimic what we do with folio_mk_pmd().
While at it, make sure that the pud that is non-none is actually present before comparing PFNs.
--- include/linux/mm.h | 15 +++++++++++++++ mm/huge_memory.c | 33 +++++++++++++++++++++++---------- 2 files changed, 38 insertions(+), 10 deletions(-)
include/linux/mm.h +15 -0
@@ -1816,6 +1816,21 @@ static inline pmd_t folio_mk_pmd(struct folio *folio, pgprot_t pgprot) {return pmd_mkhuge (pfn_pmd (folio_pfn (folio), pgprot)); } + +/** +* folio_mk_pud - Create a PUD for this folio +* @folio: The folio to create a PUD for +* @pgprot: The page protection bits to use +* +* Create a page table entry for the first page of this folio. +* This is suitable for passing to set_pud_at(). +* +* Return: A page table entry suitable for mapping this folio. +*/ +static inline pud_t folio_mk_pud (struct folio *folio,pgprot_t pgprot) +{ +return pud_mkhuge (pfn_pud (folio_pfn (folio), pgprot)); +}#endif #endif /* CONFIG_MMU */
mm/huge_memory.c +23 -10
@@ -1629,6 +1629,7 @@ vm_fault_t vmf_insert_folio_pud(struct vm_fault *vmf, struct folio *folio,pud_t *pud = vmf->pud;struct mm_struct *mm = vma->vm_mm;spinlock_t *ptl; +pud_t entry;if (addr < vma->vm_start || addr >= vma->vm_end)return VM_FAULT_SIGBUS ;
@@ -1637,20 +1638,32 @@ vm_fault_t vmf_insert_folio_pud(struct vm_fault *vmf, struct folio *folio,return VM_FAULT_SIGBUS ; ptl =pud_lock (mm, pud); - -/* -* If there is already an entry present we assume the folio is -* already mapped, hence no need to take another reference. We -* still call insert_pfn_pud() though in case the mapping needs -* upgrading to writeable. -*/ -if (pud_none (*vmf->pud)) { +if (pud_none (*pud)) {folio_get (folio);folio_add_file_rmap_pud (folio, &folio->page, vma);add_mm_counter (mm,mm_counter_file (folio),HPAGE_PUD_NR ); + + entry =folio_mk_pud (folio, vma->vm_page_prot); +if (write) { + entry =pud_mkyoung (pud_mkdirty (entry)); + entry =maybe_pud_mkwrite (entry, vma); + } +set_pud_at (mm, addr, pud, entry); +update_mmu_cache_pud (vma, addr, pud); + }else if (pud_present (*pud) && write) { +/* +* We only allow for upgrading write permissions if the +* same folio is already mapped. +*/ +if (pud_pfn (*pud) ==folio_pfn (folio)) { + entry =pud_mkyoung (*pud); + entry =maybe_pud_mkwrite (pud_mkdirty (entry), vma); +if (pudp_set_access_flags (vma, addr, pud, entry,1 )) +update_mmu_cache_pud (vma, addr, pud); + }else { +WARN_ON_ONCE (1 ); + } } -insert_pfn_pud (vma, addr, vmf->pud,pfn_to_pfn_t (folio_pfn (folio)), - write);spin_unlock (ptl);return VM_FAULT_NOPAGE ;
Signature · 1 line
2.49.0
Re: [PATCH v1 2/2] mm/huge_memory: don't mark refcounted pages special in vmf_insert_folio_pud()
Quoted 44 lines · David Hildenbrand
Marking PUDs that map a "normal" refcounted folios as special is against our rules documented for
vm_normal_page().Fortunately, there are not that many
pud_special()check that can be mislead and are right now rather harmless: e.g., none so far bases decisions whether to grab a folio reference on that decision.Well, and GUP-fast will fallback to GUP-slow. All in all, so far no big implications as it seems.
Getting this right will get more important as we introduce
folio_normal_page_pud()and start using it in more place where we currently special-case based on other VMA flags.Fix it by just inlining the relevant code, making the whole
pud_none()handling cleaner.Add
folio_mk_pud()to mimic what we do withfolio_mk_pmd().While at it, make sure that the pud that is non-none is actually present before comparing PFNs.
Fixes dbe54153296d ("mm/huge_memory: add vmf_insert_folio_pud()")Signed-off-by David Hildenbrand <[email protected]>--- include/linux/mm.h | 15 +++++++++++++++ mm/huge_memory.c | 33 +++++++++++++++++++++++---------- 2 files changed, 38 insertions(+), 10 deletions(-)include/linux/mm.h +7
-0@@ -1816,6 +1816,21 @@ static inline pmd_t folio_mk_pmd(struct folio *folio, pgprot_t pgprot) {return pmd_mkhuge (pfn_pmd (folio_pfn (folio), pgprot)); } + +/** +* folio_mk_pud - Create a PUD for this folio +* @folio: The folio to create a PUD for +* @pgprot: The page protection bits to use +* +* Create a page table entry for the first page of this folio.
include/linux/mm.h static inline pmd_t folio_mk_pmd(struct folio *folio, pgprot_t pgprot) · line 1835
Quoted 10 lines
include/linux/mm.h +8
-0+* This is suitable for passing to set_pud_at(). +* +* Return: A page table entry suitable for mapping this folio. +*/ +static inline pud_t folio_mk_pud (struct folio *folio,pgprot_t pgprot) +{ +return pud_mkhuge (pfn_pud (folio_pfn (folio), pgprot)); +}#endif #endif /* CONFIG_MMU */
The following on top should make cross-compiles happy (git diff output because it's late here, probably whitespace messed up):
include/linux/mm.h +3 -1
@@ -1817,6 +1817,7 @@ static inline pmd_t folio_mk_pmd(struct folio *folio, pgprot_t pgprot)return pmd_mkhuge (pfn_pmd (folio_pfn (folio), pgprot)); } +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD /** * folio_mk_pud - Create a PUD for this folio * @folio: The folio to create a PUD for
@@ -1831,7 +1832,8 @@ static inline pud_t folio_mk_pud(struct folio *folio, pgprot_t pgprot) {return pud_mkhuge (pfn_pud (folio_pfn (folio), pgprot)); } -#endif +#endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* CONFIG_MMU */ static inline bool folio_has_pincount(const struct folio *folio
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v1 2/2] mm/huge_memory: don't mark refcounted pages special in vmf_insert_folio_pud()
In reply to David HildenbrandQuoted 17 lines · David Hildenbrand
Marking PUDs that map a "normal" refcounted folios as special is against our rules documented for
vm_normal_page().Fortunately, there are not that many
pud_special()check that can be mislead and are right now rather harmless: e.g., none so far bases decisions whether to grab a folio reference on that decision.Well, and GUP-fast will fallback to GUP-slow. All in all, so far no big implications as it seems.
Getting this right will get more important as we introduce
folio_normal_page_pud()and start using it in more place where we currently special-case based on other VMA flags.Fix it by just inlining the relevant code, making the whole
pud_none()handling cleaner.
Quoted 7 lines
Add
folio_mk_pud()to mimic what we do withfolio_mk_pmd().While at it, make sure that the pud that is non-none is actually present before comparing PFNs.
Fixes dbe54153296d ("mm/huge_memory: add vmf_insert_folio_pud()")Signed-off-by David Hildenbrand <[email protected]>
Signature · 2 lines
Oscar Salvador SUSE Labs
Quoted 10 lines · David Hildenbrand
Based on Linus' master.
While working on improving
vm_normal_page()and friends, I stumbled over this issues: refcounted "normal" pages must not be marked usingpmd_special()/pud_special().Fortunately, so far there doesn't seem to be serious damage.
This is only compile-tested so far. Still looking for an easy way to test PMD/PUD mappings with DAX. Any tests I can easily run?
... aaaand I should have waited for the cross compiles.
s390x and loongarch are not happy about pud_mkhuge() and pfn_pud().
Need to fence folio_mk_pud() by CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 10 lines · David Hildenbrand
Based on Linus' master.
While working on improving
vm_normal_page()and friends, I stumbled over this issues: refcounted "normal" pages must not be marked usingpmd_special()/pud_special().Fortunately, so far there doesn't seem to be serious damage.
This is only compile-tested so far. Still looking for an easy way to test PMD/PUD mappings with DAX. Any tests I can easily run?
The way I test this I would not classify as "easy", it is a bit of a pain to setup, but it is passing here:
[root@host ndctl]# meson test -C build --suite ndctl:dax
ninja: Entering directory `/root/git/ndctl/build' [168/168] Linking target cxl/cxl
1/13 ndctl:dax / daxdev-errors.sh OK 14.30s 2/13 ndctl:dax / multi-dax.sh OK 2.89s 3/13 ndctl:dax / sub-section.sh OK 8.40s 4/13 ndctl:dax / dax-dev OK 0.06s 5/13 ndctl:dax / dax-ext4.sh OK 20.53s 6/13 ndctl:dax / dax-xfs.sh OK 20.34s 7/13 ndctl:dax / device-dax OK 11.67s 8/13 ndctl:dax / revoke-devmem OK 0.25s 9/13 ndctl:dax / device-dax-fio.sh OK 34.02s 10/13 ndctl:dax / daxctl-devices.sh OK 3.44s 11/13 ndctl:dax / daxctl-create.sh SKIP 0.32s exit status 77 12/13 ndctl:dax / dm.sh OK 1.33s 13/13 ndctl:dax / mmap.sh OK 85.12s
...ignore the SKIP, that seems to be caused by an acpi-einj regression.
However, how about not duplicating the internals of insert_pfn_p[mu]d() with something like the below. Either way you can add:
base-commit: a9dfb7db96f7bc1f30feae673aab7fdbfbc94e9c
-- 8< --
mm/huge_memory.c +15 -20
@@ -1372,9 +1372,9 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)return __do_huge_pmd_anonymous_page (vmf); } -static int insert_pfn_pmd (struct vm_area_struct *vma,unsigned long addr, -pmd_t *pmd,pfn_t pfn,pgprot_t prot,bool write, -pgtable_t pgtable) +static int insert_pmd (struct vm_area_struct *vma,unsigned long addr, +pmd_t *pmd,pfn_t pfn,struct folio *folio, +pgprot_t prot,bool write,pgtable_t pgtable) {struct mm_struct *mm = vma->vm_mm;pmd_t entry;
@@ -1397,9 +1397,7 @@ static int insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, } entry =pmd_mkhuge (pfn_t_pmd (pfn, prot)); -if (pfn_t_devmap (pfn)) - entry =pmd_mkdevmap (entry); -else +if (!folio) entry =pmd_mkspecial (entry);if (write) { entry =pmd_mkyoung (pmd_mkdirty (entry));
@@ -1458,8 +1456,8 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write)pfnmap_setup_cachemode_pfn (pfn_t_to_pfn (pfn), &pgprot); ptl =pmd_lock (vma->vm_mm, vmf->pmd); - error =insert_pfn_pmd (vma, addr, vmf->pmd, pfn, pgprot, write, - pgtable); + error =insert_pmd (vma, addr, vmf->pmd, pfn,NULL , pgprot, write, + pgtable);spin_unlock (ptl);if (error && pgtable)pte_free (vma->vm_mm, pgtable);
@@ -1496,9 +1494,8 @@ vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vmf, struct folio *folio,folio_add_file_rmap_pmd (folio, &folio->page, vma);add_mm_counter (mm,mm_counter_file (folio),HPAGE_PMD_NR ); } - error =insert_pfn_pmd (vma, addr, vmf->pmd, -pfn_to_pfn_t (folio_pfn (folio)), vma->vm_page_prot, - write, pgtable); + error =insert_pmd (vma, addr, vmf->pmd,pfn_to_pfn_t (folio_pfn (folio)), + folio, vma->vm_page_prot, write, pgtable);spin_unlock (ptl);if (error && pgtable)pte_free (mm, pgtable);
@@ -1515,8 +1512,8 @@ static pud_t maybe_pud_mkwrite(pud_t pud, struct vm_area_struct *vma)return pud; } -static void insert_pfn_pud (struct vm_area_struct *vma,unsigned long addr, -pud_t *pud,pfn_t pfn,bool write) +static void insert_pud (struct vm_area_struct *vma,unsigned long addr, +pud_t *pud,pfn_t pfn,struct folio *folio,bool write) {struct mm_struct *mm = vma->vm_mm;pgprot_t prot = vma->vm_page_prot;
@@ -1535,9 +1532,7 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, } entry =pud_mkhuge (pfn_t_pud (pfn, prot)); -if (pfn_t_devmap (pfn)) - entry =pud_mkdevmap (entry); -else +if (!folio) entry =pud_mkspecial (entry);if (write) { entry =pud_mkyoung (pud_mkdirty (entry));
@@ -1581,7 +1576,7 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write)pfnmap_setup_cachemode_pfn (pfn_t_to_pfn (pfn), &pgprot); ptl =pud_lock (vma->vm_mm, vmf->pud); -insert_pfn_pud (vma, addr, vmf->pud, pfn, write); +insert_pud (vma, addr, vmf->pud, pfn,NULL , write);spin_unlock (ptl);return VM_FAULT_NOPAGE ;
@@ -1616,7 +1611,7 @@ vm_fault_t vmf_insert_folio_pud(struct vm_fault *vmf, struct folio *folio,/* * If there is already an entry present we assume the folio is * already mapped, hence no need to take another reference. We -* still call insert_pfn_pud() though in case the mapping needs +* still call insert_pud() though in case the mapping needs * upgrading to writeable. */ if (pud_none (*vmf->pud)) {
@@ -1624,8 +1619,8 @@ vm_fault_t vmf_insert_folio_pud(struct vm_fault *vmf, struct folio *folio,folio_add_file_rmap_pud (folio, &folio->page, vma);add_mm_counter (mm,mm_counter_file (folio),HPAGE_PUD_NR ); } -insert_pfn_pud (vma, addr, vmf->pud,pfn_to_pfn_t (folio_pfn (folio)), - write); +insert_pud (vma, addr, vmf->pud,pfn_to_pfn_t (folio_pfn (folio)), folio, + write);spin_unlock (ptl);return VM_FAULT_NOPAGE ;
Hi Dan,
Quoted 7 lines · Dan Williams
Quoted 6 lines · David Hildenbrand
Based on Linus' master.
While working on improving
vm_normal_page()and friends, I stumbled over this issues: refcounted "normal" pages must not be marked usingpmd_special()/pud_special().
Quoted 7 lines
Quoted 4 lines
Fortunately, so far there doesn't seem to be serious damage.
This is only compile-tested so far. Still looking for an easy way to test PMD/PUD mappings with DAX. Any tests I can easily run?
The way I test this I would not classify as "easy", it is a bit of a pain to setup, but it is passing here:
I guess most of the instructions are in
?
I would assume that we need to set aside some special dax area using early boot params (memmap=).
Might come in handy in the future.
Quoted 18 lines
[root@host ndctl]# meson test -C build --suite ndctl:dax
ninja: Entering directory `/root/git/ndctl/build' [168/168] Linking target cxl/cxl1/13 ndctl:dax / daxdev-errors.sh OK 14.30s 2/13 ndctl:dax / multi-dax.sh OK 2.89s 3/13 ndctl:dax / sub-section.sh OK 8.40s 4/13 ndctl:dax / dax-dev OK 0.06s 5/13 ndctl:dax / dax-ext4.sh OK 20.53s 6/13 ndctl:dax / dax-xfs.sh OK 20.34s 7/13 ndctl:dax / device-dax OK 11.67s 8/13 ndctl:dax / revoke-devmem OK 0.25s 9/13 ndctl:dax / device-dax-fio.sh OK 34.02s 10/13 ndctl:dax / daxctl-devices.sh OK 3.44s 11/13 ndctl:dax / daxctl-create.sh SKIP 0.32s exit status 77 12/13 ndctl:dax / dm.sh OK 1.33s 13/13 ndctl:dax / mmap.sh OK 85.12s
Quoted 1 line
...ignore the SKIP, that seems to be caused by an acpi-einj regression.
Thanks for running these tests!
Quoted 3 lines
However, how about not duplicating the internals of insert_pfn_p[mu]d() with something like the below. Either way you can add:
I considered that, but I prefer the current end result where we cleanup the pmd_none() handling and not mess with folios and pfns at the same time.
... just like we do for insert_pfn() vs. insert_page(), I don't think these code paths should be unified.
(we should do more sanity checks like validate_page_before_insert() etc., but that's something for another day :) )
Quoted 3 lines
Tested-by Dan Williams <[email protected]>Reviewed-by Dan Williams <[email protected]>
Thanks!
Let me resend with the fixup squashed.
Signature · 3 lines
Cheers, David / dhildenb