mm/huge_memory: vmf_insert_folio_*() and vmf_insert_pfn_pud() fixes
Subsystems (5)
v3 cover mm/huge_memory: vmf_insert_folio_*() and vmf_insert_pfn_pud() fixes
Based on mm/mm-unstable.
While working on improving vm_normal_page() and friends, I stumbled over this issues: refcounted "normal" folios must not be marked using pmd_special() / pud_special(). Otherwise, we're effectively telling the system that these folios are no "normal", violating the rules we documented for vm_normal_page().
Fortunately, there are not many pmd_special()/pud_special() users yet. So far there doesn't seem to be serious damage.
Tested using the ndctl tests ("ndctl:dax" suite).
v2 -> v3:
* Added tags (thanks for all the review!)
* Smaller fixups (add empty lines) and patch description improvements
v1 -> v2:
* "mm/huge_memory: don't ignore queried cachemode in vmf_insert_pfn_pud()" -> Added after stumbling over that
* Modified the other tests to reuse the existing function by passing a new struct
* Renamed the patches to talk about "folios" instead of pages and adjusted the patch descriptions
* Dropped RB/TB from Dan and Oscar due to the changes
David Hildenbrand (3):
mm/huge_memory: don't ignore queried cachemode in vmf_insert_pfn_pud() mm/huge_memory: don't mark refcounted folios special in
vmf_insert_folio_pmd()
mm/huge_memory: don't mark refcounted folios special in
vmf_insert_folio_pud()
include/linux/mm.h | 19 +++++++- mm/huge_memory.c | 112 ++++++++++++++++++++++++++++----------------- 2 files changed, 87 insertions(+), 44 deletions(-)
Signature · 1 line
2.49.0
PATCH v3 1/3 mm/huge_memory: don't ignore queried cachemode in vmf_insert_pfn_pud()
We setup the cache mode but ... don't forward the updated pgprot to insert_pfn_pud().
Only a problem on x86-64 PAT when mapping PFNs using PUDs that require a special cachemode.
Fix it by using the proper pgprot where the cachemode was setup.
It is unclear in which configurations we would get the cachemode wrong:
through vfio seems possible. Getting cachemodes wrong is usually ... bad. As the fix is easy, let's backport it to stable.
Identified by code inspection.
--- mm/huge_memory.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
mm/huge_memory.c +3 -4
@@ -1516,10 +1516,9 @@ static pud_t maybe_pud_mkwrite(pud_t pud, struct vm_area_struct *vma) }static void insert_pfn_pud (struct vm_area_struct *vma,unsigned long addr, -pud_t *pud,pfn_t pfn,bool write) +pud_t *pud,pfn_t pfn,pgprot_t prot,bool write) {struct mm_struct *mm = vma->vm_mm; -pgprot_t prot = vma->vm_page_prot;pud_t entry;if (!pud_none (*pud)) {
@@ -1581,7 +1580,7 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write)pfnmap_setup_cachemode_pfn (pfn_t_to_pfn (pfn), &pgprot); ptl =pud_lock (vma->vm_mm, vmf->pud); -insert_pfn_pud (vma, addr, vmf->pud, pfn, write); +insert_pfn_pud (vma, addr, vmf->pud, pfn, pgprot, write);spin_unlock (ptl);return VM_FAULT_NOPAGE ;
@@ -1625,7 +1624,7 @@ vm_fault_t vmf_insert_folio_pud(struct vm_fault *vmf, struct folio *folio,add_mm_counter (mm,mm_counter_file (folio),HPAGE_PUD_NR ); }insert_pfn_pud (vma, addr, vmf->pud,pfn_to_pfn_t (folio_pfn (folio)), - write); + vma->vm_page_prot, write);spin_unlock (ptl);return VM_FAULT_NOPAGE ;
Signature · 1 line
2.49.0
Re: [PATCH v3 1/3] mm/huge_memory: don't ignore queried cachemode in vmf_insert_pfn_pud()
Quoted 14 lines · David Hildenbrand
We setup the cache mode but ... don't forward the updated pgprot to
insert_pfn_pud().Only a problem on x86-64 PAT when mapping PFNs using PUDs that require a special cachemode.
Fix it by using the proper pgprot where the cachemode was setup.
It is unclear in which configurations we would get the cachemode wrong:
through vfio seems possible. Getting cachemodes wrong is usually ... bad. As the fix is easy, let's backport it to stable.Identified by code inspection.
Quoted 7 lines
Fixes 7b806d229ef1 ("mm: remove vmf_insert_pfn_xxx_prot() for huge page-table entries")Reviewed-by Dan Williams <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Reviewed-by Jason Gunthorpe <[email protected]>Tested-by Dan Williams <[email protected]>Cc <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Signature · 2 lines
Oscar Salvador SUSE Labs
PATCH v3 2/3 mm/huge_memory: don't mark refcounted folios special in vmf_insert_folio_pmd()
In reply to David HildenbrandMarking PMDs that map a "normal" refcounted folios as special is against our rules documented for vm_normal_page(): normal (refcounted) folios shall never have the page table mapping marked as special.
Fortunately, there are not that many pmd_special() check that can be mislead, and most vm_normal_page_pmd()/vm_normal_folio_pmd() users that would get this wrong right now are rather harmless: e.g., none so far bases decisions whether to grab a folio reference on that decision.
Well, and GUP-fast will fallback to GUP-slow. All in all, so far no big implications as it seems.
Getting this right will get more important as we use folio_normal_page_pmd() in more places.
Fix it by teaching insert_pfn_pmd() to properly handle folios and pfns -- moving refcount/mapcount/etc handling in there, renaming it to insert_pmd(), and distinguishing between both cases using a new simple "struct folio_or_pfn" structure.
Use folio_mk_pmd() to create a pmd for a folio cleanly.
--- mm/huge_memory.c | 59 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 40 insertions(+), 19 deletions(-)
mm/huge_memory.c +40 -19
@@ -1372,9 +1372,17 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)return __do_huge_pmd_anonymous_page (vmf); } -static int insert_pfn_pmd (struct vm_area_struct *vma,unsigned long addr, -pmd_t *pmd,pfn_t pfn,pgprot_t prot,bool write, -pgtable_t pgtable) +struct folio_or_pfn { +union { +struct folio *folio; +pfn_t pfn; + }; +bool is_folio; +}; + +static int insert_pmd (struct vm_area_struct *vma,unsigned long addr, +pmd_t *pmd,struct folio_or_pfn fop,pgprot_t prot, +bool write,pgtable_t pgtable) {struct mm_struct *mm = vma->vm_mm;pmd_t entry;
@@ -1382,8 +1390,11 @@ static int insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,lockdep_assert_held (pmd_lockptr (mm, pmd));if (!pmd_none (*pmd)) { +const unsigned long pfn = fop.is_folio ?folio_pfn (fop.folio) : +pfn_t_to_pfn (fop.pfn); +if (write) { -if (pmd_pfn (*pmd) !=pfn_t_to_pfn (pfn)) { +if (pmd_pfn (*pmd) != pfn) {WARN_ON_ONCE (!is_huge_zero_pmd (*pmd));return -EEXIST ; }
@@ -1396,11 +1407,20 @@ static int insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,return -EEXIST ; } - entry =pmd_mkhuge (pfn_t_pmd (pfn, prot)); -if (pfn_t_devmap (pfn)) - entry =pmd_mkdevmap (entry); -else - entry =pmd_mkspecial (entry); +if (fop.is_folio) { + entry =folio_mk_pmd (fop.folio, vma->vm_page_prot); + +folio_get (fop.folio); +folio_add_file_rmap_pmd (fop.folio, &fop.folio->page, vma); +add_mm_counter (mm,mm_counter_file (fop.folio),HPAGE_PMD_NR ); + }else { + entry =pmd_mkhuge (pfn_t_pmd (fop.pfn, prot)); + +if (pfn_t_devmap (fop.pfn)) + entry =pmd_mkdevmap (entry); +else + entry =pmd_mkspecial (entry); + }if (write) { entry =pmd_mkyoung (pmd_mkdirty (entry)); entry =maybe_pmd_mkwrite (entry, vma);
@@ -1431,6 +1451,9 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write)unsigned long addr = vmf->address &PMD_MASK ;struct vm_area_struct *vma = vmf->vma;pgprot_t pgprot = vma->vm_page_prot; +struct folio_or_pfn fop = { + .pfn = pfn, + };pgtable_t pgtable =NULL ;spinlock_t *ptl;int error;
@@ -1458,8 +1481,8 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write)pfnmap_setup_cachemode_pfn (pfn_t_to_pfn (pfn), &pgprot); ptl =pmd_lock (vma->vm_mm, vmf->pmd); - error =insert_pfn_pmd (vma, addr, vmf->pmd, pfn, pgprot, write, - pgtable); + error =insert_pmd (vma, addr, vmf->pmd, fop, pgprot, write, + pgtable);spin_unlock (ptl);if (error && pgtable)pte_free (vma->vm_mm, pgtable);
@@ -1474,6 +1497,10 @@ vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vmf, struct folio *folio,struct vm_area_struct *vma = vmf->vma;unsigned long addr = vmf->address &PMD_MASK ;struct mm_struct *mm = vma->vm_mm; +struct folio_or_pfn fop = { + .folio = folio, + .is_folio = true, + };spinlock_t *ptl;pgtable_t pgtable =NULL ;int error;
@@ -1491,14 +1518,8 @@ vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vmf, struct folio *folio, } ptl =pmd_lock (mm, vmf->pmd); -if (pmd_none (*vmf->pmd)) { -folio_get (folio); -folio_add_file_rmap_pmd (folio, &folio->page, vma); -add_mm_counter (mm,mm_counter_file (folio),HPAGE_PMD_NR ); - } - error =insert_pfn_pmd (vma, addr, vmf->pmd, -pfn_to_pfn_t (folio_pfn (folio)), vma->vm_page_prot, - write, pgtable); + error =insert_pmd (vma, addr, vmf->pmd, fop, vma->vm_page_prot, + write, pgtable);spin_unlock (ptl);if (error && pgtable)pte_free (mm, pgtable);
Signature · 1 line
2.49.0
Re: [PATCH v3 2/3] mm/huge_memory: don't mark refcounted folios special in vmf_insert_folio_pmd()
Quoted 20 lines · David Hildenbrand
Marking PMDs that map a "normal" refcounted folios as special is against our rules documented for
vm_normal_page(): normal (refcounted) folios shall never have the page table mapping marked as special.Fortunately, there are not that many
pmd_special()check that can be mislead, and most vm_normal_page_pmd()/vm_normal_folio_pmd() users that would get this wrong right now are rather harmless: e.g., none so far bases decisions whether to grab a folio reference on that decision.Well, and GUP-fast will fallback to GUP-slow. All in all, so far no big implications as it seems.
Getting this right will get more important as we use
folio_normal_page_pmd()in more places.Fix it by teaching
insert_pfn_pmd()to properly handle folios and pfns -- moving refcount/mapcount/etc handling in there, renaming it toinsert_pmd(), and distinguishing between both cases using a new simple "structfolio_or_pfn" structure.
Quoted 8 lines
Use
folio_mk_pmd()to create a pmd for a folio cleanly.Fixes 6c88f72691f8 ("mm/huge_memory: add vmf_insert_folio_pmd()")Reviewed-by Jason Gunthorpe <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Reviewed-by Dan Williams <[email protected]>Tested-by Dan Williams <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Altough we have it quite well explained here in the changelog, maybe having a little comment in insert_pmd() noting why pmds mapping normal folios cannot be marked special would be nice.
But just saying :-)
Signature · 2 lines
Oscar Salvador SUSE Labs
Oscar Salvador
Reviewed-byRe: [PATCH v3 2/3] mm/huge_memory: don't mark refcounted folios special in vmf_insert_folio_pmd()
Quoted 1 line · Oscar Salvador
Reviewed-by Oscar salvador <[email protected]>
Fat-fingers on a Friday afternoon:
Signature · 2 lines
Oscar Salvador SUSE Labs
Re: [PATCH v3 2/3] mm/huge_memory: don't mark refcounted folios special in vmf_insert_folio_pmd()
In reply to Oscar SalvadorQuoted 30 lines · Oscar Salvador
Quoted 20 lines · David Hildenbrand
Marking PMDs that map a "normal" refcounted folios as special is against our rules documented for
vm_normal_page(): normal (refcounted) folios shall never have the page table mapping marked as special.Fortunately, there are not that many
pmd_special()check that can be mislead, and most vm_normal_page_pmd()/vm_normal_folio_pmd() users that would get this wrong right now are rather harmless: e.g., none so far bases decisions whether to grab a folio reference on that decision.Well, and GUP-fast will fallback to GUP-slow. All in all, so far no big implications as it seems.
Getting this right will get more important as we use
folio_normal_page_pmd()in more places.Fix it by teaching
insert_pfn_pmd()to properly handle folios and pfns -- moving refcount/mapcount/etc handling in there, renaming it toinsert_pmd(), and distinguishing between both cases using a new simple "structfolio_or_pfn" structure.Quoted 8 lines
Use
folio_mk_pmd()to create a pmd for a folio cleanly.Fixes 6c88f72691f8 ("mm/huge_memory: add vmf_insert_folio_pmd()")Reviewed-by Jason Gunthorpe <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Reviewed-by Dan Williams <[email protected]>Tested-by Dan Williams <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Quoted 3 lines
Altough we have it quite well explained here in the changelog, maybe having a little comment in
insert_pmd()noting why pmds mapping normal folios cannot be marked special would be nice.
Well, I don't think we should be replicating that all over the place. The big comment above vm_normal_page() is currently our source of truth (which I will teak soon further).
Thanks!
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 2/3] mm/huge_memory: don't mark refcounted folios special in vmf_insert_folio_pmd()
Quoted 31 lines · David Hildenbrand
Quoted 30 lines · Oscar Salvador
Quoted 28 lines · David Hildenbrand
Marking PMDs that map a "normal" refcounted folios as special is against our rules documented for
vm_normal_page(): normal (refcounted) folios shall never have the page table mapping marked as special.Fortunately, there are not that many
pmd_special()check that can be mislead, and most vm_normal_page_pmd()/vm_normal_folio_pmd() users that would get this wrong right now are rather harmless: e.g., none so far bases decisions whether to grab a folio reference on that decision.Well, and GUP-fast will fallback to GUP-slow. All in all, so far no big implications as it seems.
Getting this right will get more important as we use
folio_normal_page_pmd()in more places.Fix it by teaching
insert_pfn_pmd()to properly handle folios and pfns -- moving refcount/mapcount/etc handling in there, renaming it toinsert_pmd(), and distinguishing between both cases using a new simple "structfolio_or_pfn" structure.Use
folio_mk_pmd()to create a pmd for a folio cleanly.Fixes 6c88f72691f8 ("mm/huge_memory: add vmf_insert_folio_pmd()")Reviewed-by Jason Gunthorpe <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Reviewed-by Dan Williams <[email protected]>Tested-by Dan Williams <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Quoted 7 lines
Quoted 3 lines
Altough we have it quite well explained here in the changelog, maybe having a little comment in
insert_pmd()noting why pmds mapping normal folios cannot be marked special would be nice.Well, I don't think we should be replicating that all over the place. The big comment above
vm_normal_page()is currently our source of truth (which I will teak soon further).
Suggestion:
"Kinda self-explanatory (special means don't touch) unless you use museum piece hardware OR IF YOU ARE XEN!"
;)
Quoted 8 lines
Thanks!
--
Cheers,David / dhildenb
Re: [PATCH v3 2/3] mm/huge_memory: don't mark refcounted folios special in vmf_insert_folio_pmd()
Quoted 40 lines · Lorenzo Stoakes
Quoted 31 lines · David Hildenbrand
Quoted 30 lines · Oscar Salvador
Quoted 28 lines · David Hildenbrand
Marking PMDs that map a "normal" refcounted folios as special is against our rules documented for
vm_normal_page(): normal (refcounted) folios shall never have the page table mapping marked as special.Fortunately, there are not that many
pmd_special()check that can be mislead, and most vm_normal_page_pmd()/vm_normal_folio_pmd() users that would get this wrong right now are rather harmless: e.g., none so far bases decisions whether to grab a folio reference on that decision.Well, and GUP-fast will fallback to GUP-slow. All in all, so far no big implications as it seems.
Getting this right will get more important as we use
folio_normal_page_pmd()in more places.Fix it by teaching
insert_pfn_pmd()to properly handle folios and pfns -- moving refcount/mapcount/etc handling in there, renaming it toinsert_pmd(), and distinguishing between both cases using a new simple "structfolio_or_pfn" structure.Use
folio_mk_pmd()to create a pmd for a folio cleanly.Fixes 6c88f72691f8 ("mm/huge_memory: add vmf_insert_folio_pmd()")Reviewed-by Jason Gunthorpe <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Reviewed-by Dan Williams <[email protected]>Tested-by Dan Williams <[email protected]>Signed-off-by David Hildenbrand <[email protected]>Quoted 7 lines
Quoted 3 lines
Altough we have it quite well explained here in the changelog, maybe having a little comment in
insert_pmd()noting why pmds mapping normal folios cannot be marked special would be nice.Well, I don't think we should be replicating that all over the place. The big comment above
vm_normal_page()is currently our source of truth (which I will teak soon further).
Quoted 6 lines
Suggestion:
"Kinda self-explanatory (special means don't touch) unless you use museum piece hardware OR IF YOU ARE XEN!"
;)
I looked into the XEN stuff and it is *extremely* nasty.
No, it doesn't do a pte_mkspecial(). It updates the PTE using ...
!!! A HYPERCALL !!!
WTF, why did we ever allow that.
It's documented to require GUP to work because ... QEMU AIO. Otherwise we could easily convert it to a proper PFNMAP.
Signature · 3 lines
Cheers, David / dhildenb
PATCH v3 3/3 mm/huge_memory: don't mark refcounted folios special in vmf_insert_folio_pud()
In reply to David HildenbrandMarking PUDs that map a "normal" refcounted folios as special is against our rules documented for vm_normal_page(). normal (refcounted) folios shall never have the page table mapping marked as special.
Fortunately, there are not that many pud_special() check that can be mislead and are right now rather harmless: e.g., none so far bases decisions whether to grab a folio reference on that decision.
Well, and GUP-fast will fallback to GUP-slow. All in all, so far no big implications as it seems.
Getting this right will get more important as we introduce folio_normal_page_pud() and start using it in more place where we currently special-case based on other VMA flags.
Fix it just like we fixed vmf_insert_folio_pmd().
Add folio_mk_pud() to mimic what we do with folio_mk_pmd().
--- include/linux/mm.h | 19 ++++++++++++++++- mm/huge_memory.c | 52 ++++++++++++++++++++++++++-------------------- 2 files changed, 47 insertions(+), 24 deletions(-)
include/linux/mm.h +18 -1
@@ -1816,7 +1816,24 @@ static inline pmd_t folio_mk_pmd(struct folio *folio, pgprot_t pgprot) {return pmd_mkhuge (pfn_pmd (folio_pfn (folio), pgprot)); } -#endif + +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +/** +* folio_mk_pud - Create a PUD for this folio +* @folio: The folio to create a PUD for +* @pgprot: The page protection bits to use +* +* Create a page table entry for the first page of this folio. +* This is suitable for passing to set_pud_at(). +* +* Return: A page table entry suitable for mapping this folio. +*/ +static inline pud_t folio_mk_pud (struct folio *folio,pgprot_t pgprot) +{ +return pud_mkhuge (pfn_pud (folio_pfn (folio), pgprot)); +} +#endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* CONFIG_MMU */ static inline bool folio_has_pincount (const struct folio *folio)
mm/huge_memory.c +29 -23
94 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH v3 3/3] mm/huge_memory: don't mark refcounted folios special in vmf_insert_folio_pud()
Quoted 17 lines · David Hildenbrand
Marking PUDs that map a "normal" refcounted folios as special is against our rules documented for
vm_normal_page(). normal (refcounted) folios shall never have the page table mapping marked as special.Fortunately, there are not that many
pud_special()check that can be mislead and are right now rather harmless: e.g., none so far bases decisions whether to grab a folio reference on that decision.Well, and GUP-fast will fallback to GUP-slow. All in all, so far no big implications as it seems.
Getting this right will get more important as we introduce
folio_normal_page_pud()and start using it in more place where we currently special-case based on other VMA flags.Fix it just like we fixed
vmf_insert_folio_pmd().
Quoted 8 lines
Add
folio_mk_pud()to mimic what we do withfolio_mk_pmd().Fixes dbe54153296d ("mm/huge_memory: add vmf_insert_folio_pud()")Reviewed-by Dan Williams <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Reviewed-by Jason Gunthorpe <[email protected]>Tested-by Dan Williams <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Signature · 2 lines
Oscar Salvador SUSE Labs