Convert 64-bit x86/mm/pat to ptdescs
Subsystems (4)
v4 cover Convert 64-bit x86/mm/pat to ptdescs
x86/mm/pat should be using ptdescs. One line has already been converted to pagetable_free(), while the allocation sites use get_free_pages(). This causes issues separately allocating ptdescs from struct page.
The first patch introduces new ptdesc apis that operate on addresses. These are like get_free_pages() and free_pages() helper functions.
The remaining patches convert the allocation/free sites to use ptdescs. In the short term, this helps enable Matthew's work to allocate frozen pagetables[1]. And in the long term, this will help us cleanly split ptdesc allocations from struct page[2].
The pgd_list should also be using ptdescs (for 32bit in this file). This can be done in a different patchset since there's other users of pgd_list that still need to be converted.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
------
I've also tested this on a tree that separately allocates ptdescs. That didn't find any lingering alloc/free issues.
Based on current mm-new.
v4:
- Added new ptdesc apis
- Also renamed the pte/pmd free functions as well
- Explained reasoning for renaming functions in the commit logs
- Updated the cover-letter
v3 link:
https://lore.kernel.org/linux-mm/[email protected]/
Vishal Moola (Oracle) (4):
mm: Add address apis for ptdescs
x86/mm/pat: Convert pte code to use ptdescs x86/mm/pat: Convert pmd code to use ptdescs x86/mm/pat: Convert split_large_page() to use ptdescs
arch/x86/mm/pat/set_memory.c | 49 ++++++++++++++++++------------------ include/linux/mm.h | 4 +++ mm/memory.c | 34 +++++++++++++++++++++++++ 3 files changed, 63 insertions(+), 24 deletions(-)
Signature · 1 line
2.52.0
PATCH v4 1/4 mm: Add address apis for ptdescs
Architectures frequently only care about the address associated with a page table. The current ptdesc api forced callers to acquire a ptdesc to use them. Add more apis to abstract ptdescs away from architectures that don't need the descriptor.
This patch adds pgtable_alloc() and pgtable_free() to operate on the underlying addresses associated with page table descriptors, similar to get_free_pages() and free_pages(). The allocations will be zeroed since theres no reason to want a page table with stale data.
--- include/linux/mm.h | 4 ++++ mm/memory.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+)
include/linux/mm.h +4 -0
@@ -3419,6 +3419,10 @@ static inline void __pagetable_free(struct ptdesc *pt)__free_pages (page,compound_order (page)); } +unsigned long pgtable_alloc_addr_noprof (gfp_t gfp,unsigned int order); +#define pgtable_alloc_addr (...) alloc_hooks(pgtable_alloc_addr_noprof(__VA_ARGS__)) +void pgtable_free_addr (const void *addr); +#ifdef CONFIG_ASYNC_KERNEL_PGTABLE_FREE void pagetable_free_kernel (struct ptdesc *pt);#else
mm/memory.c +34 -0
@@ -7451,6 +7451,40 @@ long copy_folio_from_user(struct folio *dst_folio, }#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS */ +/** +* pgtable_alloc_addr - Allocate pagetables to get an address +* @gfp: GFP flags +* @order: desired pagetable order +* +* pgtable_alloc_addr is like pagetable_alloc. This is for callers who only want a +* page table's address, not its ptdesc. +* +* Return: The address associated with the allocated page table, or 0 on +* failure. +*/ +unsigned long pgtable_alloc_addr_noprof (gfp_t gfp,unsigned int order) +{ +struct ptdesc *ptdesc =pagetable_alloc_noprof (gfp | __GFP_ZERO, order); + +if (!ptdesc) +return 0 ; +return (unsigned long )ptdesc_address (ptdesc); +} + +/** +* pgtable_free_addr - Free pagetables by address +* @addr: The virtual address from pgtable_alloc() +* +* This function is for callers who have the address but no ptdesc. If you +* have the ptdesc, use pagetable_free() instead. +*/ +void pgtable_free_addr (const void *addr) +{ +struct ptdesc *ptdesc =virt_to_ptdesc (addr); + +pagetable_free (ptdesc); +} +#if defined(CONFIG_SPLIT_PTE_PTLOCKS ) &&ALLOC_SPLIT_PTLOCKS static struct kmem_cache *page_ptl_cachep;
Signature · 1 line
2.52.0
Mike Rapoport
reviewerRe: [PATCH v4 1/4] mm: Add address apis for ptdescs
Hi Vishal,
Quoted 22 lines · Vishal Moola (Oracle)
Architectures frequently only care about the address associated with a page table. The current ptdesc api forced callers to acquire a ptdesc to use them. Add more apis to abstract ptdescs away from architectures that don't need the descriptor.
This patch adds
pgtable_alloc()andpgtable_free()to operate on the underlying addresses associated with page table descriptors, similar toget_free_pages()andfree_pages(). The allocations will be zeroed since theres no reason to want a page table with stale data.Suggested-by Dave Hansen <[email protected]>Signed-off-by Vishal Moola (Oracle) <[email protected]>--- include/linux/mm.h | 4 ++++ mm/memory.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+)include/linux/mm.h +0
-0
include/linux/mm.h static inline void __pagetable_free(struct ptdesc *pt) · line 3422
Quoted 4 lines
include/linux/mm.h +1
-0@@ -3419,6 +3419,10 @@ static inline void __pagetable_free(struct ptdesc *pt)__free_pages (page,compound_order (page)); } +unsigned long pgtable_alloc_addr_noprof (gfp_t gfp,unsigned int order);
Why not void *?
I don't think it should follow ancient get_zeroed_page() and force all callers to do the casting when they are expecting a pointer anyway.
Quoted 2 lines
include/linux/mm.h +2
-0+#define pgtable_alloc_addr (...) alloc_hooks(pgtable_alloc_addr_noprof(__VA_ARGS__)) +void pgtable_free_addr (const void *addr);
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v4 1/4] mm: Add address apis for ptdescs
Quoted 28 lines · Mike Rapoport
Hi Vishal,
Quoted 25 lines · Vishal Moola (Oracle)
Architectures frequently only care about the address associated with a page table. The current ptdesc api forced callers to acquire a ptdesc to use them. Add more apis to abstract ptdescs away from architectures that don't need the descriptor.
This patch adds
pgtable_alloc()andpgtable_free()to operate on the underlying addresses associated with page table descriptors, similar toget_free_pages()andfree_pages(). The allocations will be zeroed since theres no reason to want a page table with stale data.Suggested-by Dave Hansen <[email protected]>Signed-off-by Vishal Moola (Oracle) <[email protected]>--- include/linux/mm.h | 4 ++++ mm/memory.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+)include/linux/mm.h +0
-0@@ -3419,6 +3419,10 @@ static inline void __pagetable_free(struct ptdesc *pt)__free_pages (page,compound_order (page)); }
Quoted 5 lines
include/linux/mm.h static inline void __pagetable_free(struct ptdesc *pt) · line 3422
Quoted 1 line
include/linux/mm.h +1
-0+unsigned long pgtable_alloc_addr_noprof (gfp_t gfp,unsigned int order);Why not void *?
I don't think it should follow ancientget_zeroed_page()and force all callers to do the casting when they are expecting a pointer anyway.
Agreed, I'll change that in the next version.
I had misread an s390 function that expects an (unsigned long *) and assumed some callers might want the unsigned long, thinking we'd have to cast anyway. After taking a closer look I see now that every single caller actually expects a pointer.
PATCH v4 2/4 x86/mm/pat: Convert pte code to use ptdescs
In reply to Vishal Moola (Oracle)We need all allocation and free sites to use the ptdesc APIs in order to allocate them separately from regular pages. Convert these pte allocation/free sites to use ptdescs.
Also, rename *_pte_page() functions to *_pte(). Rename them now to avoid any confusion later. Eventually these allocations will be backed by a ptdesc not a page, but that's not important to callers either.
--- arch/x86/mm/pat/set_memory.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
arch/x86/mm/pat/set_memory.c +7 -7
@@ -1400,7 +1400,7 @@ static int collapse_large_pages(unsigned long addr, struct list_head *pgtables)return collapsed; } -static bool try_to_free_pte_page (pte_t *pte) +static bool try_to_free_pte (pte_t *pte) {int i;
@@ -1408,7 +1408,7 @@ static bool try_to_free_pte_page(pte_t *pte)if (!pte_none (pte[i]))return false; -free_page ((unsigned long )pte); +pgtable_free_addr (pte);return true; }
@@ -1435,7 +1435,7 @@ static bool unmap_pte_range(pmd_t *pmd, unsigned long start, unsigned long end) pte++; } -if (try_to_free_pte_page ((pte_t *)pmd_page_vaddr (*pmd))) { +if (try_to_free_pte ((pte_t *)pmd_page_vaddr (*pmd))) {pmd_clear (pmd);return true; }
@@ -1537,9 +1537,9 @@ static void unmap_pud_range(p4d_t *p4d, unsigned long start, unsigned long end)*/ } -static int alloc_pte_page (pmd_t *pmd) +static int alloc_pte (pmd_t *pmd) { -pte_t *pte = (pte_t *)get_zeroed_page (GFP_KERNEL ); +pte_t *pte = (pte_t *)pgtable_alloc_addr (GFP_KERNEL ,0 );if (!pte)return -1 ;
@@ -1600,7 +1600,7 @@ static long populate_pmd(struct cpa_data *cpa,*/ pmd =pmd_offset (pud, start);if (pmd_none (*pmd)) -if (alloc_pte_page (pmd)) +if (alloc_pte (pmd))return -1 ;populate_pte (cpa, start, pre_end, cur_pages, pmd, pgprot);
@@ -1641,7 +1641,7 @@ static long populate_pmd(struct cpa_data *cpa,if (start < end) { pmd =pmd_offset (pud, start);if (pmd_none (*pmd)) -if (alloc_pte_page (pmd)) +if (alloc_pte (pmd))return -1 ; populate_pte(cpa, start, end, num_pages - cur_pages,
Signature · 1 line
2.52.0
PATCH v4 3/4 x86/mm/pat: Convert pmd code to use ptdescs
In reply to Vishal Moola (Oracle)We need all allocation and free sites to use the ptdesc APIs in order to allocate them separately from regular pages. Convert these pmd allocation/free sites to use ptdescs.
populate_pgd() also allocates pagetables that may later be freed by try_to_free_pmd_page(), so allocate ptdescs there as well.
Also, rename *_pmd_page() functions to *_pmd(). Rename them now to avoid any confusion later. Eventually these allocations will be backed by a ptdesc not a page, but that's not important to callers either.
--- arch/x86/mm/pat/set_memory.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
arch/x86/mm/pat/set_memory.c +11 -11
@@ -1412,7 +1412,7 @@ static bool try_to_free_pte(pte_t *pte)return true; } -static bool try_to_free_pmd_page (pmd_t *pmd) +static bool try_to_free_pmd (pmd_t *pmd) {int i;
@@ -1420,7 +1420,7 @@ static bool try_to_free_pmd_page(pmd_t *pmd)if (!pmd_none (pmd[i]))return false; -free_page ((unsigned long )pmd); +pgtable_free_addr (pmd);return true; }
@@ -1446,7 +1446,7 @@ static void __unmap_pmd_range(pud_t *pud, pmd_t *pmd, unsigned long start, unsigned long end) { if (unmap_pte_range(pmd, start, end)) - if (try_to_free_pmd_page(pud_pgtable(*pud))) + if (try_to_free_pmd(pud_pgtable(*pud))) pud_clear(pud); }
@@ -1490,7 +1490,7 @@ static void unmap_pmd_range(pud_t *pud, unsigned long start, unsigned long end)* Try again to free the PMD page if haven't succeeded above. */ if (!pud_none (*pud)) -if (try_to_free_pmd_page (pud_pgtable (*pud))) +if (try_to_free_pmd (pud_pgtable (*pud)))pud_clear (pud); }
@@ -1547,9 +1547,9 @@ static int alloc_pte(pmd_t *pmd)return 0 ; } -static int alloc_pmd_page (pud_t *pud) +static int alloc_pmd (pud_t *pud) { -pmd_t *pmd = (pmd_t *)get_zeroed_page (GFP_KERNEL ); +pmd_t *pmd = (pmd_t *)pgtable_alloc_addr (GFP_KERNEL ,0 );if (!pmd)return -1 ;
@@ -1622,7 +1622,7 @@ static long populate_pmd(struct cpa_data *cpa,* We cannot use a 1G page so allocate a PMD page if needed. */ if (pud_none (*pud)) -if (alloc_pmd_page (pud)) +if (alloc_pmd (pud))return -1 ; pmd =pmd_offset (pud, start);
@@ -1678,7 +1678,7 @@ static int populate_pud(struct cpa_data *cpa, unsigned long start, p4d_t *p4d,* Need a PMD page? */ if (pud_none (*pud)) -if (alloc_pmd_page (pud)) +if (alloc_pmd (pud))return -1 ; cur_pages = populate_pmd(cpa, start, pre_end, cur_pages,
@@ -1715,7 +1715,7 @@ static int populate_pud(struct cpa_data *cpa, unsigned long start, p4d_t *p4d, pud =pud_offset (p4d, start);if (pud_none (*pud)) -if (alloc_pmd_page (pud)) +if (alloc_pmd (pud))return -1 ; tmp = populate_pmd(cpa, start, end, cpa->numpages - cur_pages,
@@ -1743,7 +1743,7 @@ static int populate_pgd(struct cpa_data *cpa, unsigned long addr) pgd_entry = cpa->pgd +pgd_index (addr);if (pgd_none (*pgd_entry)) { - p4d = (p4d_t *)get_zeroed_page (GFP_KERNEL ); + p4d = (p4d_t *)pgtable_alloc_addr (GFP_KERNEL ,0 );if (!p4d)return -1 ;
@@ -1755,7 +1755,7 @@ static int populate_pgd(struct cpa_data *cpa, unsigned long addr)*/ p4d =p4d_offset (pgd_entry, addr);if (p4d_none (*p4d)) { - pud = (pud_t *)get_zeroed_page (GFP_KERNEL ); + pud = (pud_t *)pgtable_alloc_addr (GFP_KERNEL ,0 );if (!pud)return -1 ;
Signature · 1 line
2.52.0
PATCH v4 4/4 x86/mm/pat: Convert split_large_page() to use ptdescs
In reply to Vishal Moola (Oracle)In order to separately allocate ptdescs from pages, we need all allocation and free sites to use the appropriate functions.
split_large_page() allocates a page to be used as a page table. This should be allocating a ptdesc, so convert it.
--- arch/x86/mm/pat/set_memory.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
arch/x86/mm/pat/set_memory.c +7 -6
@@ -1119,9 +1119,10 @@ static void split_set_pte(struct cpa_data *cpa, pte_t *pte, unsigned long pfn,static int __split_large_page (struct cpa_data *cpa,pte_t *kpte,unsigned long address, -struct page *base) +struct ptdesc *ptdesc) {unsigned long lpaddr, lpinc, ref_pfn, pfn, pfninc =1 ; +struct page *base =ptdesc_page (ptdesc);pte_t *pbase = (pte_t *)page_address (base);unsigned int i, level;pgprot_t ref_prot;
@@ -1226,18 +1227,18 @@ __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address,static int split_large_page (struct cpa_data *cpa,pte_t *kpte,unsigned long address) { -struct page *base; +struct ptdesc *ptdesc;if (!debug_pagealloc_enabled ())spin_unlock (&cpa_lock); - base =alloc_pages (GFP_KERNEL ,0 ); + ptdesc =pagetable_alloc (GFP_KERNEL ,0 );if (!debug_pagealloc_enabled ())spin_lock (&cpa_lock); -if (!base) +if (!ptdesc)return -ENOMEM ; -if (__split_large_page (cpa, kpte, address, base)) -__free_page (base); +if (__split_large_page (cpa, kpte, address, ptdesc)) +pagetable_free (ptdesc);return 0 ; }
Signature · 1 line
2.52.0