Convert 64-bit x86/mm/pat to ptdescs
Subsystems (3)
v3 cover Convert 64-bit x86/mm/pat to ptdescs
x86/mm/pat should be using ptdescs. One line has already been converted to pagetable_free(), while the allocation sites use get_free_pages(). This causes issues separately allocating ptdescs from struct page.
These patches convert the allocation/free sites to use ptdescs. In the short term, this helps enable Matthew's work to allocate frozen pagetables[1]. And in the long term, this will help us cleanly split ptdesc allocations from struct page.
The pgd_list should also be using ptdescs (for 32bit in this file). This can be done in a different patchset since there's other users of pgd_list that still need to be converted.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
------
I've also tested this on a tree that separately allocates ptdescs. That didn't find any lingering alloc/free issues.
Based on current mm-new.
v3:
- Move comment regarding 32-bit conversions into the cover letter
- Correct the handling for the pagetable_alloc() error path
Vishal Moola (Oracle) (3):
x86/mm/pat: Convert pte code to use ptdescs x86/mm/pat: Convert pmd code to use ptdescs x86/mm/pat: Convert split_large_page() to use ptdescs
arch/x86/mm/pat/set_memory.c | 56 +++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 23 deletions(-)
Signature · 1 line
2.52.0
PATCH v3 1/3 x86/mm/pat: Convert pte code to use ptdescs
In order to separately allocate ptdescs from pages, we need all allocation and free sites to use the appropriate functions. Convert these pte allocation/free sites to use ptdescs.
--- arch/x86/mm/pat/set_memory.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/mm/pat/set_memory.c +9 -6
@@ -1408,7 +1408,7 @@ static bool try_to_free_pte_page(pte_t *pte)if (!pte_none (pte[i]))return false; -free_page ((unsigned long )pte); +pagetable_free (virt_to_ptdesc ((void *)pte));return true; }
@@ -1537,12 +1537,15 @@ static void unmap_pud_range(p4d_t *p4d, unsigned long start, unsigned long end)*/ } -static int alloc_pte_page (pmd_t *pmd) +static int alloc_pte_ptdesc (pmd_t *pmd) { -pte_t *pte = (pte_t *)get_zeroed_page (GFP_KERNEL ); -if (!pte) +pte_t *pte; +struct ptdesc *ptdesc =pagetable_alloc (GFP_KERNEL | __GFP_ZERO,0 ); + +if (!ptdesc)return -1 ; + pte = (pte_t *)ptdesc_address (ptdesc);set_pmd (pmd,__pmd (__pa (pte) | _KERNPG_TABLE));return 0 ; }
@@ -1600,7 +1603,7 @@ static long populate_pmd(struct cpa_data *cpa,*/ pmd =pmd_offset (pud, start);if (pmd_none (*pmd)) -if (alloc_pte_page (pmd)) +if (alloc_pte_ptdesc (pmd))return -1 ;populate_pte (cpa, start, pre_end, cur_pages, pmd, pgprot);
@@ -1641,7 +1644,7 @@ static long populate_pmd(struct cpa_data *cpa,if (start < end) { pmd =pmd_offset (pud, start);if (pmd_none (*pmd)) -if (alloc_pte_page (pmd)) +if (alloc_pte_ptdesc (pmd))return -1 ; populate_pte(cpa, start, end, num_pages - cur_pages,
Signature · 1 line
2.52.0
Re: [PATCH v3 1/3] x86/mm/pat: Convert pte code to use ptdescs
Hi Vishal,
Quoted 23 lines · Vishal Moola (Oracle)
In order to separately allocate ptdescs from pages, we need all allocation and free sites to use the appropriate functions. Convert these pte allocation/free sites to use ptdescs.
Signed-off-by Vishal Moola (Oracle) <[email protected]>--- arch/x86/mm/pat/set_memory.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)arch/x86/mm/pat/set_memory.c +1
-1@@ -1408,7 +1408,7 @@ static bool try_to_free_pte_page(pte_t *pte)if (!pte_none (pte[i]))return false; -free_page ((unsigned long )pte); +pagetable_free (virt_to_ptdesc ((void *)pte));return true; }
arch/x86/mm/pat/set_memory.c static void unmap_pud_range(p4d_t *p4d, unsigned long start, unsigned long end) · line 1543
Quoted 10 lines
arch/x86/mm/pat/set_memory.c +3
-3@@ -1537,12 +1537,15 @@ static void unmap_pud_range(p4d_t *p4d, unsigned long start, unsigned long end)*/ } -static int alloc_pte_page (pmd_t *pmd) +static int alloc_pte_ptdesc (pmd_t *pmd) { -pte_t *pte = (pte_t *)get_zeroed_page (GFP_KERNEL ); -if (!pte) +pte_t *pte; +struct ptdesc *ptdesc =pagetable_alloc (GFP_KERNEL | __GFP_ZERO,0 );
AFAIR, x86 folks like reverse xmas tree for variable declarations.
arch/x86/mm/pat/set_memory.c static void unmap_pud_range(p4d_t *p4d, unsigned long start, unsigned long end) · line 1548
Quoted 5 lines
arch/x86/mm/pat/set_memory.c +3
-0+ +if (!ptdesc)return -1 ; + pte = (pte_t *)ptdesc_address (ptdesc);
No need to cast void * to another pointer type.
Same comments are relevant for two other patches as well.
Quoted 24 lines
arch/x86/mm/pat/set_memory.c +2
-2set_pmd (pmd,__pmd (__pa (pte) | _KERNPG_TABLE));return 0 ; }@@ -1600,7 +1603,7 @@ static long populate_pmd(struct cpa_data *cpa,*/ pmd =pmd_offset (pud, start);if (pmd_none (*pmd)) -if (alloc_pte_page (pmd)) +if (alloc_pte_ptdesc (pmd))return -1 ;populate_pte (cpa, start, pre_end, cur_pages, pmd, pgprot);@@ -1641,7 +1644,7 @@ static long populate_pmd(struct cpa_data *cpa,if (start < end) { pmd =pmd_offset (pud, start);if (pmd_none (*pmd)) -if (alloc_pte_page (pmd)) +if (alloc_pte_ptdesc (pmd))return -1 ; populate_pte(cpa, start, end, num_pages - cur_pages,Signature · 1 line
2.52.0
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v3 1/3] x86/mm/pat: Convert pte code to use ptdescs
In reply to Vishal Moola (Oracle)Quoted 3 lines · Vishal Moola (Oracle)
In order to separately allocate ptdescs from pages, we need all allocation and free sites to use the appropriate functions. Convert these pte allocation/free sites to use ptdescs.
Imperative voice, please.
arch/x86/mm/pat/set_memory.c static bool try_to_free_pte_page(pte_t *pte) · line 1413
Quoted 12 lines
arch/x86/mm/pat/set_memory.c +1
-1@@ -1408,7 +1408,7 @@ static bool try_to_free_pte_page(pte_t *pte)if (!pte_none (pte[i]))return false; -free_page ((unsigned long )pte); +pagetable_free (virt_to_ptdesc ((void *)pte));return true; }
This looks wrong to me, or at least that the API needs improvement. Most callers are going to have a pointer that they've been modifying. They're not going to have a ptdesc handy.
So I think this needs to look like:
pagetable_free(pte);
You can convert to ptdescs internally or do whatever you want with ptdesc sanity checks, but the API needs to be on writeable pointers. If the API takes a const pointer that requires callers to cast it, I think the API is broken.
arch/x86/mm/pat/set_memory.c static void unmap_pud_range(p4d_t *p4d, unsigned long start, unsigned long end) · line 1540
Quoted 6 lines
arch/x86/mm/pat/set_memory.c +1
-1@@ -1537,12 +1537,15 @@ static void unmap_pud_range(p4d_t *p4d, unsigned long start, unsigned long end)*/ } -static int alloc_pte_page (pmd_t *pmd) +static int alloc_pte_ptdesc (pmd_t *pmd)
Why change the name? Nobody cares what this is doing internally.
arch/x86/mm/pat/set_memory.c static void unmap_pud_range(p4d_t *p4d, unsigned long start, unsigned long end) · line 1546
Quoted 8 lines
arch/x86/mm/pat/set_memory.c +4
-2{ -pte_t *pte = (pte_t *)get_zeroed_page (GFP_KERNEL ); -if (!pte) +pte_t *pte; +struct ptdesc *ptdesc =pagetable_alloc (GFP_KERNEL | __GFP_ZERO,0 ); + +if (!ptdesc)return -1 ;
This also looks wrong.
What kind of maniac is ever going to allocate page tables without __GFP_ZERO? __GFP_ZERO really should be a part of pagetable_alloc(), don't you think?
arch/x86/mm/pat/set_memory.c static void unmap_pud_range(p4d_t *p4d, unsigned long start, unsigned long end) · line 1551
Quoted 4 lines
arch/x86/mm/pat/set_memory.c +1
-0+ pte = (pte_t *)ptdesc_address (ptdesc);set_pmd (pmd,__pmd (__pa (pte) | _KERNPG_TABLE));return 0 ; }
Why is there a cast here? ptdesc_address() returns void*, no?
Also, if there a ptdesc_pa(), this could be:
static int alloc_pte_ptdesc(pmd_t *pmd)
{
struct ptdesc *ptdesc = pagetable_alloc(GFP_KERNEL, 0);
if (!ptdesc)return -1;
set_pmd(pmd, __pmd(ptdesc_pa(ptdesc) | _KERNPG_TABLE)); return 0;
}
This *should* be a very common pattern. After you allocate a page table page, you almost always need its physical address because it's going to get pointed to by other page table or hardware register.
To me, it doesn't look like the ptdesc API is very mature yet, or at least hasn't been expanded for ease for actual users. I don't want to grow its use in arch/x86 until it's a wee bit more mature.
Re: [PATCH v3 1/3] x86/mm/pat: Convert pte code to use ptdescs
Quoted 6 lines · Dave Hansen
Quoted 3 lines · Vishal Moola (Oracle)
In order to separately allocate ptdescs from pages, we need all allocation and free sites to use the appropriate functions. Convert these pte allocation/free sites to use ptdescs.
Imperative voice, please.
I'll fix it.
Quoted 8 lines
Quoted 8 lines
arch/x86/mm/pat/set_memory.c +0
-0@@ -1408,7 +1408,7 @@ static bool try_to_free_pte_page(pte_t *pte)if (!pte_none (pte[i]))return false;
Quoted 8 lines
arch/x86/mm/pat/set_memory.c static bool try_to_free_pte_page(pte_t *pte) · line 1413
Quoted 4 lines
arch/x86/mm/pat/set_memory.c +1
-1-free_page ((unsigned long )pte); +pagetable_free (virt_to_ptdesc ((void *)pte));return true; }This looks wrong to me, or at least that the API needs improvement. Most callers are going to have a pointer that they've been modifying. They're not going to have a ptdesc handy.
Yeah the API needs improvement. The initial API I wrote was very barebones back when I didn't understand enough about arch differences and similarities in page table implementation.
Quoted 8 lines
So I think this needs to look like:
pagetable_free(pte);You can convert to ptdescs internally or do whatever you want with ptdesc sanity checks, but the API needs to be on writeable pointers. If the API takes a const pointer that requires callers to cast it, I think the API is broken.
Your logic makes sense to me. I can add ptdesc-using-address apis.
Quoted 20 lines
arch/x86/mm/pat/set_memory.c static void unmap_pud_range(p4d_t *p4d, unsigned long start, unsigned long end) · line 1540
Quoted 6 lines
arch/x86/mm/pat/set_memory.c +1
-1@@ -1537,12 +1537,15 @@ static void unmap_pud_range(p4d_t *p4d, unsigned long start, unsigned long end)*/ } -static int alloc_pte_page (pmd_t *pmd) +static int alloc_pte_ptdesc (pmd_t *pmd)Why change the name? Nobody cares what this is doing internally.
arch/x86/mm/pat/set_memory.c static void unmap_pud_range(p4d_t *p4d, unsigned long start, unsigned long end) · line 1546
Quoted 8 lines
arch/x86/mm/pat/set_memory.c +4
-2{ -pte_t *pte = (pte_t *)get_zeroed_page (GFP_KERNEL ); -if (!pte) +pte_t *pte; +struct ptdesc *ptdesc =pagetable_alloc (GFP_KERNEL | __GFP_ZERO,0 ); + +if (!ptdesc)return -1 ;This also looks wrong.
Quoted 3 lines
What kind of maniac is ever going to allocate page tables without
__GFP_ZERO?__GFP_ZEROreally should be a part ofpagetable_alloc(), don't you think?
I thought the same thing... Turns out some architectures do. I didn't question it, they might not even have good reason to do so.
Regardless, I do agree with you. I'm tempted to include __GFP_ZERO as part of the ptdesc-using-address apis.
Quoted 6 lines
arch/x86/mm/pat/set_memory.c static void unmap_pud_range(p4d_t *p4d, unsigned long start, unsigned long end) · line 1551
Quoted 4 lines
arch/x86/mm/pat/set_memory.c +1
-0+ pte = (pte_t *)ptdesc_address (ptdesc);set_pmd (pmd,__pmd (__pa (pte) | _KERNPG_TABLE));return 0 ; }Why is there a cast here?
ptdesc_address()returns void*, no?
Yes it does.
Personally, I view casts as human hints to make implicit conversions obvious. I didn't think it hurt readability so I left it in.
I don't have strong feelings either way, I can remove the casts. The type is obvious enough here anyway.
Quoted 13 lines
Also, if there a
ptdesc_pa(), this could be:static int
alloc_pte_ptdesc(pmd_t *pmd){ struct ptdesc *ptdesc = pagetable_alloc(GFP_KERNEL, 0); if (!ptdesc)arch/x86/mm/pat/set_memory.c +0
-0return -1 ;set_pmd (pmd,__pmd (ptdesc_pa (ptdesc) | _KERNPG_TABLE));return 0 ;}
Quoted 3 lines
This *should* be a very common pattern. After you allocate a page table page, you almost always need its physical address because it's going to get pointed to by other page table or hardware register.
I don't recall running into this pattern much, but I'll take a look. It sounds sensible. If this happens far from the allocation sites, I definitely would've missed them.
Quoted 3 lines
To me, it doesn't look like the ptdesc API is very mature yet, or at least hasn't been expanded for ease for actual users. I don't want to grow its use in
arch/x86until it's a wee bit more mature.
I truly appreciate the review and comments :).
The use of struct ptdesc in cpa_collapse_large_pages() gets in the way of short-term (stop refcounting page tables) and long term (shrinking struct page) goals. Particularly the pagetable_free() call.
Would you be ok with taking these patches if I add these relevant apis:
1) A function that returns and address (like get_zeroed_page())
2) A function that frees by address (like free_page())
Re: [PATCH v3 1/3] x86/mm/pat: Convert pte code to use ptdescs
Quoted 3 lines · Vishal Moola (Oracle)
Would you be ok with taking these patches if I add these relevant apis:
1) A function that returns and address (likeget_zeroed_page())
2) A function that frees by address (likefree_page())
I don't see any problems taking them if those changes are made. No promises, but I don't see any other issues.
PATCH v3 2/3 x86/mm/pat: Convert pmd code to use ptdescs
In reply to Vishal Moola (Oracle)In order to separately allocate ptdescs from pages, we need all allocation and free sites to use the appropriate functions. Convert these pmd allocation/free sites to use ptdescs.
populate_pgd() also allocates pagetables that may later be freed by try_to_free_pmd_page(), so allocate ptdescs there as well.
--- arch/x86/mm/pat/set_memory.c | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-)
arch/x86/mm/pat/set_memory.c +17 -11
@@ -1420,7 +1420,7 @@ static bool try_to_free_pmd_page(pmd_t *pmd)if (!pmd_none (pmd[i]))return false; -free_page ((unsigned long )pmd); +pagetable_free (virt_to_ptdesc ((void *)pmd));return true; }
@@ -1550,12 +1550,15 @@ static int alloc_pte_ptdesc(pmd_t *pmd)return 0 ; } -static int alloc_pmd_page (pud_t *pud) +static int alloc_pmd_ptdesc (pud_t *pud) { -pmd_t *pmd = (pmd_t *)get_zeroed_page (GFP_KERNEL ); -if (!pmd) +pmd_t *pmd; +struct ptdesc *ptdesc =pagetable_alloc (GFP_KERNEL | __GFP_ZERO,0 ); + +if (!ptdesc)return -1 ; + pmd = (pmd_t *)ptdesc_address (ptdesc);set_pud (pud,__pud (__pa (pmd) | _KERNPG_TABLE));return 0 ; }
@@ -1625,7 +1628,7 @@ static long populate_pmd(struct cpa_data *cpa,* We cannot use a 1G page so allocate a PMD page if needed. */ if (pud_none (*pud)) -if (alloc_pmd_page (pud)) +if (alloc_pmd_ptdesc (pud))return -1 ; pmd =pmd_offset (pud, start);
@@ -1681,7 +1684,7 @@ static int populate_pud(struct cpa_data *cpa, unsigned long start, p4d_t *p4d,* Need a PMD page? */ if (pud_none (*pud)) -if (alloc_pmd_page (pud)) +if (alloc_pmd_ptdesc (pud))return -1 ; cur_pages = populate_pmd(cpa, start, pre_end, cur_pages,
@@ -1718,7 +1721,7 @@ static int populate_pud(struct cpa_data *cpa, unsigned long start, p4d_t *p4d, pud =pud_offset (p4d, start);if (pud_none (*pud)) -if (alloc_pmd_page (pud)) +if (alloc_pmd_ptdesc (pud))return -1 ; tmp = populate_pmd(cpa, start, end, cpa->numpages - cur_pages,
@@ -1742,14 +1745,16 @@ static int populate_pgd(struct cpa_data *cpa, unsigned long addr)p4d_t *p4d;pgd_t *pgd_entry;long ret; +struct ptdesc *ptdesc; pgd_entry = cpa->pgd +pgd_index (addr);if (pgd_none (*pgd_entry)) { - p4d = (p4d_t *)get_zeroed_page (GFP_KERNEL ); -if (!p4d) + ptdesc =pagetable_alloc (GFP_KERNEL | __GFP_ZERO,0 ); +if (!ptdesc)return -1 ; + p4d = (p4d_t *)ptdesc_address (ptdesc);set_pgd (pgd_entry,__pgd (__pa (p4d) | _KERNPG_TABLE)); }
@@ -1758,10 +1763,11 @@ static int populate_pgd(struct cpa_data *cpa, unsigned long addr)*/ p4d =p4d_offset (pgd_entry, addr);if (p4d_none (*p4d)) { - pud = (pud_t *)get_zeroed_page (GFP_KERNEL ); -if (!pud) + ptdesc =pagetable_alloc (GFP_KERNEL | __GFP_ZERO,0 ); +if (!ptdesc)return -1 ; + pud = (pud_t *)ptdesc_address (ptdesc);set_p4d (p4d,__p4d (__pa (pud) | _KERNPG_TABLE)); }
Signature · 1 line
2.52.0
PATCH v3 3/3 x86/mm/pat: Convert split_large_page() to use ptdescs
In reply to Vishal Moola (Oracle)In order to separately allocate ptdescs from pages, we need all allocation and free sites to use the appropriate functions.
split_large_page() allocates a page to be used as a page table. This should be allocating a ptdesc, so convert it.
--- arch/x86/mm/pat/set_memory.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
arch/x86/mm/pat/set_memory.c +7 -6
@@ -1119,9 +1119,10 @@ static void split_set_pte(struct cpa_data *cpa, pte_t *pte, unsigned long pfn,static int __split_large_page (struct cpa_data *cpa,pte_t *kpte,unsigned long address, -struct page *base) +struct ptdesc *ptdesc) {unsigned long lpaddr, lpinc, ref_pfn, pfn, pfninc =1 ; +struct page *base =ptdesc_page (ptdesc);pte_t *pbase = (pte_t *)page_address (base);unsigned int i, level;pgprot_t ref_prot;
@@ -1226,18 +1227,18 @@ __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address,static int split_large_page (struct cpa_data *cpa,pte_t *kpte,unsigned long address) { -struct page *base; +struct ptdesc *ptdesc;if (!debug_pagealloc_enabled ())spin_unlock (&cpa_lock); - base =alloc_pages (GFP_KERNEL ,0 ); + ptdesc =pagetable_alloc (GFP_KERNEL ,0 );if (!debug_pagealloc_enabled ())spin_lock (&cpa_lock); -if (!base) +if (!ptdesc)return -ENOMEM ; -if (__split_large_page (cpa, kpte, address, base)) -__free_page (base); +if (__split_large_page (cpa, kpte, address, ptdesc)) +pagetable_free (ptdesc);return 0 ; }
Signature · 1 line
2.52.0
Quoted 24 lines · Vishal Moola (Oracle)
x86/mm/pat should be using ptdescs. One line has already been converted to
pagetable_free(), while the allocation sites useget_free_pages(). This causes issues separately allocating ptdescs from struct page.These patches convert the allocation/free sites to use ptdescs. In the short term, this helps enable Matthew's work to allocate frozen pagetables[1]. And in the long term, this will help us cleanly split ptdesc allocations from struct page.
The
pgd_listshould also be using ptdescs (for 32bit in this file). This can be done in a different patchset since there's other users ofpgd_listthat still need to be converted.[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/------
I've also tested this on a tree that separately allocates ptdescs. That didn't find any lingering alloc/free issues.
Based on current mm-new.
Quoted 8 lines
v3:
- Move comment regarding 32-bit conversions into the cover letter
- Correct the handling for thepagetable_alloc()error pathVishal Moola (Oracle) (3):
x86/mm/pat: Convert pte code to use ptdescs x86/mm/pat: Convert pmd code to use ptdescs x86/mm/pat: Convertsplit_large_page()to use ptdescs
With some minor comments in patch 1/3
Quoted 6 lines
arch/x86/mm/pat/set_memory.c | 56 +++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 23 deletions(-)Signature · 1 line
2.52.0
Signature · 2 lines
Sincerely yours, Mike.