Skip to contentSkip to conversation

use struct ptdesc to replace pgtable_t

RFC PATCH In review [email protected] 4 participants 24 messages Jul 30, 2024 – Aug 15, 2024

Subsystems (33)
lore
Share & tools
lore
#0 lore raw

cover use struct ptdesc to replace pgtable_t

From: Alex Shi <[email protected]>

We have struct ptdesc for page table descriptor a year ago, but it has no much usages in kernel, while pgtable_t is used widely.

The pgtable_t is typedefed as 'pte_t *' in sparc, s390, powerpc and m68k except SUN3, others archs are all same as 'struct page *'.

These blocks the conception and code update for page table descriptor to struct ptdesc.

So, the simple idea to push the ptdesc conception forward is to update all pgtable_t by ptdesc or pte_t pointer. But this needs widely knowledges for most all of different archs. Common code change is easy for include/ and mm/ directory, but it's hard in all archs.

Thanks for intel LKP framework, I fixed most all of build issues except a bug on powerpc which reports a "struct ptdesc *" incompatible with struct ptdesc *' pointer issue...

Another trouble is pmd_pgtable() conversion in the last patch. Maybe some of arch need define theirself own pmd_ptdesc()?

This patchset is immature, even except above 2 issues, I just tested virutal machine booting and kselftest mm on x86 and arm64.

Anyway any input are appreciated!

Thanks
Alex

Alex Shi (18):
mm/pgtable: use ptdesc in pte_free_now/pte_free_defer mm/pgtable: convert ptdesc.pmd_huge_pte to ptdesc pointer fs/dax: use ptdesc in dax_pmd_load_hole
mm/thp: use ptdesc pointer in __do_huge_pmd_anonymous_page mm/thp: use ptdesc in do_huge_pmd_anonymous_page
mm/thp: convert insert_pfn_pmd and its caller to use ptdesc mm/thp: use ptdesc in copy_huge_pmd
mm/memory: use ptdesc in __pte_alloc
mm/pgtable: fully use ptdesc in pte_alloc_one series functions mm/pgtable: pass ptdesc to pte_free()
mm/pgtable: introduce ptdesc_pfn and use ptdesc in free_pte_range() mm/thp: pass ptdesc to set_huge_zero_folio function mm/pgtable: return ptdesc pointer in pgtable_trans_huge_withdraw mm/pgtable: use ptdesc in pgtable_trans_huge_deposit mm/pgtable: pass ptdesc to pmd_populate
mm/pgtable: pass ptdesc to pmd_install
mm: convert vmf.prealloc_pte to struct ptdesc pointer mm/pgtable: pass ptdesc in pte_free_defer

 arch/alpha/include/asm/pgalloc.h              |   4 +-
 arch/arc/include/asm/pgalloc.h                |   4 +-
 arch/arm/include/asm/pgalloc.h                |  13 +--
 arch/arm/include/asm/tlb.h                    |   4 +-
 arch/arm/mm/pgd.c                             |   2 +-
 arch/arm64/include/asm/pgalloc.h              |   4 +-
 arch/arm64/include/asm/tlb.h                  |   4 +-
 arch/csky/include/asm/pgalloc.h               |   4 +-
 arch/hexagon/include/asm/pgalloc.h            |   8 +-
 arch/loongarch/include/asm/pgalloc.h          |   8 +-
 arch/m68k/include/asm/motorola_pgalloc.h      |  12 +-
 arch/m68k/include/asm/sun3_pgalloc.h          |   4 +-
 arch/microblaze/include/asm/pgalloc.h         |   2 +-
 arch/mips/include/asm/pgalloc.h               |   4 +-
 arch/nios2/include/asm/pgalloc.h              |   4 +-
 arch/openrisc/include/asm/pgalloc.h           |   8 +-
 arch/parisc/include/asm/pgalloc.h             |   2 +-
 arch/powerpc/include/asm/book3s/32/pgalloc.h  |   4 +-
 arch/powerpc/include/asm/book3s/64/hash-4k.h  |   4 +-
 arch/powerpc/include/asm/book3s/64/hash-64k.h |   4 +-
 arch/powerpc/include/asm/book3s/64/pgalloc.h  |   4 +-
 arch/powerpc/include/asm/book3s/64/pgtable.h  |   8 +-
 arch/powerpc/include/asm/book3s/64/radix.h    |   4 +-
 arch/powerpc/include/asm/pgalloc.h            |   8 +-
 arch/powerpc/mm/book3s64/hash_pgtable.c       |  10 +-
 arch/powerpc/mm/book3s64/radix_pgtable.c      |  10 +-
 arch/riscv/include/asm/pgalloc.h              |   8 +-
 arch/s390/include/asm/pgalloc.h               |   4 +-
 arch/s390/include/asm/pgtable.h               |   4 +-
 arch/s390/mm/pgalloc.c                        |   2 +-
 arch/s390/mm/pgtable.c                        |  14 +--
 arch/sh/include/asm/pgalloc.h                 |   4 +-
 arch/sparc/include/asm/pgalloc_32.h           |   6 +-
 arch/sparc/include/asm/pgalloc_64.h           |   2 +-
 arch/sparc/include/asm/pgtable_64.h           |   4 +-
 arch/sparc/mm/init_64.c                       |   2 +-
 arch/sparc/mm/srmmu.c                         |   6 +-
 arch/sparc/mm/tlb.c                           |  14 +--
 arch/x86/include/asm/pgalloc.h                |  10 +-
 arch/x86/mm/pgtable.c                         |   8 +-
 arch/xtensa/include/asm/pgalloc.h             |  12 +-
 fs/dax.c                                      |  14 +--
 include/asm-generic/pgalloc.h                 |  10 +-
 include/linux/mm.h                            |  16 ++-
 include/linux/mm_types.h                      |   4 +-
 include/linux/pgtable.h                       |   6 +-
 mm/debug_vm_pgtable.c                         |   6 +-
 mm/huge_memory.c                              | 103 +++++++++---------
 mm/internal.h                                 |   2 +-
 mm/khugepaged.c                               |  14 +--
 mm/memory.c                                   |  15 +--
 mm/mremap.c                                   |   2 +-
 mm/pgtable-generic.c                          |  37 +++----
 53 files changed, 240 insertions(+), 236 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

RFC PATCH 1/18 mm/pgtable: use ptdesc in pte_free_now/pte_free_defer

From: Alex Shi <[email protected]>

page table descriptor is splited from struct page, use it to replace struct page in right place.

Cc Andrew Morton <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc David Hildenbrand <[email protected]>
---
 mm/pgtable-generic.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
mm/pgtable-generic.c +6 -7
index 13a7705df3f8..2ce714f1dd15 100644
@@ -238,18 +238,17 @@ pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address,
 #ifndef pte_free_defer
 static void pte_free_now(struct rcu_head *head)
 {
-	struct page *page;
+	struct ptdesc *ptdesc;
 
-	page = container_of(head, struct page, rcu_head);
-	pte_free(NULL /* mm not passed and not used */, (pgtable_t)page);
+	ptdesc = container_of(head, struct ptdesc, pt_rcu_head);
+	pte_free(NULL /* mm not passed and not used */, (pgtable_t)ptdesc);
 }
 
 void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable)
 {
-	struct page *page;
+	struct ptdesc *ptdesc = page_ptdesc(pgtable);
 
-	page = pgtable;
-	call_rcu(&page->rcu_head, pte_free_now);
+	call_rcu(&ptdesc->pt_rcu_head, pte_free_now);
 }
 #endif /* pte_free_defer */
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
@@ -330,7 +329,7 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd,
  * kmapped if necessary (when CONFIG_HIGHPTE), and locked against concurrent
  * modification by software, with a pointer to that spinlock in ptlp (in some
  * configs mm->page_table_lock, in SPLIT_PTLOCK configs a spinlock in table's
- * struct page).  pte_unmap_unlock(pte, ptl) to unlock and unmap afterwards.
+ * struct ptdesc).  pte_unmap_unlock(pte, ptl) to unlock and unmap afterwards.
  *
  * But it is unsuccessful, returning NULL with *ptlp unchanged, if there is no
  * page table at *pmd: if, for example, the page table has just been removed,
Signature · 1 line
2.43.0
#2 lore raw

RFC PATCH 2/18 mm/pgtable: convert ptdesc.pmd_huge_pte to ptdesc pointer

In reply to [email protected]

From: Alex Shi <[email protected]>

folio/page.pmd_huge_pte is a pointer to pagetable descriptor: pgtable_t. In most arch, it is a typedef of 'struct page *'. But we have ptdesc now, tt's better to convert it to right one: struct ptdesc pointer.

Different from others, s390/sparc use typedef 'pte_t *' as pgtable_t, so they need different cost in their arch.

Thanks for lkp found the build issue for s390/sparc, it fixed now.

Cc Ryan Roberts <[email protected]>
Cc Andreas Larsson <[email protected]>
Cc David S. Miller <[email protected]>
Cc Sven Schnelle <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Vasily Gorbik <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Gerald Schaefer <[email protected]>
Cc Alexander Gordeev <[email protected]>
Cc Zi Yan <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc David Hildenbrand <[email protected]>
---
 arch/s390/mm/pgtable.c   |  6 +++---
 arch/sparc/mm/tlb.c      |  6 +++---
 include/linux/mm_types.h |  4 ++--
 mm/pgtable-generic.c     | 16 ++++++++--------
 4 files changed, 16 insertions(+), 16 deletions(-)
arch/s390/mm/pgtable.c +3 -3
index 2c944bafb030..201d350abd1e 100644
@@ -574,7 +574,7 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 		INIT_LIST_HEAD(lh);
 	else
 		list_add(lh, (struct list_head *) pmd_huge_pte(mm, pmdp));
-	pmd_huge_pte(mm, pmdp) = pgtable;
+	pmd_huge_pte(mm, pmdp) = (struct ptdesc *)pgtable;
 }
 
 pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
@@ -586,12 +586,12 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
 	assert_spin_locked(pmd_lockptr(mm, pmdp));
 
 	/* FIFO */
-	pgtable = pmd_huge_pte(mm, pmdp);
+	pgtable = (pte_t *)pmd_huge_pte(mm, pmdp);
 	lh = (struct list_head *) pgtable;
 	if (list_empty(lh))
 		pmd_huge_pte(mm, pmdp) = NULL;
 	else {
-		pmd_huge_pte(mm, pmdp) = (pgtable_t) lh->next;
+		pmd_huge_pte(mm, pmdp) = (struct ptdesc *) lh->next;
 		list_del(lh);
 	}
 	ptep = (pte_t *) pgtable;
arch/sparc/mm/tlb.c +3 -3
index 8648a50afe88..903825b4c997 100644
@@ -278,7 +278,7 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 		INIT_LIST_HEAD(lh);
 	else
 		list_add(lh, (struct list_head *) pmd_huge_pte(mm, pmdp));
-	pmd_huge_pte(mm, pmdp) = pgtable;
+	pmd_huge_pte(mm, pmdp) = (struct ptdesc *)pgtable;
 }
 
 pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
@@ -289,12 +289,12 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
 	assert_spin_locked(&mm->page_table_lock);
 
 	/* FIFO */
-	pgtable = pmd_huge_pte(mm, pmdp);
+	pgtable = (pte_t *)pmd_huge_pte(mm, pmdp);
 	lh = (struct list_head *) pgtable;
 	if (list_empty(lh))
 		pmd_huge_pte(mm, pmdp) = NULL;
 	else {
-		pmd_huge_pte(mm, pmdp) = (pgtable_t) lh->next;
+		pmd_huge_pte(mm, pmdp) = (struct ptdesc *) lh->next;
 		list_del(lh);
 	}
 	pte_val(pgtable[0]) = 0;
include/linux/mm_types.h +2 -2
index 485424979254..2e3eddf6edc9 100644
@@ -462,7 +462,7 @@ struct ptdesc {
 		struct list_head pt_list;
 		struct {
 			unsigned long _pt_pad_1;
-			pgtable_t pmd_huge_pte;
+			struct ptdesc *pmd_huge_pte;
 		};
 	};
 	unsigned long __page_mapping;
@@ -948,7 +948,7 @@ struct mm_struct {
 		struct mmu_notifier_subscriptions *notifier_subscriptions;
 #endif
 #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS
-		pgtable_t pmd_huge_pte; /* protected by page_table_lock */
+		struct ptdesc *pmd_huge_pte; /* protected by page_table_lock */
 #endif
 #ifdef CONFIG_NUMA_BALANCING
 		/*
mm/pgtable-generic.c +8 -8
index 2ce714f1dd15..f34a8d115f5b 100644
@@ -171,8 +171,8 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 	if (!pmd_huge_pte(mm, pmdp))
 		INIT_LIST_HEAD(&pgtable->lru);
 	else
-		list_add(&pgtable->lru, &pmd_huge_pte(mm, pmdp)->lru);
-	pmd_huge_pte(mm, pmdp) = pgtable;
+		list_add(&pgtable->lru, &pmd_huge_pte(mm, pmdp)->pt_list);
+	pmd_huge_pte(mm, pmdp) = page_ptdesc(pgtable);
 }
 #endif
 
@@ -180,17 +180,17 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 /* no "address" argument so destroys page coloring of some arch */
 pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
 {
-	pgtable_t pgtable;
+	struct ptdesc *ptdesc;
 
 	assert_spin_locked(pmd_lockptr(mm, pmdp));
 
 	/* FIFO */
-	pgtable = pmd_huge_pte(mm, pmdp);
-	pmd_huge_pte(mm, pmdp) = list_first_entry_or_null(&pgtable->lru,
-							  struct page, lru);
+	ptdesc = pmd_huge_pte(mm, pmdp);
+	pmd_huge_pte(mm, pmdp) = list_first_entry_or_null(&ptdesc->pt_list,
+							  struct ptdesc, pt_list);
 	if (pmd_huge_pte(mm, pmdp))
-		list_del(&pgtable->lru);
-	return pgtable;
+		list_del(&ptdesc->pt_list);
+	return ptdesc_page(ptdesc);
 }
 #endif
 
Signature · 1 line
2.43.0
#3 lore raw

RFC PATCH 3/18 fs/dax: use ptdesc in dax_pmd_load_hole

In reply to [email protected]

From: Alex Shi <[email protected]>

Since we have ptdesc struct now, better to use replace pgtable_t, aka 'struct page *'.
It's a prepare for return ptdesc pointer in pte_alloc_one series function.

Cc Christian Brauner <[email protected]>
Cc Alexander Viro <[email protected]>
Cc Jan Kara <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Dan Williams <[email protected]>
---
 fs/dax.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
fs/dax.c +7 -7
index becb4a6920c6..6f7cea248206 100644
@@ -1206,7 +1206,7 @@ static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault *vmf,
 	unsigned long pmd_addr = vmf->address & PMD_MASK;
 	struct vm_area_struct *vma = vmf->vma;
 	struct inode *inode = mapping->host;
-	pgtable_t pgtable = NULL;
+	struct ptdesc *ptdesc = NULL;
 	struct folio *zero_folio;
 	spinlock_t *ptl;
 	pmd_t pmd_entry;
@@ -1222,8 +1222,8 @@ static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault *vmf,
 				  DAX_PMD | DAX_ZERO_PAGE);
 
 	if (arch_needs_pgtable_deposit()) {
-		pgtable = pte_alloc_one(vma->vm_mm);
-		if (!pgtable)
+		ptdesc = page_ptdesc(pte_alloc_one(vma->vm_mm));
+		if (!ptdesc)
 			return VM_FAULT_OOM;
 	}
 
@@ -1233,8 +1233,8 @@ static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault *vmf,
 		goto fallback;
 	}
 
-	if (pgtable) {
-		pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable);
+	if (ptdesc) {
+		pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, ptdesc_page(ptdesc));
 		mm_inc_nr_ptes(vma->vm_mm);
 	}
 	pmd_entry = mk_pmd(&zero_folio->page, vmf->vma->vm_page_prot);
@@ -1245,8 +1245,8 @@ static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault *vmf,
 	return VM_FAULT_NOPAGE;
 
 fallback:
-	if (pgtable)
-		pte_free(vma->vm_mm, pgtable);
+	if (ptdesc)
+		pte_free(vma->vm_mm, ptdesc_page(ptdesc));
 	trace_dax_pmd_load_hole_fallback(inode, vmf, zero_folio, *entry);
 	return VM_FAULT_FALLBACK;
 }
Signature · 1 line
2.43.0
#4 lore raw

RFC PATCH 4/18 mm/thp: use ptdesc pointer in __do_huge_pmd_anonymous_page

In reply to [email protected]

From: Alex Shi <[email protected]>

Since we have ptdesc struct now, better to use replace pgtable_t, aka 'struct page *'. It's alaos a preparation for return ptdesc pointer in pte_alloc_one series function.

---
 mm/huge_memory.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
mm/huge_memory.c +7 -7
index 0167dc27e365..0ee104093121 100644
@@ -943,7 +943,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
 {
 	struct vm_area_struct *vma = vmf->vma;
 	struct folio *folio = page_folio(page);
-	pgtable_t pgtable;
+	struct ptdesc *ptdesc;
 	unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
 	vm_fault_t ret = 0;
 
@@ -959,8 +959,8 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
 	}
 	folio_throttle_swaprate(folio, gfp);
 
-	pgtable = pte_alloc_one(vma->vm_mm);
-	if (unlikely(!pgtable)) {
+	ptdesc = page_ptdesc(pte_alloc_one(vma->vm_mm));
+	if (unlikely(!ptdesc)) {
 		ret = VM_FAULT_OOM;
 		goto release;
 	}
@@ -987,7 +987,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
 		if (userfaultfd_missing(vma)) {
 			spin_unlock(vmf->ptl);
 			folio_put(folio);
-			pte_free(vma->vm_mm, pgtable);
+			pte_free(vma->vm_mm, ptdesc_page(ptdesc));
 			ret = handle_userfault(vmf, VM_UFFD_MISSING);
 			VM_BUG_ON(ret & VM_FAULT_FALLBACK);
 			return ret;
@@ -997,7 +997,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
 		entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
 		folio_add_new_anon_rmap(folio, vma, haddr, RMAP_EXCLUSIVE);
 		folio_add_lru_vma(folio, vma);
-		pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable);
+		pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, ptdesc_page(ptdesc));
 		set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
 		update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
 		add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
@@ -1012,8 +1012,8 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
 unlock_release:
 	spin_unlock(vmf->ptl);
 release:
-	if (pgtable)
-		pte_free(vma->vm_mm, pgtable);
+	if (ptdesc)
+		pte_free(vma->vm_mm, ptdesc_page(ptdesc));
 	folio_put(folio);
 	return ret;
 
Signature · 1 line
2.43.0
#5 lore raw

RFC PATCH 5/18 mm/thp: use ptdesc in do_huge_pmd_anonymous_page

In reply to [email protected]

From: Alex Shi <[email protected]>

ince we have ptdesc struct now, better to use replace pgtable_t, aka 'struct page *'. It's alaos a preparation for return ptdesc pointer in pte_alloc_one series function.

---
 mm/huge_memory.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
mm/huge_memory.c +8 -8
index 0ee104093121..d86108d81a99 100644
@@ -1087,16 +1087,16 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
 	if (!(vmf->flags & FAULT_FLAG_WRITE) &&
 			!mm_forbids_zeropage(vma->vm_mm) &&
 			transparent_hugepage_use_zero_page()) {
-		pgtable_t pgtable;
+		struct ptdesc *ptdesc;
 		struct folio *zero_folio;
 		vm_fault_t ret;
 
-		pgtable = pte_alloc_one(vma->vm_mm);
-		if (unlikely(!pgtable))
+		ptdesc = page_ptdesc(pte_alloc_one(vma->vm_mm));
+		if (unlikely(!ptdesc))
 			return VM_FAULT_OOM;
 		zero_folio = mm_get_huge_zero_folio(vma->vm_mm);
 		if (unlikely(!zero_folio)) {
-			pte_free(vma->vm_mm, pgtable);
+			pte_free(vma->vm_mm, ptdesc_page(ptdesc));
 			count_vm_event(THP_FAULT_FALLBACK);
 			return VM_FAULT_FALLBACK;
 		}
@@ -1106,21 +1106,21 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
 			ret = check_stable_address_space(vma->vm_mm);
 			if (ret) {
 				spin_unlock(vmf->ptl);
-				pte_free(vma->vm_mm, pgtable);
+				pte_free(vma->vm_mm, ptdesc_page(ptdesc));
 			} else if (userfaultfd_missing(vma)) {
 				spin_unlock(vmf->ptl);
-				pte_free(vma->vm_mm, pgtable);
+				pte_free(vma->vm_mm, ptdesc_page(ptdesc));
 				ret = handle_userfault(vmf, VM_UFFD_MISSING);
 				VM_BUG_ON(ret & VM_FAULT_FALLBACK);
 			} else {
-				set_huge_zero_folio(pgtable, vma->vm_mm, vma,
+				set_huge_zero_folio(ptdesc_page(ptdesc), vma->vm_mm, vma,
 						   haddr, vmf->pmd, zero_folio);
 				update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
 				spin_unlock(vmf->ptl);
 			}
 		} else {
 			spin_unlock(vmf->ptl);
-			pte_free(vma->vm_mm, pgtable);
+			pte_free(vma->vm_mm, ptdesc_page(ptdesc));
 		}
 		return ret;
 	}
Signature · 1 line
2.43.0
#6 lore raw

RFC PATCH 6/18 mm/thp: convert insert_pfn_pmd and its caller to use ptdesc

In reply to [email protected]

From: Alex Shi <[email protected]>

Since we have ptdesc struct now, better to use replace pgtable_t, aka 'struct page *'. It's alaos a preparation for return ptdesc pointer in pte_alloc_one series function.

---
 mm/huge_memory.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
mm/huge_memory.c +10 -10
index d86108d81a99..a331d4504d52 100644
@@ -1136,7 +1136,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
 
 static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,
 		pmd_t *pmd, pfn_t pfn, pgprot_t prot, bool write,
-		pgtable_t pgtable)
+		struct ptdesc *ptdesc)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	pmd_t entry;
@@ -1166,10 +1166,10 @@ static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,
 		entry = maybe_pmd_mkwrite(entry, vma);
 	}
 
-	if (pgtable) {
-		pgtable_trans_huge_deposit(mm, pmd, pgtable);
+	if (ptdesc) {
+		pgtable_trans_huge_deposit(mm, pmd, ptdesc_page(ptdesc));
 		mm_inc_nr_ptes(mm);
-		pgtable = NULL;
+		ptdesc = NULL;
 	}
 
 	set_pmd_at(mm, addr, pmd, entry);
@@ -1177,8 +1177,8 @@ static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,
 
 out_unlock:
 	spin_unlock(ptl);
-	if (pgtable)
-		pte_free(mm, pgtable);
+	if (ptdesc)
+		pte_free(mm, ptdesc_page(ptdesc));
 }
 
 /**
@@ -1196,7 +1196,7 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write)
 	unsigned long addr = vmf->address & PMD_MASK;
 	struct vm_area_struct *vma = vmf->vma;
 	pgprot_t pgprot = vma->vm_page_prot;
-	pgtable_t pgtable = NULL;
+	struct ptdesc *ptdesc = NULL;
 
 	/*
 	 * If we had pmd_special, we could avoid all these restrictions,
@@ -1213,14 +1213,14 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write)
 		return VM_FAULT_SIGBUS;
 
 	if (arch_needs_pgtable_deposit()) {
-		pgtable = pte_alloc_one(vma->vm_mm);
-		if (!pgtable)
+		ptdesc = page_ptdesc(pte_alloc_one(vma->vm_mm));
+		if (!ptdesc)
 			return VM_FAULT_OOM;
 	}
 
 	track_pfn_insert(vma, &pgprot, pfn);
 
-	insert_pfn_pmd(vma, addr, vmf->pmd, pfn, pgprot, write, pgtable);
+	insert_pfn_pmd(vma, addr, vmf->pmd, pfn, pgprot, write, ptdesc);
 	return VM_FAULT_NOPAGE;
 }
 EXPORT_SYMBOL_GPL(vmf_insert_pfn_pmd);
Signature · 1 line
2.43.0
#7 lore raw

RFC PATCH 7/18 mm/thp: use ptdesc in copy_huge_pmd

In reply to [email protected]

From: Alex Shi <[email protected]>

Since we have ptdesc struct now, better to use replace pgtable_t, aka 'struct page *'. It's alaos a preparation for return ptdesc pointer in pte_alloc_one series function.

---
 mm/huge_memory.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
mm/huge_memory.c +7 -7
index a331d4504d52..236e1582d97e 100644
@@ -1369,15 +1369,15 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	struct page *src_page;
 	struct folio *src_folio;
 	pmd_t pmd;
-	pgtable_t pgtable = NULL;
+	struct ptdesc *ptdesc = NULL;
 	int ret = -ENOMEM;
 
 	/* Skip if can be re-fill on fault */
 	if (!vma_is_anonymous(dst_vma))
 		return 0;
 
-	pgtable = pte_alloc_one(dst_mm);
-	if (unlikely(!pgtable))
+	ptdesc = page_ptdesc(pte_alloc_one(dst_mm));
+	if (unlikely(!ptdesc))
 		goto out;
 
 	dst_ptl = pmd_lock(dst_mm, dst_pmd);
@@ -1404,7 +1404,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		}
 		add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
 		mm_inc_nr_ptes(dst_mm);
-		pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
+		pgtable_trans_huge_deposit(dst_mm, dst_pmd, ptdesc_page(ptdesc));
 		if (!userfaultfd_wp(dst_vma))
 			pmd = pmd_swp_clear_uffd_wp(pmd);
 		set_pmd_at(dst_mm, addr, dst_pmd, pmd);
@@ -1414,7 +1414,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 #endif
 
 	if (unlikely(!pmd_trans_huge(pmd))) {
-		pte_free(dst_mm, pgtable);
+		pte_free(dst_mm, ptdesc_page(ptdesc));
 		goto out_unlock;
 	}
 	/*
@@ -1440,7 +1440,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	if (unlikely(folio_try_dup_anon_rmap_pmd(src_folio, src_page, src_vma))) {
 		/* Page maybe pinned: split and retry the fault on PTEs. */
 		folio_put(src_folio);
-		pte_free(dst_mm, pgtable);
+		pte_free(dst_mm, ptdesc_page(ptdesc));
 		spin_unlock(src_ptl);
 		spin_unlock(dst_ptl);
 		__split_huge_pmd(src_vma, src_pmd, addr, false, NULL);
@@ -1449,7 +1449,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
 out_zero_page:
 	mm_inc_nr_ptes(dst_mm);
-	pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
+	pgtable_trans_huge_deposit(dst_mm, dst_pmd, ptdesc_page(ptdesc));
 	pmdp_set_wrprotect(src_mm, addr, src_pmd);
 	if (!userfaultfd_wp(dst_vma))
 		pmd = pmd_clear_uffd_wp(pmd);
Signature · 1 line
2.43.0
#8 lore raw

RFC PATCH 8/18 mm/memory: use ptdesc in __pte_alloc

In reply to [email protected]

From: Alex Shi <[email protected]>

Replace pgtable_t by ptdesc in function __pte_alloc. We will remove pgtable_t from all place.

---
 mm/memory.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
mm/memory.c +5 -5
index b9f5cc0db3eb..5b01d94a0b5f 100644
@@ -445,13 +445,13 @@ void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte)
 
 int __pte_alloc(struct mm_struct *mm, pmd_t *pmd)
 {
-	pgtable_t new = pte_alloc_one(mm);
-	if (!new)
+	struct ptdesc *ptdesc = page_ptdesc(pte_alloc_one(mm));
+	if (!ptdesc)
 		return -ENOMEM;
 
-	pmd_install(mm, pmd, &new);
-	if (new)
-		pte_free(mm, new);
+	pmd_install(mm, pmd, (pgtable_t *)&ptdesc);
+	if (ptdesc)
+		pte_free(mm, ptdesc_page(ptdesc));
 	return 0;
 }
 
Signature · 1 line
2.43.0
#9 lore raw

RFC PATCH 9/18 mm/pgtable: fully use ptdesc in pte_alloc_one series functions

In reply to [email protected]

From: Alex Shi <[email protected]>

Replace pgtable_t and struct page by ptdesc in pte_alloc_one series functions.

Cc Dawei Li <[email protected]>
Cc Vishal Moola <[email protected]>
Cc Arnd Bergmann <[email protected]>
Cc Christian Brauner <[email protected]>
Cc Alexander Viro <[email protected]>
Cc Jan Kara <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Dan Williams <[email protected]>
Cc Max Filippov <[email protected]>
Cc Chris Zankel <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc H. Peter Anvin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc David S. Miller <[email protected]>
Cc Naveen N. Rao <[email protected]>
Cc Christophe Leroy <[email protected]>
Cc Nicholas Piggin <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Russell King <[email protected]>
Cc Breno Leitao <[email protected]>
Cc Josh Poimboeuf <[email protected]>
Cc Bjorn Helgaas <[email protected]>
Cc Sam Ravnborg <[email protected]>
Cc Peter Xu <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Hugh Dickins <[email protected]>
---
 arch/arm/include/asm/pgalloc.h      |  9 ++++-----
 arch/powerpc/include/asm/pgalloc.h  |  4 ++--
 arch/s390/include/asm/pgalloc.h     |  2 +-
 arch/sparc/include/asm/pgalloc_32.h |  2 +-
 arch/sparc/include/asm/pgalloc_64.h |  2 +-
 arch/sparc/mm/init_64.c             |  2 +-
 arch/sparc/mm/srmmu.c               |  4 ++--
 arch/x86/include/asm/pgalloc.h      |  2 +-
 arch/x86/mm/pgtable.c               |  2 +-
 arch/xtensa/include/asm/pgalloc.h   | 12 ++++++------
 fs/dax.c                            |  2 +-
 include/asm-generic/pgalloc.h       |  6 +++---
 mm/huge_memory.c                    |  8 ++++----
 mm/memory.c                         |  8 ++++----
 14 files changed, 32 insertions(+), 33 deletions(-)
arch/arm/include/asm/pgalloc.h +4 -5
index a17f01235c29..e8501a6c3336 100644
@@ -91,16 +91,15 @@ pte_alloc_one_kernel(struct mm_struct *mm)
 #define PGTABLE_HIGHMEM 0
 #endif
 
-static inline pgtable_t
-pte_alloc_one(struct mm_struct *mm)
+static inline struct ptdesc *pte_alloc_one(struct mm_struct *mm)
 {
-	struct page *pte;
+	struct ptdesc *pte;
 
 	pte = __pte_alloc_one(mm, GFP_PGTABLE_USER | PGTABLE_HIGHMEM);
 	if (!pte)
 		return NULL;
-	if (!PageHighMem(pte))
-		clean_pte_table(page_address(pte));
+	if (!PageHighMem(ptdesc_page(pte)))
+		clean_pte_table(ptdesc_address(pte));
 	return pte;
 }
 
arch/powerpc/include/asm/pgalloc.h +2 -2
index 3a971e2a8c73..37512f344b37 100644
@@ -27,9 +27,9 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
 	return (pte_t *)pte_fragment_alloc(mm, 1);
 }
 
-static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
+static inline struct ptdesc *pte_alloc_one(struct mm_struct *mm)
 {
-	return (pgtable_t)pte_fragment_alloc(mm, 0);
+	return (struct ptdesc *)pte_fragment_alloc(mm, 0);
 }
 
 void pte_frag_destroy(void *pte_frag);
arch/s390/include/asm/pgalloc.h +1 -1
index 7b84ef6dc4b6..771494526f6e 100644
@@ -137,7 +137,7 @@ static inline void pmd_populate(struct mm_struct *mm,
  * page table entry allocation/free routines.
  */
 #define pte_alloc_one_kernel(mm) ((pte_t *)page_table_alloc(mm))
-#define pte_alloc_one(mm) ((pte_t *)page_table_alloc(mm))
+#define pte_alloc_one(mm) ((struct ptdesc *)page_table_alloc(mm))
 
 #define pte_free_kernel(mm, pte) page_table_free(mm, (unsigned long *) pte)
 #define pte_free(mm, pte) page_table_free(mm, (unsigned long *) pte)
arch/sparc/include/asm/pgalloc_32.h +1 -1
index 4f73e87b22a3..bc3ef54d9564 100644
@@ -55,7 +55,7 @@ static inline void free_pmd_fast(pmd_t * pmd)
 void pmd_set(pmd_t *pmdp, pte_t *ptep);
 #define pmd_populate_kernel		pmd_populate
 
-pgtable_t pte_alloc_one(struct mm_struct *mm);
+struct ptdesc *pte_alloc_one(struct mm_struct *mm);
 
 static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
 {
arch/sparc/include/asm/pgalloc_64.h +1 -1
index caa7632be4c2..285aa7958912 100644
@@ -61,7 +61,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
 }
 
 pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
-pgtable_t pte_alloc_one(struct mm_struct *mm);
+struct ptdesc *pte_alloc_one(struct mm_struct *mm);
 void pte_free_kernel(struct mm_struct *mm, pte_t *pte);
 void pte_free(struct mm_struct *mm, pgtable_t ptepage);
 
arch/sparc/mm/init_64.c +1 -1
index 53d7cb5bbffe..e1b33f996469 100644
@@ -2900,7 +2900,7 @@ pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
 	return pte;
 }
 
-pgtable_t pte_alloc_one(struct mm_struct *mm)
+struct ptdesc *pte_alloc_one(struct mm_struct *mm)
 {
 	struct ptdesc *ptdesc = pagetable_alloc(GFP_KERNEL | __GFP_ZERO, 0);
 
arch/sparc/mm/srmmu.c +2 -2
index 9df51a62333d..60bb8628bb1f 100644
@@ -346,7 +346,7 @@ pgd_t *get_pgd_fast(void)
  * Alignments up to the page size are the same for physical and virtual
  * addresses of the nocache area.
  */
-pgtable_t pte_alloc_one(struct mm_struct *mm)
+struct ptdesc *pte_alloc_one(struct mm_struct *mm)
 {
 	pte_t *ptep;
 	struct page *page;
@@ -362,7 +362,7 @@ pgtable_t pte_alloc_one(struct mm_struct *mm)
 	}
 	spin_unlock(&mm->page_table_lock);
 
-	return ptep;
+	return (struct ptdesc *)ptep;
 }
 
 void pte_free(struct mm_struct *mm, pgtable_t ptep)
arch/x86/include/asm/pgalloc.h +1 -1
index dcd836b59beb..497c757b5b98 100644
@@ -51,7 +51,7 @@ extern gfp_t __userpte_alloc_gfp;
 extern pgd_t *pgd_alloc(struct mm_struct *);
 extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
 
-extern pgtable_t pte_alloc_one(struct mm_struct *);
+extern struct ptdesc *pte_alloc_one(struct mm_struct *);
 
 extern void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte);
 
arch/x86/mm/pgtable.c +1 -1
index 93e54ba91fbf..c27d15cd01b9 100644
@@ -28,7 +28,7 @@ void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table)
 
 gfp_t __userpte_alloc_gfp = GFP_PGTABLE_USER | PGTABLE_HIGHMEM;
 
-pgtable_t pte_alloc_one(struct mm_struct *mm)
+struct ptdesc *pte_alloc_one(struct mm_struct *mm)
 {
 	return __pte_alloc_one(mm, __userpte_alloc_gfp);
 }
arch/xtensa/include/asm/pgalloc.h +6 -6
index 7fc0f9126dd3..a9206c02956e 100644
@@ -51,15 +51,15 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)
 	return ptep;
 }
 
-static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
+static inline struct ptdesc *pte_alloc_one(struct mm_struct *mm)
 {
-	struct page *page;
+	struct ptdesc *ptdesc;
 
-	page = __pte_alloc_one(mm, GFP_PGTABLE_USER);
-	if (!page)
+	ptdesc = __pte_alloc_one(mm, GFP_PGTABLE_USER);
+	if (!ptdesc)
 		return NULL;
-	ptes_clear(page_address(page));
-	return page;
+	ptes_clear(ptdesc_address(ptdesc));
+	return ptdesc;
 }
 
 #endif /* CONFIG_MMU */
fs/dax.c +1 -1
index 6f7cea248206..51cbc08b22e7 100644
@@ -1222,7 +1222,7 @@ static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault *vmf,
 				  DAX_PMD | DAX_ZERO_PAGE);
 
 	if (arch_needs_pgtable_deposit()) {
-		ptdesc = page_ptdesc(pte_alloc_one(vma->vm_mm));
+		ptdesc = pte_alloc_one(vma->vm_mm);
 		if (!ptdesc)
 			return VM_FAULT_OOM;
 	}
include/asm-generic/pgalloc.h +3 -3
index 7c48f5fbf8aa..1a4070f8d5dd 100644
@@ -63,7 +63,7 @@ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
  *
  * Return: `struct page` referencing the ptdesc or %NULL on error
  */
-static inline pgtable_t __pte_alloc_one_noprof(struct mm_struct *mm, gfp_t gfp)
+static inline struct ptdesc *__pte_alloc_one_noprof(struct mm_struct *mm, gfp_t gfp)
 {
 	struct ptdesc *ptdesc;
 
@@ -75,7 +75,7 @@ static inline pgtable_t __pte_alloc_one_noprof(struct mm_struct *mm, gfp_t gfp)
 		return NULL;
 	}
 
-	return ptdesc_page(ptdesc);
+	return ptdesc;
 }
 #define __pte_alloc_one(...)	alloc_hooks(__pte_alloc_one_noprof(__VA_ARGS__))
 
@@ -88,7 +88,7 @@ static inline pgtable_t __pte_alloc_one_noprof(struct mm_struct *mm, gfp_t gfp)
  *
  * Return: `struct page` referencing the ptdesc or %NULL on error
  */
-static inline pgtable_t pte_alloc_one_noprof(struct mm_struct *mm)
+static inline struct ptdesc *pte_alloc_one_noprof(struct mm_struct *mm)
 {
 	return __pte_alloc_one_noprof(mm, GFP_PGTABLE_USER);
 }
mm/huge_memory.c +4 -4
index 236e1582d97e..6274eb7559ac 100644
@@ -959,7 +959,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
 	}
 	folio_throttle_swaprate(folio, gfp);
 
-	ptdesc = page_ptdesc(pte_alloc_one(vma->vm_mm));
+	ptdesc = pte_alloc_one(vma->vm_mm);
 	if (unlikely(!ptdesc)) {
 		ret = VM_FAULT_OOM;
 		goto release;
@@ -1091,7 +1091,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
 		struct folio *zero_folio;
 		vm_fault_t ret;
 
-		ptdesc = page_ptdesc(pte_alloc_one(vma->vm_mm));
+		ptdesc = pte_alloc_one(vma->vm_mm);
 		if (unlikely(!ptdesc))
 			return VM_FAULT_OOM;
 		zero_folio = mm_get_huge_zero_folio(vma->vm_mm);
@@ -1213,7 +1213,7 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write)
 		return VM_FAULT_SIGBUS;
 
 	if (arch_needs_pgtable_deposit()) {
-		ptdesc = page_ptdesc(pte_alloc_one(vma->vm_mm));
+		ptdesc = pte_alloc_one(vma->vm_mm);
 		if (!ptdesc)
 			return VM_FAULT_OOM;
 	}
@@ -1376,7 +1376,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	if (!vma_is_anonymous(dst_vma))
 		return 0;
 
-	ptdesc = page_ptdesc(pte_alloc_one(dst_mm));
+	ptdesc = pte_alloc_one(dst_mm);
 	if (unlikely(!ptdesc))
 		goto out;
 
mm/memory.c +4 -4
index 5b01d94a0b5f..37529e0a9ce2 100644
@@ -445,7 +445,7 @@ void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte)
 
 int __pte_alloc(struct mm_struct *mm, pmd_t *pmd)
 {
-	struct ptdesc *ptdesc = page_ptdesc(pte_alloc_one(mm));
+	struct ptdesc *ptdesc = pte_alloc_one(mm);
 	if (!ptdesc)
 		return -ENOMEM;
 
@@ -4647,7 +4647,7 @@ static vm_fault_t __do_fault(struct vm_fault *vmf)
 	 *				# flush A, B to clear the writeback
 	 */
 	if (pmd_none(*vmf->pmd) && !vmf->prealloc_pte) {
-		vmf->prealloc_pte = pte_alloc_one(vma->vm_mm);
+		vmf->prealloc_pte = ptdesc_page(pte_alloc_one(vma->vm_mm));
 		if (!vmf->prealloc_pte)
 			return VM_FAULT_OOM;
 	}
@@ -4725,7 +4725,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page)
 	 * related to pte entry. Use the preallocated table for that.
 	 */
 	if (arch_needs_pgtable_deposit() && !vmf->prealloc_pte) {
-		vmf->prealloc_pte = pte_alloc_one(vma->vm_mm);
+		vmf->prealloc_pte = ptdesc_page(pte_alloc_one(vma->vm_mm));
 		if (!vmf->prealloc_pte)
 			return VM_FAULT_OOM;
 	}
@@ -5010,7 +5010,7 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf)
 		      pte_off + vma_pages(vmf->vma) - vma_off) - 1;
 
 	if (pmd_none(*vmf->pmd)) {
-		vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm);
+		vmf->prealloc_pte = ptdesc_page(pte_alloc_one(vmf->vma->vm_mm));
 		if (!vmf->prealloc_pte)
 			return VM_FAULT_OOM;
 	}
Signature · 1 line
2.43.0
#10 lore raw

RFC PATCH 10/18 mm/pgtable: pass ptdesc to pte_free()

In reply to [email protected]

From: Alex Shi <[email protected]>

Now we could remove couple of page<->ptdesc converters now.

Cc Bjorn Helgaas <[email protected]>
Cc Vishal Moola <[email protected]>
Cc Arnd Bergmann <[email protected]>
Cc Christian Brauner <[email protected]>
Cc Alexander Viro <[email protected]>
Cc Jan Kara <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Dan Williams <[email protected]>
Cc David S. Miller <[email protected]>
Cc Naveen N. Rao <[email protected]>
Cc Christophe Leroy <[email protected]>
Cc Nicholas Piggin <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Geert Uytterhoeven <[email protected]>
Cc Russell King <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Dawei Li <[email protected]>
Cc Hugh Dickins <[email protected]>
---
 arch/arm/mm/pgd.c                            |  2 +-
 arch/m68k/include/asm/motorola_pgalloc.h     |  4 ++--
 arch/powerpc/include/asm/book3s/64/pgalloc.h |  2 +-
 arch/powerpc/include/asm/pgalloc.h           |  2 +-
 arch/sparc/include/asm/pgalloc_32.h          |  2 +-
 arch/sparc/mm/srmmu.c                        |  2 +-
 fs/dax.c                                     |  2 +-
 include/asm-generic/pgalloc.h                |  4 +---
 mm/debug_vm_pgtable.c                        |  2 +-
 mm/huge_memory.c                             | 20 ++++++++++----------
 mm/memory.c                                  |  4 ++--
 mm/pgtable-generic.c                         |  2 +-
 12 files changed, 23 insertions(+), 25 deletions(-)
arch/arm/mm/pgd.c +1 -1
index f8e9bc58a84f..c384b734d752 100644
@@ -168,7 +168,7 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd_base)
 
 	pte = pmd_pgtable(*pmd);
 	pmd_clear(pmd);
-	pte_free(mm, pte);
+	pte_free(mm, page_ptdesc(pte));
 	mm_dec_nr_ptes(mm);
 no_pmd:
 	pud_clear(pud);
arch/m68k/include/asm/motorola_pgalloc.h +2 -2
index 74a817d9387f..f6bb375971dc 100644
@@ -39,9 +39,9 @@ static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
 	return get_pointer_table(TABLE_PTE);
 }
 
-static inline void pte_free(struct mm_struct *mm, pgtable_t pgtable)
+static inline void pte_free(struct mm_struct *mm, struct ptdesc *ptdesc)
 {
-	free_pointer_table(pgtable, TABLE_PTE);
+	free_pointer_table(ptdesc_page(ptdesc), TABLE_PTE);
 }
 
 static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pgtable,
arch/powerpc/include/asm/book3s/64/pgalloc.h +1 -1
index dd2cff53a111..eb7d2ca59f62 100644
@@ -162,7 +162,7 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd,
 }
 
 static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,
-				pgtable_t pte_page)
+				struct ptdesc *pte_page)
 {
 	*pmd = __pmd(__pgtable_ptr_val(pte_page) | PMD_VAL_BITS);
 }
arch/powerpc/include/asm/pgalloc.h +1 -1
index 37512f344b37..12520521163e 100644
@@ -40,7 +40,7 @@ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
 	pte_fragment_free((unsigned long *)pte, 1);
 }
 
-static inline void pte_free(struct mm_struct *mm, pgtable_t ptepage)
+static inline void pte_free(struct mm_struct *mm, struct ptdesc *ptepage)
 {
 	pte_fragment_free((unsigned long *)ptepage, 0);
 }
arch/sparc/include/asm/pgalloc_32.h +1 -1
index bc3ef54d9564..addaade56f21 100644
@@ -71,7 +71,7 @@ static inline void free_pte_fast(pte_t *pte)
 
 #define pte_free_kernel(mm, pte)	free_pte_fast(pte)
 
-void pte_free(struct mm_struct * mm, pgtable_t pte);
+void pte_free(struct mm_struct *mm, struct ptdesc *pte);
 #define __pte_free_tlb(tlb, pte, addr)	pte_free((tlb)->mm, pte)
 
 #endif /* _SPARC_PGALLOC_H */
arch/sparc/mm/srmmu.c +1 -1
index 60bb8628bb1f..05be7d86eda3 100644
@@ -365,7 +365,7 @@ struct ptdesc *pte_alloc_one(struct mm_struct *mm)
 	return (struct ptdesc *)ptep;
 }
 
-void pte_free(struct mm_struct *mm, pgtable_t ptep)
+void pte_free(struct mm_struct *mm, struct ptdesc *ptep)
 {
 	struct page *page;
 
fs/dax.c +1 -1
index 51cbc08b22e7..61b9bd5200da 100644
@@ -1246,7 +1246,7 @@ static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault *vmf,
 
 fallback:
 	if (ptdesc)
-		pte_free(vma->vm_mm, ptdesc_page(ptdesc));
+		pte_free(vma->vm_mm, ptdesc);
 	trace_dax_pmd_load_hole_fallback(inode, vmf, zero_folio, *entry);
 	return VM_FAULT_FALLBACK;
 }
include/asm-generic/pgalloc.h +1 -3
index 1a4070f8d5dd..5f249ec9d289 100644
@@ -105,10 +105,8 @@ static inline struct ptdesc *pte_alloc_one_noprof(struct mm_struct *mm)
  * @mm: the mm_struct of the current context
  * @pte_page: the `struct page` referencing the ptdesc
  */
-static inline void pte_free(struct mm_struct *mm, struct page *pte_page)
+static inline void pte_free(struct mm_struct *mm, struct ptdesc *ptdesc)
 {
-	struct ptdesc *ptdesc = page_ptdesc(pte_page);
-
 	pagetable_pte_dtor(ptdesc);
 	pagetable_free(ptdesc);
 }
mm/debug_vm_pgtable.c +1 -1
index e4969fb54da3..f256bc816744 100644
@@ -1049,7 +1049,7 @@ static void __init destroy_args(struct pgtable_debug_args *args)
 
 	/* Free page table entries */
 	if (args->start_ptep) {
-		pte_free(args->mm, args->start_ptep);
+		pte_free(args->mm, page_ptdesc(args->start_ptep));
 		mm_dec_nr_ptes(args->mm);
 	}
 
mm/huge_memory.c +10 -10
index 6274eb7559ac..dc323453fa02 100644
@@ -987,7 +987,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
 		if (userfaultfd_missing(vma)) {
 			spin_unlock(vmf->ptl);
 			folio_put(folio);
-			pte_free(vma->vm_mm, ptdesc_page(ptdesc));
+			pte_free(vma->vm_mm, ptdesc);
 			ret = handle_userfault(vmf, VM_UFFD_MISSING);
 			VM_BUG_ON(ret & VM_FAULT_FALLBACK);
 			return ret;
@@ -1013,7 +1013,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
 	spin_unlock(vmf->ptl);
 release:
 	if (ptdesc)
-		pte_free(vma->vm_mm, ptdesc_page(ptdesc));
+		pte_free(vma->vm_mm, ptdesc);
 	folio_put(folio);
 	return ret;
 
@@ -1096,7 +1096,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
 			return VM_FAULT_OOM;
 		zero_folio = mm_get_huge_zero_folio(vma->vm_mm);
 		if (unlikely(!zero_folio)) {
-			pte_free(vma->vm_mm, ptdesc_page(ptdesc));
+			pte_free(vma->vm_mm, ptdesc);
 			count_vm_event(THP_FAULT_FALLBACK);
 			return VM_FAULT_FALLBACK;
 		}
@@ -1106,10 +1106,10 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
 			ret = check_stable_address_space(vma->vm_mm);
 			if (ret) {
 				spin_unlock(vmf->ptl);
-				pte_free(vma->vm_mm, ptdesc_page(ptdesc));
+				pte_free(vma->vm_mm, ptdesc);
 			} else if (userfaultfd_missing(vma)) {
 				spin_unlock(vmf->ptl);
-				pte_free(vma->vm_mm, ptdesc_page(ptdesc));
+				pte_free(vma->vm_mm, ptdesc);
 				ret = handle_userfault(vmf, VM_UFFD_MISSING);
 				VM_BUG_ON(ret & VM_FAULT_FALLBACK);
 			} else {
@@ -1120,7 +1120,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
 			}
 		} else {
 			spin_unlock(vmf->ptl);
-			pte_free(vma->vm_mm, ptdesc_page(ptdesc));
+			pte_free(vma->vm_mm, ptdesc);
 		}
 		return ret;
 	}
@@ -1178,7 +1178,7 @@ static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,
 out_unlock:
 	spin_unlock(ptl);
 	if (ptdesc)
-		pte_free(mm, ptdesc_page(ptdesc));
+		pte_free(mm, ptdesc);
 }
 
 /**
@@ -1414,7 +1414,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 #endif
 
 	if (unlikely(!pmd_trans_huge(pmd))) {
-		pte_free(dst_mm, ptdesc_page(ptdesc));
+		pte_free(dst_mm, ptdesc);
 		goto out_unlock;
 	}
 	/*
@@ -1440,7 +1440,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	if (unlikely(folio_try_dup_anon_rmap_pmd(src_folio, src_page, src_vma))) {
 		/* Page maybe pinned: split and retry the fault on PTEs. */
 		folio_put(src_folio);
-		pte_free(dst_mm, ptdesc_page(ptdesc));
+		pte_free(dst_mm, ptdesc);
 		spin_unlock(src_ptl);
 		spin_unlock(dst_ptl);
 		__split_huge_pmd(src_vma, src_pmd, addr, false, NULL);
@@ -1830,7 +1830,7 @@ static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)
 	pgtable_t pgtable;
 
 	pgtable = pgtable_trans_huge_withdraw(mm, pmd);
-	pte_free(mm, pgtable);
+	pte_free(mm, page_ptdesc(pgtable));
 	mm_dec_nr_ptes(mm);
 }
 
mm/memory.c +2 -2
index 37529e0a9ce2..3014168e7296 100644
@@ -451,7 +451,7 @@ int __pte_alloc(struct mm_struct *mm, pmd_t *pmd)
 
 	pmd_install(mm, pmd, (pgtable_t *)&ptdesc);
 	if (ptdesc)
-		pte_free(mm, ptdesc_page(ptdesc));
+		pte_free(mm, ptdesc);
 	return 0;
 }
 
@@ -5196,7 +5196,7 @@ static vm_fault_t do_fault(struct vm_fault *vmf)
 
 	/* preallocated pagetable is unused: free it */
 	if (vmf->prealloc_pte) {
-		pte_free(vm_mm, vmf->prealloc_pte);
+		pte_free(vm_mm, page_ptdesc(vmf->prealloc_pte));
 		vmf->prealloc_pte = NULL;
 	}
 	return ret;
mm/pgtable-generic.c +1 -1
index f34a8d115f5b..92245a32656b 100644
@@ -241,7 +241,7 @@ static void pte_free_now(struct rcu_head *head)
 	struct ptdesc *ptdesc;
 
 	ptdesc = container_of(head, struct ptdesc, pt_rcu_head);
-	pte_free(NULL /* mm not passed and not used */, (pgtable_t)ptdesc);
+	pte_free(NULL /* mm not passed and not used */, ptdesc);
 }
 
 void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable)
Signature · 1 line
2.43.0
#11 lore raw

RFC PATCH 11/18 mm/pgtable: introduce ptdesc_pfn and use ptdesc in free_pte_range()

In reply to [email protected]

From: Alex Shi <[email protected]>

Replace pgtable_t by ptdesc in free_pte_range and it's callee pte_free_tlb series functions. And save some converters now. We have to use type casting for pmd_pgtable() instead of page_ptdesc() helper since different arch has different type of pgtable_t.

btw, we can not simplify pmd_ptdesc() via replace pmd_pgtable_page by pmd_page, since some arch may have no pmd_page yet.

Cc Anup Patel <[email protected]>
Cc Samuel Holland <[email protected]>
Cc Jisheng Zhang <[email protected]>
Cc Alexandre Ghiti <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Palmer Dabbelt <[email protected]>
Cc Guo Ren <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc H. Peter Anvin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Naveen N. Rao <[email protected]>
Cc Christophe Leroy <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Stafford Horne <[email protected]>
Cc Stefan Kristiansson <[email protected]>
Cc Jonas Bonn <[email protected]>
Cc Geert Uytterhoeven <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Russell King <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Nick Piggin <[email protected]>
Cc Aneesh Kumar K.V <[email protected]>
Cc Will Deacon <[email protected]>
Cc Breno Leitao <[email protected]>
Cc Josh Poimboeuf <[email protected]>
Cc Vishal Moola <[email protected]>
Cc Mike Rapoport <[email protected]>
---
 arch/arm/include/asm/tlb.h                   |  4 +---
 arch/arm64/include/asm/tlb.h                 |  4 +---
 arch/csky/include/asm/pgalloc.h              |  4 ++--
 arch/hexagon/include/asm/pgalloc.h           |  4 ++--
 arch/loongarch/include/asm/pgalloc.h         |  4 ++--
 arch/m68k/include/asm/motorola_pgalloc.h     |  4 ++--
 arch/openrisc/include/asm/pgalloc.h          |  4 ++--
 arch/powerpc/include/asm/book3s/32/pgalloc.h |  2 +-
 arch/powerpc/include/asm/book3s/64/pgalloc.h |  2 +-
 arch/riscv/include/asm/pgalloc.h             |  8 +++-----
 arch/x86/include/asm/pgalloc.h               |  4 ++--
 arch/x86/mm/pgtable.c                        |  6 +++---
 include/linux/mm.h                           | 14 ++++++++++++++
 mm/memory.c                                  |  3 ++-
 14 files changed, 38 insertions(+), 29 deletions(-)
arch/arm/include/asm/tlb.h +1 -3
index f40d06ad5d2a..ed6aa4255518 100644
@@ -37,10 +37,8 @@ static inline void __tlb_remove_table(void *_table)
 #include <asm-generic/tlb.h>
 
 static inline void
-__pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr)
+__pte_free_tlb(struct mmu_gather *tlb, struct ptdesc *ptdesc, unsigned long addr)
 {
-	struct ptdesc *ptdesc = page_ptdesc(pte);
-
 	pagetable_pte_dtor(ptdesc);
 
 #ifndef CONFIG_ARM_LPAE
arch/arm64/include/asm/tlb.h +1 -3
index a947c6e784ed..cee7234af6e7 100644
@@ -77,11 +77,9 @@ static inline void tlb_flush(struct mmu_gather *tlb)
 			  last_level, tlb_level);
 }
 
-static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
+static inline void __pte_free_tlb(struct mmu_gather *tlb, struct ptdesc *ptdesc,
 				  unsigned long addr)
 {
-	struct ptdesc *ptdesc = page_ptdesc(pte);
-
 	pagetable_pte_dtor(ptdesc);
 	tlb_remove_ptdesc(tlb, ptdesc);
 }
arch/csky/include/asm/pgalloc.h +2 -2
index 9c84c9012e53..b24b4611436e 100644
@@ -63,8 +63,8 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
 
 #define __pte_free_tlb(tlb, pte, address)		\
 do {							\
-	pagetable_pte_dtor(page_ptdesc(pte));		\
-	tlb_remove_page_ptdesc(tlb, page_ptdesc(pte));	\
+	pagetable_pte_dtor(pte);			\
+	tlb_remove_page_ptdesc(tlb, pte);		\
 } while (0)
 
 extern void pagetable_init(void);
arch/hexagon/include/asm/pgalloc.h +2 -2
index 55988625e6fb..a3e082e54b74 100644
@@ -89,8 +89,8 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd,
 
 #define __pte_free_tlb(tlb, pte, addr)				\
 do {								\
-	pagetable_pte_dtor((page_ptdesc(pte)));			\
-	tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte)));	\
+	pagetable_pte_dtor((pte));				\
+	tlb_remove_page_ptdesc((tlb), (pte));			\
 } while (0)
 
 #endif
arch/loongarch/include/asm/pgalloc.h +2 -2
index 4e2d6b7ca2ee..c96d7160babc 100644
@@ -46,8 +46,8 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm);
 
 #define __pte_free_tlb(tlb, pte, address)			\
 do {								\
-	pagetable_pte_dtor(page_ptdesc(pte));			\
-	tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));	\
+	pagetable_pte_dtor(pte);				\
+	tlb_remove_page_ptdesc((tlb), pte);			\
 } while (0)
 
 #ifndef __PAGETABLE_PMD_FOLDED
arch/m68k/include/asm/motorola_pgalloc.h +2 -2
index f6bb375971dc..f9ee5ec4574d 100644
@@ -44,10 +44,10 @@ static inline void pte_free(struct mm_struct *mm, struct ptdesc *ptdesc)
 	free_pointer_table(ptdesc_page(ptdesc), TABLE_PTE);
 }
 
-static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pgtable,
+static inline void __pte_free_tlb(struct mmu_gather *tlb, struct ptdesc *ptdesc,
 				  unsigned long address)
 {
-	free_pointer_table(pgtable, TABLE_PTE);
+	free_pointer_table(ptdesc_page(ptdesc), TABLE_PTE);
 }
 
 
arch/openrisc/include/asm/pgalloc.h +2 -2
index c6a73772a546..2251d940c3d8 100644
@@ -68,8 +68,8 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
 
 #define __pte_free_tlb(tlb, pte, addr)				\
 do {								\
-	pagetable_pte_dtor(page_ptdesc(pte));			\
-	tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte)));	\
+	pagetable_pte_dtor(pte);			\
+	tlb_remove_page_ptdesc((tlb), (pte));	\
 } while (0)
 
 #endif
arch/powerpc/include/asm/book3s/32/pgalloc.h +1 -1
index dd4eb3063175..a435c84d1f9a 100644
@@ -64,7 +64,7 @@ static inline void __tlb_remove_table(void *_table)
 	pgtable_free(table, shift);
 }
 
-static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t table,
+static inline void __pte_free_tlb(struct mmu_gather *tlb, struct ptdesc *table,
 				  unsigned long address)
 {
 	pgtable_free_tlb(tlb, table, 0);
arch/powerpc/include/asm/book3s/64/pgalloc.h +1 -1
index eb7d2ca59f62..675eca34fe40 100644
@@ -167,7 +167,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,
 	*pmd = __pmd(__pgtable_ptr_val(pte_page) | PMD_VAL_BITS);
 }
 
-static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t table,
+static inline void __pte_free_tlb(struct mmu_gather *tlb, struct ptdesc *table,
 				  unsigned long address)
 {
 	pgtable_free_tlb(tlb, table, PTE_INDEX);
arch/riscv/include/asm/pgalloc.h +3 -5
index f52264304f77..63596efcd528 100644
@@ -183,13 +183,11 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
 
 #endif /* __PAGETABLE_PMD_FOLDED */
 
-static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
+static inline void __pte_free_tlb(struct mmu_gather *tlb, struct ptdesc *pte,
 				  unsigned long addr)
 {
-	struct ptdesc *ptdesc = page_ptdesc(pte);
-
-	pagetable_pte_dtor(ptdesc);
-	riscv_tlb_remove_ptdesc(tlb, ptdesc);
+	pagetable_pte_dtor(pte);
+	riscv_tlb_remove_ptdesc(tlb, pte);
 }
 #endif /* CONFIG_MMU */
 
arch/x86/include/asm/pgalloc.h +2 -2
index 497c757b5b98..06a9a5867a86 100644
@@ -53,9 +53,9 @@ extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
 
 extern struct ptdesc *pte_alloc_one(struct mm_struct *);
 
-extern void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte);
+extern void ___pte_free_tlb(struct mmu_gather *tlb, struct ptdesc *pte);
 
-static inline void __pte_free_tlb(struct mmu_gather *tlb, struct page *pte,
+static inline void __pte_free_tlb(struct mmu_gather *tlb, struct ptdesc *pte,
 				  unsigned long address)
 {
 	___pte_free_tlb(tlb, pte);
arch/x86/mm/pgtable.c +3 -3
index c27d15cd01b9..3cf9c0d25dbd 100644
@@ -50,10 +50,10 @@ static int __init setup_userpte(char *arg)
 }
 early_param("userpte", setup_userpte);
 
-void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte)
+void ___pte_free_tlb(struct mmu_gather *tlb, struct ptdesc *pte)
 {
-	pagetable_pte_dtor(page_ptdesc(pte));
-	paravirt_release_pte(page_to_pfn(pte));
+	pagetable_pte_dtor(pte);
+	paravirt_release_pte(ptdesc_pfn(pte));
 	paravirt_tlb_remove_table(tlb, pte);
 }
 
include/linux/mm.h +14 -0
index 381750f41767..7424f964dff3 100644
@@ -2859,6 +2859,20 @@ static inline bool pagetable_is_reserved(struct ptdesc *pt)
 	return folio_test_reserved(ptdesc_folio(pt));
 }
 
+/**
+ * ptdesc_pfn - Return the Page Frame Number of a ptdesc.
+ * @ptdesc: The ptdesc.
+ *
+ * A ptdesc may contain multiple pages.  The pages have consecutive
+ * Page Frame Numbers.
+ *
+ * Return: The Page Frame Number of the first page in the ptdesc.
+ */
+static inline unsigned long ptdesc_pfn(struct ptdesc *ptdesc)
+{
+	return page_to_pfn(ptdesc_page(ptdesc));
+}
+
 /**
  * pagetable_alloc - Allocate pagetables
  * @gfp:    GFP flags
mm/memory.c +2 -1
index 3014168e7296..27c2f63b7487 100644
@@ -189,7 +189,8 @@ void mm_trace_rss_stat(struct mm_struct *mm, int member)
 static void free_pte_range(struct mmu_gather *tlb, pmd_t *pmd,
 			   unsigned long addr)
 {
-	pgtable_t token = pmd_pgtable(*pmd);
+	struct ptdesc *token = (struct ptdesc *)pmd_pgtable(*pmd);
+
 	pmd_clear(pmd);
 	pte_free_tlb(tlb, token, addr);
 	mm_dec_nr_ptes(tlb->mm);
Signature · 1 line
2.43.0
#12 lore raw

RFC PATCH 12/18 mm/thp: pass ptdesc to set_huge_zero_folio function

From: Alex Shi <[email protected]>

Aim is still replace struct page to ptdesc.

---
 mm/huge_memory.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
mm/huge_memory.c +3 -3
index dc323453fa02..1c121ec85447 100644
@@ -1055,7 +1055,7 @@ gfp_t vma_thp_gfp_mask(struct vm_area_struct *vma)
 }
 
 /* Caller must hold page table lock. */
-static void set_huge_zero_folio(pgtable_t pgtable, struct mm_struct *mm,
+static void set_huge_zero_folio(struct ptdesc *ptdesc, struct mm_struct *mm,
 		struct vm_area_struct *vma, unsigned long haddr, pmd_t *pmd,
 		struct folio *zero_folio)
 {
@@ -1064,7 +1064,7 @@ static void set_huge_zero_folio(pgtable_t pgtable, struct mm_struct *mm,
 		return;
 	entry = mk_pmd(&zero_folio->page, vma->vm_page_prot);
 	entry = pmd_mkhuge(entry);
-	pgtable_trans_huge_deposit(mm, pmd, pgtable);
+	pgtable_trans_huge_deposit(mm, pmd, ptdesc_page(ptdesc));
 	set_pmd_at(mm, haddr, pmd, entry);
 	mm_inc_nr_ptes(mm);
 }
@@ -1113,7 +1113,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
 				ret = handle_userfault(vmf, VM_UFFD_MISSING);
 				VM_BUG_ON(ret & VM_FAULT_FALLBACK);
 			} else {
-				set_huge_zero_folio(ptdesc_page(ptdesc), vma->vm_mm, vma,
+				set_huge_zero_folio(ptdesc, vma->vm_mm, vma,
 						   haddr, vmf->pmd, zero_folio);
 				update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
 				spin_unlock(vmf->ptl);
Signature · 1 line
2.43.0
#13 lore raw

RFC PATCH 13/18 mm/pgtable: return ptdesc pointer in pgtable_trans_huge_withdraw

In reply to [email protected]

From: Alex Shi <[email protected]>

Way to replace pgtable_t aka struct page in most of archs.

Cc Barry Song <[email protected]>
Cc Lance Yang <[email protected]>
Cc Kinsey Ho <[email protected]>
Cc Aneesh Kumar K.V <[email protected]>
Cc Benjamin Gray <[email protected]>
Cc Andreas Larsson <[email protected]>
Cc David S. Miller <[email protected]>
Cc Sven Schnelle <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Vasily Gorbik <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Gerald Schaefer <[email protected]>
Cc Alexander Gordeev <[email protected]>
Cc Naveen N. Rao <[email protected]>
Cc Nicholas Piggin <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Aneesh Kumar K.V <[email protected]>
Cc Peter Xu <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Christophe Leroy <[email protected]>
Cc Michael Ellerman <[email protected]>
---
 arch/powerpc/include/asm/book3s/64/hash-4k.h  |  4 +--
 arch/powerpc/include/asm/book3s/64/hash-64k.h |  4 +--
 arch/powerpc/include/asm/book3s/64/pgtable.h  |  2 +-
 arch/powerpc/include/asm/book3s/64/radix.h    |  4 +--
 arch/powerpc/mm/book3s64/hash_pgtable.c       |  4 +--
 arch/powerpc/mm/book3s64/radix_pgtable.c      |  4 +--
 arch/s390/include/asm/pgtable.h               |  2 +-
 arch/s390/mm/pgtable.c                        |  4 +--
 arch/sparc/include/asm/pgtable_64.h           |  2 +-
 arch/sparc/mm/tlb.c                           |  4 +--
 include/linux/pgtable.h                       |  2 +-
 mm/huge_memory.c                              | 35 ++++++++++---------
 mm/pgtable-generic.c                          |  4 +--
 13 files changed, 38 insertions(+), 37 deletions(-)
arch/powerpc/include/asm/book3s/64/hash-4k.h +2 -2
index c654c376ef8b..3a99a0229c37 100644
@@ -133,8 +133,8 @@ extern unsigned long hash__pmd_hugepage_update(struct mm_struct *mm,
 extern pmd_t hash__pmdp_collapse_flush(struct vm_area_struct *vma,
 				   unsigned long address, pmd_t *pmdp);
 extern void hash__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
-					 pgtable_t pgtable);
-extern pgtable_t hash__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
+					 struct ptdesc *ptdesc);
+extern struct ptdesc *hash__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
 extern pmd_t hash__pmdp_huge_get_and_clear(struct mm_struct *mm,
 				       unsigned long addr, pmd_t *pmdp);
 extern int hash__has_transparent_hugepage(void);
arch/powerpc/include/asm/book3s/64/hash-64k.h +2 -2
index 0bf6fd0bf42a..8f497e1617bd 100644
@@ -274,8 +274,8 @@ extern unsigned long hash__pmd_hugepage_update(struct mm_struct *mm,
 extern pmd_t hash__pmdp_collapse_flush(struct vm_area_struct *vma,
 				   unsigned long address, pmd_t *pmdp);
 extern void hash__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
-					 pgtable_t pgtable);
-extern pgtable_t hash__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
+					     struct ptdesc *ptdesc);
+extern struct ptdesc *hash__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
 extern pmd_t hash__pmdp_huge_get_and_clear(struct mm_struct *mm,
 				       unsigned long addr, pmd_t *pmdp);
 extern int hash__has_transparent_hugepage(void);
arch/powerpc/include/asm/book3s/64/pgtable.h +1 -1
index 519b1743a0f4..0ee440b819d7 100644
@@ -1373,7 +1373,7 @@ static inline void pgtable_trans_huge_deposit(struct mm_struct *mm,
 }
 
 #define __HAVE_ARCH_PGTABLE_WITHDRAW
-static inline pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm,
+static inline struct ptdesc *pgtable_trans_huge_withdraw(struct mm_struct *mm,
 						    pmd_t *pmdp)
 {
 	if (radix_enabled())
arch/powerpc/include/asm/book3s/64/radix.h +2 -2
index 8f55ff74bb68..a8630b249f4c 100644
@@ -291,8 +291,8 @@ extern unsigned long radix__pud_hugepage_update(struct mm_struct *mm, unsigned l
 extern pmd_t radix__pmdp_collapse_flush(struct vm_area_struct *vma,
 				  unsigned long address, pmd_t *pmdp);
 extern void radix__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
-					pgtable_t pgtable);
-extern pgtable_t radix__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
+					struct ptdesc *ptdesc);
+extern struct ptdesc *radix__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
 extern pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm,
 				      unsigned long addr, pmd_t *pmdp);
 pud_t radix__pudp_huge_get_and_clear(struct mm_struct *mm,
arch/powerpc/mm/book3s64/hash_pgtable.c +2 -2
index 988948d69bc1..35562d1f4267 100644
@@ -284,7 +284,7 @@ void hash__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 	smp_wmb();
 }
 
-pgtable_t hash__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
+struct ptdesc *hash__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
 {
 	pgtable_t pgtable;
 	pgtable_t *pgtable_slot;
@@ -302,7 +302,7 @@ pgtable_t hash__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
 	 * zero out the content on withdraw.
 	 */
 	memset(pgtable, 0, PTE_FRAG_SIZE);
-	return pgtable;
+	return (struct ptdesc *)pgtable;
 }
 
 /*
arch/powerpc/mm/book3s64/radix_pgtable.c +2 -2
index b0d927009af8..3b9bb19510e3 100644
@@ -1492,7 +1492,7 @@ void radix__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 	pmd_huge_pte(mm, pmdp) = pgtable;
 }
 
-pgtable_t radix__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
+struct ptdesc *radix__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
 {
 	pte_t *ptep;
 	pgtable_t pgtable;
@@ -1513,7 +1513,7 @@ pgtable_t radix__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
 	*ptep = __pte(0);
 	ptep++;
 	*ptep = __pte(0);
-	return pgtable;
+	return (struct ptdesc *)pgtable;
 }
 
 pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm,
arch/s390/include/asm/pgtable.h +1 -1
index 3fa280d0672a..cf0baf4bfe5c 100644
@@ -1738,7 +1738,7 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 				pgtable_t pgtable);
 
 #define __HAVE_ARCH_PGTABLE_WITHDRAW
-pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
+struct ptdesc *pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
 
 #define  __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS
 static inline int pmdp_set_access_flags(struct vm_area_struct *vma,
arch/s390/mm/pgtable.c +2 -2
index 201d350abd1e..b9016ee145cb 100644
@@ -577,7 +577,7 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 	pmd_huge_pte(mm, pmdp) = (struct ptdesc *)pgtable;
 }
 
-pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
+struct ptdesc *pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
 {
 	struct list_head *lh;
 	pgtable_t pgtable;
@@ -598,7 +598,7 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
 	set_pte(ptep, __pte(_PAGE_INVALID));
 	ptep++;
 	set_pte(ptep, __pte(_PAGE_INVALID));
-	return pgtable;
+	return (struct ptdesc *)pgtable;
 }
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
 
arch/sparc/include/asm/pgtable_64.h +1 -1
index 3fe429d73a65..bfefd678e220 100644
@@ -998,7 +998,7 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 				pgtable_t pgtable);
 
 #define __HAVE_ARCH_PGTABLE_WITHDRAW
-pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
+struct ptdesc *pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
 #endif
 
 /*
arch/sparc/mm/tlb.c +2 -2
index 903825b4c997..bd2d3b1f6ba3 100644
@@ -281,7 +281,7 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 	pmd_huge_pte(mm, pmdp) = (struct ptdesc *)pgtable;
 }
 
-pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
+struct ptdesc *pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
 {
 	struct list_head *lh;
 	pgtable_t pgtable;
@@ -300,6 +300,6 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
 	pte_val(pgtable[0]) = 0;
 	pte_val(pgtable[1]) = 0;
 
-	return pgtable;
+	return (struct ptdesc *)pgtable;
 }
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
include/linux/pgtable.h +1 -1
index 2a6a3cccfc36..3fa7b93580a3 100644
@@ -929,7 +929,7 @@ extern void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 #endif
 
 #ifndef __HAVE_ARCH_PGTABLE_WITHDRAW
-extern pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
+extern struct ptdesc *pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
 #endif
 
 #ifndef arch_needs_pgtable_deposit
mm/huge_memory.c +18 -17
index 1c121ec85447..4dc36910c8aa 100644
@@ -1827,10 +1827,10 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 
 static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)
 {
-	pgtable_t pgtable;
+	struct ptdesc *ptdesc;
 
-	pgtable = pgtable_trans_huge_withdraw(mm, pmd);
-	pte_free(mm, page_ptdesc(pgtable));
+	ptdesc = pgtable_trans_huge_withdraw(mm, pmd);
+	pte_free(mm, ptdesc);
 	mm_dec_nr_ptes(mm);
 }
 
@@ -1959,9 +1959,10 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 		VM_BUG_ON(!pmd_none(*new_pmd));
 
 		if (pmd_move_must_withdraw(new_ptl, old_ptl, vma)) {
-			pgtable_t pgtable;
-			pgtable = pgtable_trans_huge_withdraw(mm, old_pmd);
-			pgtable_trans_huge_deposit(mm, new_pmd, pgtable);
+			struct ptdesc *ptdesc;
+
+			ptdesc = pgtable_trans_huge_withdraw(mm, old_pmd);
+			pgtable_trans_huge_deposit(mm, new_pmd, ptdesc_page(ptdesc));
 		}
 		pmd = move_soft_dirty_pmd(pmd);
 		set_pmd_at(mm, new_addr, new_pmd, pmd);
@@ -2130,7 +2131,7 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm
 	struct folio *src_folio;
 	struct anon_vma *src_anon_vma;
 	spinlock_t *src_ptl, *dst_ptl;
-	pgtable_t src_pgtable;
+	struct ptdesc *src_ptdesc;
 	struct mmu_notifier_range range;
 	int err = 0;
 
@@ -2234,8 +2235,8 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm
 	}
 	set_pmd_at(mm, dst_addr, dst_pmd, _dst_pmd);
 
-	src_pgtable = pgtable_trans_huge_withdraw(mm, src_pmd);
-	pgtable_trans_huge_deposit(mm, dst_pmd, src_pgtable);
+	src_ptdesc = pgtable_trans_huge_withdraw(mm, src_pmd);
+	pgtable_trans_huge_deposit(mm, dst_pmd, ptdesc_page(src_ptdesc));
 unlock_ptls:
 	double_pt_unlock(src_ptl, dst_ptl);
 	if (src_anon_vma) {
@@ -2347,7 +2348,7 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma,
 		unsigned long haddr, pmd_t *pmd)
 {
 	struct mm_struct *mm = vma->vm_mm;
-	pgtable_t pgtable;
+	struct ptdesc *ptdesc;
 	pmd_t _pmd, old_pmd;
 	unsigned long addr;
 	pte_t *pte;
@@ -2363,8 +2364,8 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma,
 	 */
 	old_pmd = pmdp_huge_clear_flush(vma, haddr, pmd);
 
-	pgtable = pgtable_trans_huge_withdraw(mm, pmd);
-	pmd_populate(mm, &_pmd, pgtable);
+	ptdesc = pgtable_trans_huge_withdraw(mm, pmd);
+	pmd_populate(mm, &_pmd, ptdesc_page(ptdesc));
 
 	pte = pte_offset_map(&_pmd, haddr);
 	VM_BUG_ON(!pte);
@@ -2381,7 +2382,7 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma,
 	}
 	pte_unmap(pte - 1);
 	smp_wmb(); /* make pte visible before pmd */
-	pmd_populate(mm, pmd, pgtable);
+	pmd_populate(mm, pmd, ptdesc_page(ptdesc));
 }
 
 static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
@@ -2390,7 +2391,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
 	struct mm_struct *mm = vma->vm_mm;
 	struct folio *folio;
 	struct page *page;
-	pgtable_t pgtable;
+	struct ptdesc *ptdesc;
 	pmd_t old_pmd, _pmd;
 	bool young, write, soft_dirty, pmd_migration = false, uffd_wp = false;
 	bool anon_exclusive = false, dirty = false;
@@ -2535,8 +2536,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
 	 * Withdraw the table only after we mark the pmd entry invalid.
 	 * This's critical for some architectures (Power).
 	 */
-	pgtable = pgtable_trans_huge_withdraw(mm, pmd);
-	pmd_populate(mm, &_pmd, pgtable);
+	ptdesc = pgtable_trans_huge_withdraw(mm, pmd);
+	pmd_populate(mm, &_pmd, ptdesc_page(ptdesc));
 
 	pte = pte_offset_map(&_pmd, haddr);
 	VM_BUG_ON(!pte);
@@ -2601,7 +2602,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
 		put_page(page);
 
 	smp_wmb(); /* make pte visible before pmd */
-	pmd_populate(mm, pmd, pgtable);
+	pmd_populate(mm, pmd, ptdesc_page(ptdesc));
 }
 
 void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address,
mm/pgtable-generic.c +2 -2
index 92245a32656b..de1ed30fea16 100644
@@ -178,7 +178,7 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 
 #ifndef __HAVE_ARCH_PGTABLE_WITHDRAW
 /* no "address" argument so destroys page coloring of some arch */
-pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
+struct ptdesc *pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
 {
 	struct ptdesc *ptdesc;
 
@@ -190,7 +190,7 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
 							  struct ptdesc, pt_list);
 	if (pmd_huge_pte(mm, pmdp))
 		list_del(&ptdesc->pt_list);
-	return ptdesc_page(ptdesc);
+	return ptdesc;
 }
 #endif
 
Signature · 1 line
2.43.0
#14 lore raw

RFC PATCH 14/18 mm/pgtable: use ptdesc in pgtable_trans_huge_deposit

In reply to [email protected]

From: Alex Shi <[email protected]>

A step to replace pgtable_t to struct ptdesc.

Cc Barry Song <[email protected]>
Cc Lance Yang <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Kinsey Ho <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Aneesh Kumar K.V <[email protected]>
Cc Christian Brauner <[email protected]>
Cc Alexander Viro <[email protected]>
Cc Jan Kara <[email protected]>
Cc Dan Williams <[email protected]>
Cc Andreas Larsson <[email protected]>
Cc David S. Miller <[email protected]>
Cc Sven Schnelle <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Vasily Gorbik <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Gerald Schaefer <[email protected]>
Cc Alexander Gordeev <[email protected]>
Cc Naveen N. Rao <[email protected]>
Cc Nicholas Piggin <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Aneesh Kumar K.V <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Peter Xu <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Christophe Leroy <[email protected]>
Cc Michael Ellerman <[email protected]>
---
 arch/powerpc/include/asm/book3s/64/pgtable.h |  6 +++---
 arch/powerpc/mm/book3s64/hash_pgtable.c      |  6 +++---
 arch/powerpc/mm/book3s64/radix_pgtable.c     |  6 +++---
 arch/s390/include/asm/pgtable.h              |  2 +-
 arch/s390/mm/pgtable.c                       |  6 +++---
 arch/sparc/include/asm/pgtable_64.h          |  2 +-
 arch/sparc/mm/tlb.c                          |  6 +++---
 fs/dax.c                                     |  2 +-
 include/linux/pgtable.h                      |  2 +-
 mm/debug_vm_pgtable.c                        |  2 +-
 mm/huge_memory.c                             | 14 +++++++-------
 mm/khugepaged.c                              |  2 +-
 mm/memory.c                                  |  2 +-
 mm/pgtable-generic.c                         |  8 ++++----
 14 files changed, 33 insertions(+), 33 deletions(-)
arch/powerpc/include/asm/book3s/64/pgtable.h +3 -3
index 0ee440b819d7..cf44e2440825 100644
@@ -1365,11 +1365,11 @@ pud_t pudp_huge_get_and_clear_full(struct vm_area_struct *vma,
 
 #define __HAVE_ARCH_PGTABLE_DEPOSIT
 static inline void pgtable_trans_huge_deposit(struct mm_struct *mm,
-					      pmd_t *pmdp, pgtable_t pgtable)
+					      pmd_t *pmdp, struct ptdesc *ptdesc)
 {
 	if (radix_enabled())
-		return radix__pgtable_trans_huge_deposit(mm, pmdp, pgtable);
-	return hash__pgtable_trans_huge_deposit(mm, pmdp, pgtable);
+		return radix__pgtable_trans_huge_deposit(mm, pmdp, ptdesc);
+	return hash__pgtable_trans_huge_deposit(mm, pmdp, ptdesc);
 }
 
 #define __HAVE_ARCH_PGTABLE_WITHDRAW
arch/powerpc/mm/book3s64/hash_pgtable.c +3 -3
index 35562d1f4267..8fd2c833dc3d 100644
@@ -265,16 +265,16 @@ pmd_t hash__pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long addres
  * the base page size hptes
  */
 void hash__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
-				  pgtable_t pgtable)
+				      struct ptdesc *ptdesc)
 {
-	pgtable_t *pgtable_slot;
+	pte_t **pgtable_slot;
 
 	assert_spin_locked(pmd_lockptr(mm, pmdp));
 	/*
 	 * we store the pgtable in the second half of PMD
 	 */
 	pgtable_slot = (pgtable_t *)pmdp + PTRS_PER_PMD;
-	*pgtable_slot = pgtable;
+	*pgtable_slot = (pte_t)ptdesc;
 	/*
 	 * expose the deposited pgtable to other cpus.
 	 * before we set the hugepage PTE at pmd level
arch/powerpc/mm/book3s64/radix_pgtable.c +3 -3
index 3b9bb19510e3..c33e860966ad 100644
@@ -1478,9 +1478,9 @@ pmd_t radix__pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long addre
  * list_head memory area.
  */
 void radix__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
-				 pgtable_t pgtable)
+				       struct ptdesc *ptdesc)
 {
-	struct list_head *lh = (struct list_head *) pgtable;
+	struct list_head *lh = (struct list_head *)ptdesc;
 
 	assert_spin_locked(pmd_lockptr(mm, pmdp));
 
@@ -1489,7 +1489,7 @@ void radix__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 		INIT_LIST_HEAD(lh);
 	else
 		list_add(lh, (struct list_head *) pmd_huge_pte(mm, pmdp));
-	pmd_huge_pte(mm, pmdp) = pgtable;
+	pmd_huge_pte(mm, pmdp) = ptdesc_page(ptdesc);
 }
 
 struct ptdesc *radix__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
arch/s390/include/asm/pgtable.h +1 -1
index cf0baf4bfe5c..d7b635f5e1e7 100644
@@ -1735,7 +1735,7 @@ pud_t pudp_xchg_direct(struct mm_struct *, unsigned long, pud_t *, pud_t);
 
 #define __HAVE_ARCH_PGTABLE_DEPOSIT
 void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
-				pgtable_t pgtable);
+				struct ptdesc *ptdesc);
 
 #define __HAVE_ARCH_PGTABLE_WITHDRAW
 struct ptdesc *pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
arch/s390/mm/pgtable.c +3 -3
index b9016ee145cb..cf1a6aeb66d4 100644
@@ -563,9 +563,9 @@ EXPORT_SYMBOL(pudp_xchg_direct);
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
-				pgtable_t pgtable)
+				struct ptdesc *ptdesc)
 {
-	struct list_head *lh = (struct list_head *) pgtable;
+	struct list_head *lh = (struct list_head *)ptdesc;
 
 	assert_spin_locked(pmd_lockptr(mm, pmdp));
 
@@ -574,7 +574,7 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 		INIT_LIST_HEAD(lh);
 	else
 		list_add(lh, (struct list_head *) pmd_huge_pte(mm, pmdp));
-	pmd_huge_pte(mm, pmdp) = (struct ptdesc *)pgtable;
+	pmd_huge_pte(mm, pmdp) = ptdesc;
 }
 
 struct ptdesc *pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
arch/sparc/include/asm/pgtable_64.h +1 -1
index bfefd678e220..c71be5ef8b06 100644
@@ -995,7 +995,7 @@ extern pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
 
 #define __HAVE_ARCH_PGTABLE_DEPOSIT
 void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
-				pgtable_t pgtable);
+				struct ptdesc *ptdesc);
 
 #define __HAVE_ARCH_PGTABLE_WITHDRAW
 struct ptdesc *pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
arch/sparc/mm/tlb.c +3 -3
index bd2d3b1f6ba3..eeed4427f524 100644
@@ -267,9 +267,9 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
 }
 
 void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
-				pgtable_t pgtable)
+				struct ptdesc *ptdesc)
 {
-	struct list_head *lh = (struct list_head *) pgtable;
+	struct list_head *lh = (struct list_head *)ptdesc;
 
 	assert_spin_locked(&mm->page_table_lock);
 
@@ -278,7 +278,7 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
 		INIT_LIST_HEAD(lh);
 	else
 		list_add(lh, (struct list_head *) pmd_huge_pte(mm, pmdp));
-	pmd_huge_pte(mm, pmdp) = (struct ptdesc *)pgtable;
+	pmd_huge_pte(mm, pmdp) = ptdesc;
 }
 
 struct ptdesc *pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)
fs/dax.c +1 -1
index 61b9bd5200da..4b4e6acb0efc 100644
@@ -1234,7 +1234,7 @@ static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault *vmf,
 	}
 
 	if (ptdesc) {
-		pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, ptdesc_page(ptdesc));
+		pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, ptdesc);
 		mm_inc_nr_ptes(vma->vm_mm);
 	}
 	pmd_entry = mk_pmd(&zero_folio->page, vmf->vma->vm_page_prot);
include/linux/pgtable.h +1 -1
index 3fa7b93580a3..9d256c548f5e 100644
@@ -925,7 +925,7 @@ static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,
 
 #ifndef __HAVE_ARCH_PGTABLE_DEPOSIT
 extern void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
-				       pgtable_t pgtable);
+				       struct ptdesc *ptdesc);
 #endif
 
 #ifndef __HAVE_ARCH_PGTABLE_WITHDRAW
mm/debug_vm_pgtable.c +1 -1
index f256bc816744..8550eec32aba 100644
@@ -225,7 +225,7 @@ static void __init pmd_advanced_tests(struct pgtable_debug_args *args)
 	/* Align the address wrt HPAGE_PMD_SIZE */
 	vaddr &= HPAGE_PMD_MASK;
 
-	pgtable_trans_huge_deposit(args->mm, args->pmdp, args->start_ptep);
+	pgtable_trans_huge_deposit(args->mm, args->pmdp, page_ptdesc(args->start_ptep));
 
 	pmd = pfn_pmd(args->pmd_pfn, args->page_prot);
 	set_pmd_at(args->mm, vaddr, args->pmdp, pmd);
mm/huge_memory.c +7 -7
index 4dc36910c8aa..aac67e8a8cc8 100644
@@ -997,7 +997,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
 		entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
 		folio_add_new_anon_rmap(folio, vma, haddr, RMAP_EXCLUSIVE);
 		folio_add_lru_vma(folio, vma);
-		pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, ptdesc_page(ptdesc));
+		pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, ptdesc);
 		set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
 		update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
 		add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
@@ -1064,7 +1064,7 @@ static void set_huge_zero_folio(struct ptdesc *ptdesc, struct mm_struct *mm,
 		return;
 	entry = mk_pmd(&zero_folio->page, vma->vm_page_prot);
 	entry = pmd_mkhuge(entry);
-	pgtable_trans_huge_deposit(mm, pmd, ptdesc_page(ptdesc));
+	pgtable_trans_huge_deposit(mm, pmd, ptdesc);
 	set_pmd_at(mm, haddr, pmd, entry);
 	mm_inc_nr_ptes(mm);
 }
@@ -1167,7 +1167,7 @@ static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,
 	}
 
 	if (ptdesc) {
-		pgtable_trans_huge_deposit(mm, pmd, ptdesc_page(ptdesc));
+		pgtable_trans_huge_deposit(mm, pmd, ptdesc);
 		mm_inc_nr_ptes(mm);
 		ptdesc = NULL;
 	}
@@ -1404,7 +1404,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		}
 		add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
 		mm_inc_nr_ptes(dst_mm);
-		pgtable_trans_huge_deposit(dst_mm, dst_pmd, ptdesc_page(ptdesc));
+		pgtable_trans_huge_deposit(dst_mm, dst_pmd, ptdesc);
 		if (!userfaultfd_wp(dst_vma))
 			pmd = pmd_swp_clear_uffd_wp(pmd);
 		set_pmd_at(dst_mm, addr, dst_pmd, pmd);
@@ -1449,7 +1449,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
 out_zero_page:
 	mm_inc_nr_ptes(dst_mm);
-	pgtable_trans_huge_deposit(dst_mm, dst_pmd, ptdesc_page(ptdesc));
+	pgtable_trans_huge_deposit(dst_mm, dst_pmd, ptdesc);
 	pmdp_set_wrprotect(src_mm, addr, src_pmd);
 	if (!userfaultfd_wp(dst_vma))
 		pmd = pmd_clear_uffd_wp(pmd);
@@ -1962,7 +1962,7 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 			struct ptdesc *ptdesc;
 
 			ptdesc = pgtable_trans_huge_withdraw(mm, old_pmd);
-			pgtable_trans_huge_deposit(mm, new_pmd, ptdesc_page(ptdesc));
+			pgtable_trans_huge_deposit(mm, new_pmd, ptdesc);
 		}
 		pmd = move_soft_dirty_pmd(pmd);
 		set_pmd_at(mm, new_addr, new_pmd, pmd);
@@ -2236,7 +2236,7 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm
 	set_pmd_at(mm, dst_addr, dst_pmd, _dst_pmd);
 
 	src_ptdesc = pgtable_trans_huge_withdraw(mm, src_pmd);
-	pgtable_trans_huge_deposit(mm, dst_pmd, ptdesc_page(src_ptdesc));
+	pgtable_trans_huge_deposit(mm, dst_pmd, src_ptdesc);
 unlock_ptls:
 	double_pt_unlock(src_ptl, dst_ptl);
 	if (src_anon_vma) {
mm/khugepaged.c +1 -1
index f3b3db104615..48a54269472e 100644
@@ -1232,7 +1232,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
 	BUG_ON(!pmd_none(*pmd));
 	folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE);
 	folio_add_lru_vma(folio, vma);
-	pgtable_trans_huge_deposit(mm, pmd, pgtable);
+	pgtable_trans_huge_deposit(mm, pmd, page_ptdesc(pgtable));
 	set_pmd_at(mm, address, pmd, _pmd);
 	update_mmu_cache_pmd(vma, address, pmd);
 	spin_unlock(pmd_ptl);
mm/memory.c +1 -1
index 27c2f63b7487..956cfe5f644d 100644
@@ -4687,7 +4687,7 @@ static void deposit_prealloc_pte(struct vm_fault *vmf)
 {
 	struct vm_area_struct *vma = vmf->vma;
 
-	pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, vmf->prealloc_pte);
+	pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, page_ptdesc(vmf->prealloc_pte));
 	/*
 	 * We are going to consume the prealloc table,
 	 * count that as nr_ptes.
mm/pgtable-generic.c +4 -4
index de1ed30fea16..5e763682941d 100644
@@ -163,16 +163,16 @@ pud_t pudp_huge_clear_flush(struct vm_area_struct *vma, unsigned long address,
 
 #ifndef __HAVE_ARCH_PGTABLE_DEPOSIT
 void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
-				pgtable_t pgtable)
+				struct ptdesc *ptdesc)
 {
 	assert_spin_locked(pmd_lockptr(mm, pmdp));
 
 	/* FIFO */
 	if (!pmd_huge_pte(mm, pmdp))
-		INIT_LIST_HEAD(&pgtable->lru);
+		INIT_LIST_HEAD(&ptdesc->pt_list);
 	else
-		list_add(&pgtable->lru, &pmd_huge_pte(mm, pmdp)->pt_list);
-	pmd_huge_pte(mm, pmdp) = page_ptdesc(pgtable);
+		list_add(&ptdesc->pt_list, &pmd_huge_pte(mm, pmdp)->pt_list);
+	pmd_huge_pte(mm, pmdp) = ptdesc;
 }
 #endif
 
Signature · 1 line
2.43.0
#15 lore raw

RFC PATCH 15/18 mm/pgtable: pass ptdesc to pmd_populate

In reply to [email protected]

From: Alex Shi <[email protected]>

Pass struct ptdesc to pmd_populate to further replace pgtable_t. We use type casting instead of page_ptdesc() helper since different arch has different type of pgtable_t.

Helper ptdesc_pfn used for arch openrisc and hexagon.

Cc H. Peter Anvin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Helge Deller <[email protected]>
Cc James E.J. Bottomley <[email protected]>
Cc Stafford Horne <[email protected]>
Cc Stefan Kristiansson <[email protected]>
Cc Jonas Bonn <[email protected]>
Cc Sam Creasey <[email protected]>
Cc Geert Uytterhoeven <[email protected]>
Cc WANG Xuerui <[email protected]>
Cc Huacai Chen <[email protected]>
Cc Will Deacon <[email protected]>
Cc Russell King <[email protected]>
Cc Vineet Gupta <[email protected]>
Cc Matt Turner <[email protected]>
Cc Ivan Kokshaysky <[email protected]>
Cc Richard Henderson <[email protected]>
Cc Breno Leitao <[email protected]>
Cc Josh Poimboeuf <[email protected]>
Cc Bibo Mao <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Vishal Moola <[email protected]>
Cc Ard Biesheuvel <[email protected]>
---
 arch/alpha/include/asm/pgalloc.h             | 4 ++--
 arch/arc/include/asm/pgalloc.h               | 4 ++--
 arch/arm/include/asm/pgalloc.h               | 4 ++--
 arch/arm64/include/asm/pgalloc.h             | 4 ++--
 arch/hexagon/include/asm/pgalloc.h           | 4 ++--
 arch/loongarch/include/asm/pgalloc.h         | 4 ++--
 arch/m68k/include/asm/motorola_pgalloc.h     | 4 ++--
 arch/m68k/include/asm/sun3_pgalloc.h         | 4 ++--
 arch/microblaze/include/asm/pgalloc.h        | 2 +-
 arch/mips/include/asm/pgalloc.h              | 4 ++--
 arch/nios2/include/asm/pgalloc.h             | 4 ++--
 arch/openrisc/include/asm/pgalloc.h          | 4 ++--
 arch/parisc/include/asm/pgalloc.h            | 2 +-
 arch/powerpc/include/asm/book3s/32/pgalloc.h | 2 +-
 arch/sh/include/asm/pgalloc.h                | 4 ++--
 arch/sparc/include/asm/pgalloc_32.h          | 2 +-
 arch/x86/include/asm/pgalloc.h               | 4 ++--
 mm/debug_vm_pgtable.c                        | 2 +-
 mm/huge_memory.c                             | 8 ++++----
 mm/khugepaged.c                              | 4 ++--
 mm/memory.c                                  | 2 +-
 mm/mremap.c                                  | 2 +-
 22 files changed, 39 insertions(+), 39 deletions(-)
arch/alpha/include/asm/pgalloc.h +2 -2
index 68be7adbfe58..ad62056059ac 100644
@@ -14,9 +14,9 @@
  */
 
 static inline void
-pmd_populate(struct mm_struct *mm, pmd_t *pmd, pgtable_t pte)
+pmd_populate(struct mm_struct *mm, pmd_t *pmd, struct ptdesc *pte)
 {
-	pmd_set(pmd, (pte_t *)(page_to_pa(pte) + PAGE_OFFSET));
+	pmd_set(pmd, (pte_t *)(page_to_pa(ptdesc_page(pte)) + PAGE_OFFSET));
 }
 
 static inline void
arch/arc/include/asm/pgalloc.h +2 -2
index 096b8ef58edb..51233cfb1bad 100644
@@ -46,9 +46,9 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte)
 	set_pmd(pmd, __pmd((unsigned long)pte));
 }
 
-static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, pgtable_t pte_page)
+static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, struct ptdesc *pte)
 {
-	set_pmd(pmd, __pmd((unsigned long)page_address(pte_page)));
+	set_pmd(pmd, __pmd((unsigned long)ptdesc_address(pte)));
 }
 
 static inline pgd_t *pgd_alloc(struct mm_struct *mm)
arch/arm/include/asm/pgalloc.h +2 -2
index e8501a6c3336..37a15220fce7 100644
@@ -130,7 +130,7 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp, pte_t *ptep)
 }
 
 static inline void
-pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep)
+pmd_populate(struct mm_struct *mm, pmd_t *pmdp, struct ptdesc *ptep)
 {
 	extern pmdval_t user_pmd_table;
 	pmdval_t prot;
@@ -140,7 +140,7 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep)
 	else
 		prot = _PAGE_USER_TABLE;
 
-	__pmd_populate(pmdp, page_to_phys(ptep), prot);
+	__pmd_populate(pmdp, page_to_phys(ptdesc_page(ptep)), prot);
 }
 
 #endif /* CONFIG_MMU */
arch/arm64/include/asm/pgalloc.h +2 -2
index 8ff5f2a2579e..d9074b5f9dfe 100644
@@ -131,10 +131,10 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp, pte_t *ptep)
 }
 
 static inline void
-pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep)
+pmd_populate(struct mm_struct *mm, pmd_t *pmdp, struct ptdesc *ptep)
 {
 	VM_BUG_ON(mm == &init_mm);
-	__pmd_populate(pmdp, page_to_phys(ptep), PMD_TYPE_TABLE | PMD_TABLE_PXN);
+	__pmd_populate(pmdp, page_to_phys(ptdesc_page(ptep)), PMD_TYPE_TABLE | PMD_TABLE_PXN);
 }
 
 #endif
arch/hexagon/include/asm/pgalloc.h +2 -2
index a3e082e54b74..f34e9fcad066 100644
@@ -42,13 +42,13 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
 }
 
 static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,
-				pgtable_t pte)
+				struct ptdesc *pte)
 {
 	/*
 	 * Conveniently, zero in 3 LSB means indirect 4K page table.
 	 * Not so convenient when you're trying to vary the page size.
 	 */
-	set_pmd(pmd, __pmd(((unsigned long)page_to_pfn(pte) << PAGE_SHIFT) |
+	set_pmd(pmd, __pmd(((unsigned long)ptdesc_pfn(pte) << PAGE_SHIFT) |
 		HEXAGON_L1_PTE_SIZE));
 }
 
arch/loongarch/include/asm/pgalloc.h +2 -2
index c96d7160babc..3461da516ab9 100644
@@ -18,9 +18,9 @@ static inline void pmd_populate_kernel(struct mm_struct *mm,
 	set_pmd(pmd, __pmd((unsigned long)pte));
 }
 
-static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, pgtable_t pte)
+static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, struct ptdesc *pte)
 {
-	set_pmd(pmd, __pmd((unsigned long)page_address(pte)));
+	set_pmd(pmd, __pmd((unsigned long)ptdesc_address(pte)));
 }
 
 #ifndef __PAGETABLE_PMD_FOLDED
arch/m68k/include/asm/motorola_pgalloc.h +2 -2
index f9ee5ec4574d..a80c45b9d2a3 100644
@@ -84,9 +84,9 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *
 	pmd_set(pmd, pte);
 }
 
-static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, pgtable_t page)
+static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, struct ptdesc *page)
 {
-	pmd_set(pmd, page);
+	pmd_set(pmd, ptdesc_page(page));
 }
 
 static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)
arch/m68k/include/asm/sun3_pgalloc.h +2 -2
index 4a137eecb6fe..965f663a4797 100644
@@ -28,9 +28,9 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *
 	pmd_val(*pmd) = __pa((unsigned long)pte);
 }
 
-static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, pgtable_t page)
+static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, struct ptdesc *ptdesc)
 {
-	pmd_val(*pmd) = __pa((unsigned long)page_address(page));
+	pmd_val(*pmd) = __pa((unsigned long)ptdesc_address(ptdesc));
 }
 
 /*
arch/microblaze/include/asm/pgalloc.h +1 -1
index 6c33b05f730f..0f4a479e015e 100644
@@ -33,7 +33,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
 #define __pte_free_tlb(tlb, pte, addr)	pte_free((tlb)->mm, (pte))
 
 #define pmd_populate(mm, pmd, pte) \
-			(pmd_val(*(pmd)) = (unsigned long)page_address(pte))
+			(pmd_val(*(pmd)) = (unsigned long)page_address(ptdesc_page(pte)))
 
 #define pmd_populate_kernel(mm, pmd, pte) \
 		(pmd_val(*(pmd)) = (unsigned long) (pte))
arch/mips/include/asm/pgalloc.h +2 -2
index f4440edcd8fe..2ef868d93b6b 100644
@@ -25,9 +25,9 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd,
 }
 
 static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,
-	pgtable_t pte)
+				struct ptdesc *pte)
 {
-	set_pmd(pmd, __pmd((unsigned long)page_address(pte)));
+	set_pmd(pmd, __pmd((unsigned long)ptdesc_address(pte)));
 }
 
 /*
arch/nios2/include/asm/pgalloc.h +2 -2
index ce6bb8e74271..420958d91a47 100644
@@ -21,9 +21,9 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd,
 }
 
 static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,
-	pgtable_t pte)
+				struct ptdesc *pte)
 {
-	set_pmd(pmd, __pmd((unsigned long)page_address(pte)));
+	set_pmd(pmd, __pmd((unsigned long)ptdesc_address(pte)));
 }
 
 extern pgd_t *pgd_alloc(struct mm_struct *mm);
arch/openrisc/include/asm/pgalloc.h +2 -2
index 2251d940c3d8..a9479d873dca 100644
@@ -29,10 +29,10 @@ extern int mem_init_done;
 	set_pmd(pmd, __pmd(_KERNPG_TABLE + __pa(pte)))
 
 static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,
-				struct page *pte)
+				struct ptdesc *pte)
 {
 	set_pmd(pmd, __pmd(_KERNPG_TABLE +
-		     ((unsigned long)page_to_pfn(pte) <<
+		     ((unsigned long)ptdesc_pfn(pte) <<
 		     (unsigned long) PAGE_SHIFT)));
 }
 
arch/parisc/include/asm/pgalloc.h +1 -1
index e3e142b1c5c5..9fd06e2fef89 100644
@@ -68,6 +68,6 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte)
 }
 
 #define pmd_populate(mm, pmd, pte_page) \
-	pmd_populate_kernel(mm, pmd, page_address(pte_page))
+	pmd_populate_kernel(mm, pmd, page_address(ptdesc_page(pte_page)))
 
 #endif
arch/powerpc/include/asm/book3s/32/pgalloc.h +1 -1
index a435c84d1f9a..9971703d0566 100644
@@ -32,7 +32,7 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp,
 }
 
 static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmdp,
-				pgtable_t pte_page)
+				struct ptdesc *pte_page)
 {
 	*pmdp = __pmd(__pa(pte_page) | _PMD_PRESENT);
 }
arch/sh/include/asm/pgalloc.h +2 -2
index 5d8577ab1591..095521089c20 100644
@@ -27,9 +27,9 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd,
 }
 
 static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,
-				pgtable_t pte)
+				struct ptdesc *pte)
 {
-	set_pmd(pmd, __pmd((unsigned long)page_address(pte)));
+	set_pmd(pmd, __pmd((unsigned long)ptdesc_address(pte)));
 }
 
 #define __pte_free_tlb(tlb, pte, addr)				\
arch/sparc/include/asm/pgalloc_32.h +1 -1
index addaade56f21..6f0f661a380f 100644
@@ -50,7 +50,7 @@ static inline void free_pmd_fast(pmd_t * pmd)
 #define pmd_free(mm, pmd)		free_pmd_fast(pmd)
 #define __pmd_free_tlb(tlb, pmd, addr)	pmd_free((tlb)->mm, pmd)
 
-#define pmd_populate(mm, pmd, pte)	pmd_set(pmd, pte)
+#define pmd_populate(mm, pmd, pte)	pmd_set(pmd, (pte_t *)pte)
 
 void pmd_set(pmd_t *pmdp, pte_t *ptep);
 #define pmd_populate_kernel		pmd_populate
arch/x86/include/asm/pgalloc.h +2 -2
index 06a9a5867a86..5ca8ac568768 100644
@@ -76,9 +76,9 @@ static inline void pmd_populate_kernel_safe(struct mm_struct *mm,
 }
 
 static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,
-				struct page *pte)
+				struct ptdesc *pte)
 {
-	unsigned long pfn = page_to_pfn(pte);
+	unsigned long pfn = page_to_pfn(ptdesc_page(pte));
 
 	paravirt_alloc_pte(mm, pfn);
 	set_pmd(pmd, __pmd(((pteval_t)pfn << PAGE_SHIFT) | _PAGE_TABLE));
mm/debug_vm_pgtable.c +1 -1
index 8550eec32aba..bf9dc9c0a9bf 100644
@@ -645,7 +645,7 @@ static void __init pmd_populate_tests(struct pgtable_debug_args *args)
 	 * This entry points to next level page table page.
 	 * Hence this must not qualify as pmd_bad().
 	 */
-	pmd_populate(args->mm, args->pmdp, args->start_ptep);
+	pmd_populate(args->mm, args->pmdp, page_ptdesc(args->start_ptep));
 	pmd = READ_ONCE(*args->pmdp);
 	WARN_ON(pmd_bad(pmd));
 }
mm/huge_memory.c +4 -4
index aac67e8a8cc8..665445706491 100644
@@ -2365,7 +2365,7 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma,
 	old_pmd = pmdp_huge_clear_flush(vma, haddr, pmd);
 
 	ptdesc = pgtable_trans_huge_withdraw(mm, pmd);
-	pmd_populate(mm, &_pmd, ptdesc_page(ptdesc));
+	pmd_populate(mm, &_pmd, ptdesc);
 
 	pte = pte_offset_map(&_pmd, haddr);
 	VM_BUG_ON(!pte);
@@ -2382,7 +2382,7 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma,
 	}
 	pte_unmap(pte - 1);
 	smp_wmb(); /* make pte visible before pmd */
-	pmd_populate(mm, pmd, ptdesc_page(ptdesc));
+	pmd_populate(mm, pmd, ptdesc);
 }
 
 static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
@@ -2537,7 +2537,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
 	 * This's critical for some architectures (Power).
 	 */
 	ptdesc = pgtable_trans_huge_withdraw(mm, pmd);
-	pmd_populate(mm, &_pmd, ptdesc_page(ptdesc));
+	pmd_populate(mm, &_pmd, ptdesc);
 
 	pte = pte_offset_map(&_pmd, haddr);
 	VM_BUG_ON(!pte);
@@ -2602,7 +2602,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
 		put_page(page);
 
 	smp_wmb(); /* make pte visible before pmd */
-	pmd_populate(mm, pmd, ptdesc_page(ptdesc));
+	pmd_populate(mm, pmd, ptdesc);
 }
 
 void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address,
mm/khugepaged.c +2 -2
index 48a54269472e..5b466a1c2136 100644
@@ -769,7 +769,7 @@ static void __collapse_huge_page_copy_failed(pte_t *pte,
 	 * acquiring anon_vma_lock_write is unnecessary.
 	 */
 	pmd_ptl = pmd_lock(vma->vm_mm, pmd);
-	pmd_populate(vma->vm_mm, pmd, pmd_pgtable(orig_pmd));
+	pmd_populate(vma->vm_mm, pmd, (struct ptdesc *)pmd_pgtable(orig_pmd));
 	spin_unlock(pmd_ptl);
 	/*
 	 * Release both raw and compound pages isolated
@@ -1198,7 +1198,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
 		 * hugepmds and never for establishing regular pmds that
 		 * points to regular pagetables. Use pmd_populate for that
 		 */
-		pmd_populate(mm, pmd, pmd_pgtable(_pmd));
+		pmd_populate(mm, pmd, (struct ptdesc *)pmd_pgtable(_pmd));
 		spin_unlock(pmd_ptl);
 		anon_vma_unlock_write(vma->anon_vma);
 		goto out_up_write;
mm/memory.c +1 -1
index 956cfe5f644d..cbed8824059f 100644
@@ -438,7 +438,7 @@ void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte)
 		 * smp_rmb() barriers in page table walking code.
 		 */
 		smp_wmb(); /* Could be smp_wmb__xxx(before|after)_spin_lock */
-		pmd_populate(mm, pmd, *pte);
+		pmd_populate(mm, pmd, (struct ptdesc *)(*pte));
 		*pte = NULL;
 	}
 	spin_unlock(ptl);
mm/mremap.c +1 -1
index e7ae140fc640..f32d35accd97 100644
@@ -283,7 +283,7 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 
 	VM_BUG_ON(!pmd_none(*new_pmd));
 
-	pmd_populate(mm, new_pmd, pmd_pgtable(pmd));
+	pmd_populate(mm, new_pmd, (struct ptdesc *)pmd_pgtable(pmd));
 	flush_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
 	if (new_ptl != old_ptl)
 		spin_unlock(new_ptl);
Signature · 1 line
2.43.0
#16 lore raw

RFC PATCH 16/18 mm/pgtable: pass ptdesc to pmd_install

In reply to [email protected]

From: Alex Shi <[email protected]>

A new step to replace pgtable_t by ptdesc, also a preparation to change vmf.prealloc_pte to ptdesc too.

---
 mm/filemap.c  | 2 +-
 mm/internal.h | 2 +-
 mm/memory.c   | 8 ++++----
 3 files changed, 6 insertions(+), 6 deletions(-)
mm/filemap.c +1 -1
index d62150418b91..3708ef71182e 100644
@@ -3453,7 +3453,7 @@ static bool filemap_map_pmd(struct vm_fault *vmf, struct folio *folio,
 	}
 
 	if (pmd_none(*vmf->pmd) && vmf->prealloc_pte)
-		pmd_install(mm, vmf->pmd, &vmf->prealloc_pte);
+		pmd_install(mm, vmf->pmd, (struct ptdesc **)&vmf->prealloc_pte);
 
 	return false;
 }
mm/internal.h +1 -1
index 7a3bcc6d95e7..e4bc64d5176a 100644
@@ -320,7 +320,7 @@ void folio_activate(struct folio *folio);
 void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		   struct vm_area_struct *start_vma, unsigned long floor,
 		   unsigned long ceiling, bool mm_wr_locked);
-void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte);
+void pmd_install(struct mm_struct *mm, pmd_t *pmd, struct ptdesc **pte);
 
 struct zap_details;
 void unmap_page_range(struct mmu_gather *tlb,
mm/memory.c +4 -4
index cbed8824059f..79685600d23f 100644
@@ -418,7 +418,7 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 	} while (vma);
 }
 
-void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte)
+void pmd_install(struct mm_struct *mm, pmd_t *pmd, struct ptdesc **pte)
 {
 	spinlock_t *ptl = pmd_lock(mm, pmd);
 
@@ -438,7 +438,7 @@ void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte)
 		 * smp_rmb() barriers in page table walking code.
 		 */
 		smp_wmb(); /* Could be smp_wmb__xxx(before|after)_spin_lock */
-		pmd_populate(mm, pmd, (struct ptdesc *)(*pte));
+		pmd_populate(mm, pmd, *pte);
 		*pte = NULL;
 	}
 	spin_unlock(ptl);
@@ -450,7 +450,7 @@ int __pte_alloc(struct mm_struct *mm, pmd_t *pmd)
 	if (!ptdesc)
 		return -ENOMEM;
 
-	pmd_install(mm, pmd, (pgtable_t *)&ptdesc);
+	pmd_install(mm, pmd, &ptdesc);
 	if (ptdesc)
 		pte_free(mm, ptdesc);
 	return 0;
@@ -4868,7 +4868,7 @@ vm_fault_t finish_fault(struct vm_fault *vmf)
 		}
 
 		if (vmf->prealloc_pte)
-			pmd_install(vma->vm_mm, vmf->pmd, &vmf->prealloc_pte);
+			pmd_install(vma->vm_mm, vmf->pmd, (struct ptdesc **)&vmf->prealloc_pte);
 		else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd)))
 			return VM_FAULT_OOM;
 	}
Signature · 1 line
2.43.0
#17 lore raw

RFC PATCH 17/18 mm: convert vmf.prealloc_pte to struct ptdesc pointer

In reply to [email protected]

From: Alex Shi <[email protected]>

vmfs.prealloc_pte is a pointer to page table memory, so converter it to struct ptdesc pointer.

---
 include/linux/mm.h |  2 +-
 mm/filemap.c       |  2 +-
 mm/memory.c        | 12 ++++++------
 3 files changed, 8 insertions(+), 8 deletions(-)
include/linux/mm.h +1 -1
index 7424f964dff3..749d6dd311fa 100644
@@ -567,7 +567,7 @@ struct vm_fault {
 					 * Protects pte page table if 'pte'
 					 * is not NULL, otherwise pmd.
 					 */
-	pgtable_t prealloc_pte;		/* Pre-allocated pte page table.
+	struct ptdesc *prealloc_pte;	/* Pre-allocated pte page table.
 					 * vm_ops->map_pages() sets up a page
 					 * table from atomic context.
 					 * do_fault_around() pre-allocates
mm/filemap.c +1 -1
index 3708ef71182e..d62150418b91 100644
@@ -3453,7 +3453,7 @@ static bool filemap_map_pmd(struct vm_fault *vmf, struct folio *folio,
 	}
 
 	if (pmd_none(*vmf->pmd) && vmf->prealloc_pte)
-		pmd_install(mm, vmf->pmd, (struct ptdesc **)&vmf->prealloc_pte);
+		pmd_install(mm, vmf->pmd, &vmf->prealloc_pte);
 
 	return false;
 }
mm/memory.c +6 -6
index 79685600d23f..1a5fb17ab045 100644
@@ -4648,7 +4648,7 @@ static vm_fault_t __do_fault(struct vm_fault *vmf)
 	 *				# flush A, B to clear the writeback
 	 */
 	if (pmd_none(*vmf->pmd) && !vmf->prealloc_pte) {
-		vmf->prealloc_pte = ptdesc_page(pte_alloc_one(vma->vm_mm));
+		vmf->prealloc_pte = pte_alloc_one(vma->vm_mm);
 		if (!vmf->prealloc_pte)
 			return VM_FAULT_OOM;
 	}
@@ -4687,7 +4687,7 @@ static void deposit_prealloc_pte(struct vm_fault *vmf)
 {
 	struct vm_area_struct *vma = vmf->vma;
 
-	pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, page_ptdesc(vmf->prealloc_pte));
+	pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, vmf->prealloc_pte);
 	/*
 	 * We are going to consume the prealloc table,
 	 * count that as nr_ptes.
@@ -4726,7 +4726,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page)
 	 * related to pte entry. Use the preallocated table for that.
 	 */
 	if (arch_needs_pgtable_deposit() && !vmf->prealloc_pte) {
-		vmf->prealloc_pte = ptdesc_page(pte_alloc_one(vma->vm_mm));
+		vmf->prealloc_pte = pte_alloc_one(vma->vm_mm);
 		if (!vmf->prealloc_pte)
 			return VM_FAULT_OOM;
 	}
@@ -4868,7 +4868,7 @@ vm_fault_t finish_fault(struct vm_fault *vmf)
 		}
 
 		if (vmf->prealloc_pte)
-			pmd_install(vma->vm_mm, vmf->pmd, (struct ptdesc **)&vmf->prealloc_pte);
+			pmd_install(vma->vm_mm, vmf->pmd, &vmf->prealloc_pte);
 		else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd)))
 			return VM_FAULT_OOM;
 	}
@@ -5011,7 +5011,7 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf)
 		      pte_off + vma_pages(vmf->vma) - vma_off) - 1;
 
 	if (pmd_none(*vmf->pmd)) {
-		vmf->prealloc_pte = ptdesc_page(pte_alloc_one(vmf->vma->vm_mm));
+		vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm);
 		if (!vmf->prealloc_pte)
 			return VM_FAULT_OOM;
 	}
@@ -5197,7 +5197,7 @@ static vm_fault_t do_fault(struct vm_fault *vmf)
 
 	/* preallocated pagetable is unused: free it */
 	if (vmf->prealloc_pte) {
-		pte_free(vm_mm, page_ptdesc(vmf->prealloc_pte));
+		pte_free(vm_mm, vmf->prealloc_pte);
 		vmf->prealloc_pte = NULL;
 	}
 	return ret;
Signature · 1 line
2.43.0
#18 lore raw

RFC PATCH 18/18 mm/pgtable: pass ptdesc in pte_free_defer

In reply to [email protected]

From: Alex Shi <[email protected]>

pass ptdesc in pte_free_defer() and use ptdesc in collapse_huge_page().

This patch is immature, there is a issue from pmd_pgtable() conversion in few archs. The problem need a fix.

Cc Mike Rapoport <[email protected]>
Cc Barry Song <[email protected]>
Cc Peter Xu <[email protected]>
Cc Lance Yang <[email protected]>
Cc Kemeng Shi <[email protected]>
Cc Aneesh Kumar K.V <[email protected]>
Cc Qi Zheng <[email protected]>
Cc Sven Schnelle <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Vasily Gorbik <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Naveen N Rao <[email protected]>
Cc Nicholas Piggin <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Vishal Moola <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Gerald Schaefer <[email protected]>
Cc Alexander Gordeev <[email protected]>
Cc Christophe Leroy <[email protected]>
---
 arch/powerpc/include/asm/pgalloc.h |  2 +-
 arch/s390/include/asm/pgalloc.h    |  2 +-
 arch/s390/mm/pgalloc.c             |  2 +-
 include/linux/pgtable.h            |  2 +-
 mm/khugepaged.c                    | 10 +++++-----
 mm/pgtable-generic.c               |  4 +---
 6 files changed, 10 insertions(+), 12 deletions(-)
arch/powerpc/include/asm/pgalloc.h +1 -1
index 12520521163e..ca21b67c593f 100644
@@ -47,7 +47,7 @@ static inline void pte_free(struct mm_struct *mm, struct ptdesc *ptepage)
 
 /* arch use pte_free_defer() implementation in arch/powerpc/mm/pgtable-frag.c */
 #define pte_free_defer pte_free_defer
-void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable);
+void pte_free_defer(struct mm_struct *mm, struct ptdesc *pgtable);
 
 /*
  * Functions that deal with pagetables that could be at any level of
arch/s390/include/asm/pgalloc.h +1 -1
index 771494526f6e..a229cee11bbd 100644
@@ -144,7 +144,7 @@ static inline void pmd_populate(struct mm_struct *mm,
 
 /* arch use pte_free_defer() implementation in arch/s390/mm/pgalloc.c */
 #define pte_free_defer pte_free_defer
-void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable);
+void pte_free_defer(struct mm_struct *mm, struct ptdesc *pgtable);
 
 void vmem_map_init(void);
 void *vmem_crst_alloc(unsigned long val);
arch/s390/mm/pgalloc.c +1 -1
index f691e0fb66a2..c7bb38d85d81 100644
@@ -214,7 +214,7 @@ static void pte_free_now(struct rcu_head *head)
 	pagetable_pte_dtor_free(ptdesc);
 }
 
-void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable)
+void pte_free_defer(struct mm_struct *mm, struct ptdesc *pgtable)
 {
 	struct ptdesc *ptdesc = virt_to_ptdesc(pgtable);
 
include/linux/pgtable.h +1 -1
index 9d256c548f5e..e7b018de1d0f 100644
@@ -116,7 +116,7 @@ static inline void pte_unmap(pte_t *pte)
 }
 #endif
 
-void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable);
+void pte_free_defer(struct mm_struct *mm, struct ptdesc *ptdesc);
 
 /* Find an entry in the second-level page table.. */
 #ifndef pmd_offset
mm/khugepaged.c +5 -5
index 5b466a1c2136..30cf61d02c1c 100644
@@ -1094,7 +1094,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
 	LIST_HEAD(compound_pagelist);
 	pmd_t *pmd, _pmd;
 	pte_t *pte;
-	pgtable_t pgtable;
+	struct ptdesc *ptdesc;
 	struct folio *folio;
 	spinlock_t *pmd_ptl, *pte_ptl;
 	int result = SCAN_FAIL;
@@ -1223,7 +1223,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
 	 * write.
 	 */
 	__folio_mark_uptodate(folio);
-	pgtable = pmd_pgtable(_pmd);
+	ptdesc = pmd_ptdesc(&_pmd);
 
 	_pmd = mk_huge_pmd(&folio->page, vma->vm_page_prot);
 	_pmd = maybe_pmd_mkwrite(pmd_mkdirty(_pmd), vma);
@@ -1232,7 +1232,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
 	BUG_ON(!pmd_none(*pmd));
 	folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE);
 	folio_add_lru_vma(folio, vma);
-	pgtable_trans_huge_deposit(mm, pmd, page_ptdesc(pgtable));
+	pgtable_trans_huge_deposit(mm, pmd, ptdesc);
 	set_pmd_at(mm, address, pmd, _pmd);
 	update_mmu_cache_pmd(vma, address, pmd);
 	spin_unlock(pmd_ptl);
@@ -1664,7 +1664,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
 
 	mm_dec_nr_ptes(mm);
 	page_table_check_pte_clear_range(mm, haddr, pgt_pmd);
-	pte_free_defer(mm, pmd_pgtable(pgt_pmd));
+	pte_free_defer(mm, pmd_ptdesc(&pgt_pmd));
 
 maybe_install_pmd:
 	/* step 5: install pmd entry */
@@ -1777,7 +1777,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)
 		if (retracted) {
 			mm_dec_nr_ptes(mm);
 			page_table_check_pte_clear_range(mm, addr, pgt_pmd);
-			pte_free_defer(mm, pmd_pgtable(pgt_pmd));
+			pte_free_defer(mm, pmd_ptdesc(&pgt_pmd));
 		}
 	}
 	i_mmap_unlock_read(mapping);
mm/pgtable-generic.c +1 -3
index 5e763682941d..f3bc2b17893a 100644
@@ -244,10 +244,8 @@ static void pte_free_now(struct rcu_head *head)
 	pte_free(NULL /* mm not passed and not used */, ptdesc);
 }
 
-void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable)
+void pte_free_defer(struct mm_struct *mm, struct ptdesc *ptdesc)
 {
-	struct ptdesc *ptdesc = page_ptdesc(pgtable);
-
 	call_rcu(&ptdesc->pt_rcu_head, pte_free_now);
 }
 #endif /* pte_free_defer */
Signature · 1 line
2.43.0
#19 lore raw
In reply to [email protected]

BTW, the patchset based on the latest git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm mm-unstable branch.

Quoted 106 lines · On 7/30/24 2:46 PM, [email protected] wrote:

From: Alex Shi <[email protected]>

We have struct ptdesc for page table descriptor a year ago, but it has no much usages in kernel, while pgtable_t is used widely.

The pgtable_t is typedefed as 'pte_t *' in sparc, s390, powerpc and m68k except SUN3, others archs are all same as 'struct page *'.

These blocks the conception and code update for page table descriptor to struct ptdesc.

So, the simple idea to push the ptdesc conception forward is to update all pgtable_t by ptdesc or pte_t pointer. But this needs widely knowledges for most all of different archs. Common code change is easy for include/ and mm/ directory, but it's hard in all archs.

Thanks for intel LKP framework, I fixed most all of build issues except a bug on powerpc which reports a "struct ptdesc *" incompatible with struct ptdesc *' pointer issue...

Another trouble is pmd_pgtable() conversion in the last patch. Maybe some of arch need define theirself own pmd_ptdesc()?

This patchset is immature, even except above 2 issues, I just tested virutal machine booting and kselftest mm on x86 and arm64.

Anyway any input are appreciated!

Thanks
Alex

Alex Shi (18):
mm/pgtable: use ptdesc in pte_free_now/pte_free_defer mm/pgtable: convert ptdesc.pmd_huge_pte to ptdesc pointer fs/dax: use ptdesc in dax_pmd_load_hole
mm/thp: use ptdesc pointer in __do_huge_pmd_anonymous_page mm/thp: use ptdesc in do_huge_pmd_anonymous_page
mm/thp: convert insert_pfn_pmd and its caller to use ptdesc mm/thp: use ptdesc in copy_huge_pmd
mm/memory: use ptdesc in __pte_alloc
mm/pgtable: fully use ptdesc in pte_alloc_one series functions mm/pgtable: pass ptdesc to pte_free()
mm/pgtable: introduce ptdesc_pfn and use ptdesc in free_pte_range() mm/thp: pass ptdesc to set_huge_zero_folio function mm/pgtable: return ptdesc pointer in pgtable_trans_huge_withdraw mm/pgtable: use ptdesc in pgtable_trans_huge_deposit mm/pgtable: pass ptdesc to pmd_populate
mm/pgtable: pass ptdesc to pmd_install
mm: convert vmf.prealloc_pte to struct ptdesc pointer mm/pgtable: pass ptdesc in pte_free_defer

 arch/alpha/include/asm/pgalloc.h              |   4 +-
 arch/arc/include/asm/pgalloc.h                |   4 +-
 arch/arm/include/asm/pgalloc.h                |  13 +--
 arch/arm/include/asm/tlb.h                    |   4 +-
 arch/arm/mm/pgd.c                             |   2 +-
 arch/arm64/include/asm/pgalloc.h              |   4 +-
 arch/arm64/include/asm/tlb.h                  |   4 +-
 arch/csky/include/asm/pgalloc.h               |   4 +-
 arch/hexagon/include/asm/pgalloc.h            |   8 +-
 arch/loongarch/include/asm/pgalloc.h          |   8 +-
 arch/m68k/include/asm/motorola_pgalloc.h      |  12 +-
 arch/m68k/include/asm/sun3_pgalloc.h          |   4 +-
 arch/microblaze/include/asm/pgalloc.h         |   2 +-
 arch/mips/include/asm/pgalloc.h               |   4 +-
 arch/nios2/include/asm/pgalloc.h              |   4 +-
 arch/openrisc/include/asm/pgalloc.h           |   8 +-
 arch/parisc/include/asm/pgalloc.h             |   2 +-
 arch/powerpc/include/asm/book3s/32/pgalloc.h  |   4 +-
 arch/powerpc/include/asm/book3s/64/hash-4k.h  |   4 +-
 arch/powerpc/include/asm/book3s/64/hash-64k.h |   4 +-
 arch/powerpc/include/asm/book3s/64/pgalloc.h  |   4 +-
 arch/powerpc/include/asm/book3s/64/pgtable.h  |   8 +-
 arch/powerpc/include/asm/book3s/64/radix.h    |   4 +-
 arch/powerpc/include/asm/pgalloc.h            |   8 +-
 arch/powerpc/mm/book3s64/hash_pgtable.c       |  10 +-
 arch/powerpc/mm/book3s64/radix_pgtable.c      |  10 +-
 arch/riscv/include/asm/pgalloc.h              |   8 +-
 arch/s390/include/asm/pgalloc.h               |   4 +-
 arch/s390/include/asm/pgtable.h               |   4 +-
 arch/s390/mm/pgalloc.c                        |   2 +-
 arch/s390/mm/pgtable.c                        |  14 +--
 arch/sh/include/asm/pgalloc.h                 |   4 +-
 arch/sparc/include/asm/pgalloc_32.h           |   6 +-
 arch/sparc/include/asm/pgalloc_64.h           |   2 +-
 arch/sparc/include/asm/pgtable_64.h           |   4 +-
 arch/sparc/mm/init_64.c                       |   2 +-
 arch/sparc/mm/srmmu.c                         |   6 +-
 arch/sparc/mm/tlb.c                           |  14 +--
 arch/x86/include/asm/pgalloc.h                |  10 +-
 arch/x86/mm/pgtable.c                         |   8 +-
 arch/xtensa/include/asm/pgalloc.h             |  12 +-
 fs/dax.c                                      |  14 +--
 include/asm-generic/pgalloc.h                 |  10 +-
 include/linux/mm.h                            |  16 ++-
 include/linux/mm_types.h                      |   4 +-
 include/linux/pgtable.h                       |   6 +-
 mm/debug_vm_pgtable.c                         |   6 +-
 mm/huge_memory.c                              | 103 +++++++++---------
 mm/internal.h                                 |   2 +-
 mm/khugepaged.c                               |  14 +--
 mm/memory.c                                   |  15 +--
 mm/mremap.c                                   |   2 +-
 mm/pgtable-generic.c                          |  37 +++----
 53 files changed, 240 insertions(+), 236 deletions(-)
#20 lore raw
In reply to [email protected]

Hi,

Le 30/07/2024 à 08:46, [email protected] a écrit :

Quoted 11 lines

From: Alex Shi <[email protected]>

We have struct ptdesc for page table descriptor a year ago, but it has no much usages in kernel, while pgtable_t is used widely.

The pgtable_t is typedefed as 'pte_t *' in sparc, s390, powerpc and m68k except SUN3, others archs are all same as 'struct page *'.

These blocks the conception and code update for page table descriptor to struct ptdesc.

Quoted 8 lines

So, the simple idea to push the ptdesc conception forward is to update all pgtable_t by ptdesc or pte_t pointer. But this needs widely knowledges for most all of different archs. Common code change is easy for include/ and mm/ directory, but it's hard in all archs.

Thanks for intel LKP framework, I fixed most all of build issues except a bug on powerpc which reports a "struct ptdesc *" incompatible with struct ptdesc *' pointer issue...

Can you tell more about that problem on powerpc ? Which defconfig for instance ?

Quoted 8 lines

Another trouble is pmd_pgtable() conversion in the last patch. Maybe some of arch need define theirself own pmd_ptdesc()?

This patchset is immature, even except above 2 issues, I just tested virutal machine booting and kselftest mm on x86 and arm64.

Anyway any input are appreciated!

Can you tell on which tree you based this series ? Last patch doesn't apply on 6.11-rc1:

Applying: mm/pgtable: pass ptdesc in pte_free_defer
error: sha1 information is lacking or useless (mm/khugepaged.c).
error: could not build fake ancestor
Patch failed at 0019 mm/pgtable: pass ptdesc in pte_free_defer

Following hunk is the problem:

diff a/mm/khugepaged.c b/mm/khugepaged.c (rejected hunks)
@@ -1777,7 +1777,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)
if (retracted) {
mm_dec_nr_ptes(mm); page_table_check_pte_clear_range(mm, addr, pgt_pmd);
- pte_free_defer(mm, pmd_pgtable(pgt_pmd));
+ pte_free_defer(mm, pmd_ptdesc(&pgt_pmd));
}
}
i_mmap_unlock_read(mapping);

Christophe

Quoted 79 lines

Thanks
Alex

Alex Shi (18):
mm/pgtable: use ptdesc in pte_free_now/pte_free_defer mm/pgtable: convert ptdesc.pmd_huge_pte to ptdesc pointer fs/dax: use ptdesc in dax_pmd_load_hole
mm/thp: use ptdesc pointer in __do_huge_pmd_anonymous_page mm/thp: use ptdesc in do_huge_pmd_anonymous_page
mm/thp: convert insert_pfn_pmd and its caller to use ptdesc mm/thp: use ptdesc in copy_huge_pmd
mm/memory: use ptdesc in __pte_alloc
mm/pgtable: fully use ptdesc in pte_alloc_one series functions mm/pgtable: pass ptdesc to pte_free()
mm/pgtable: introduce ptdesc_pfn and use ptdesc in free_pte_range() mm/thp: pass ptdesc to set_huge_zero_folio function mm/pgtable: return ptdesc pointer in pgtable_trans_huge_withdraw mm/pgtable: use ptdesc in pgtable_trans_huge_deposit mm/pgtable: pass ptdesc to pmd_populate
mm/pgtable: pass ptdesc to pmd_install
mm: convert vmf.prealloc_pte to struct ptdesc pointer mm/pgtable: pass ptdesc in pte_free_defer

  arch/alpha/include/asm/pgalloc.h              |   4 +-
  arch/arc/include/asm/pgalloc.h                |   4 +-
  arch/arm/include/asm/pgalloc.h                |  13 +--
  arch/arm/include/asm/tlb.h                    |   4 +-
  arch/arm/mm/pgd.c                             |   2 +-
  arch/arm64/include/asm/pgalloc.h              |   4 +-
  arch/arm64/include/asm/tlb.h                  |   4 +-
  arch/csky/include/asm/pgalloc.h               |   4 +-
  arch/hexagon/include/asm/pgalloc.h            |   8 +-
  arch/loongarch/include/asm/pgalloc.h          |   8 +-
  arch/m68k/include/asm/motorola_pgalloc.h      |  12 +-
  arch/m68k/include/asm/sun3_pgalloc.h          |   4 +-
  arch/microblaze/include/asm/pgalloc.h         |   2 +-
  arch/mips/include/asm/pgalloc.h               |   4 +-
  arch/nios2/include/asm/pgalloc.h              |   4 +-
  arch/openrisc/include/asm/pgalloc.h           |   8 +-
  arch/parisc/include/asm/pgalloc.h             |   2 +-
  arch/powerpc/include/asm/book3s/32/pgalloc.h  |   4 +-
  arch/powerpc/include/asm/book3s/64/hash-4k.h  |   4 +-
  arch/powerpc/include/asm/book3s/64/hash-64k.h |   4 +-
  arch/powerpc/include/asm/book3s/64/pgalloc.h  |   4 +-
  arch/powerpc/include/asm/book3s/64/pgtable.h  |   8 +-
  arch/powerpc/include/asm/book3s/64/radix.h    |   4 +-
  arch/powerpc/include/asm/pgalloc.h            |   8 +-
  arch/powerpc/mm/book3s64/hash_pgtable.c       |  10 +-
  arch/powerpc/mm/book3s64/radix_pgtable.c      |  10 +-
  arch/riscv/include/asm/pgalloc.h              |   8 +-
  arch/s390/include/asm/pgalloc.h               |   4 +-
  arch/s390/include/asm/pgtable.h               |   4 +-
  arch/s390/mm/pgalloc.c                        |   2 +-
  arch/s390/mm/pgtable.c                        |  14 +--
  arch/sh/include/asm/pgalloc.h                 |   4 +-
  arch/sparc/include/asm/pgalloc_32.h           |   6 +-
  arch/sparc/include/asm/pgalloc_64.h           |   2 +-
  arch/sparc/include/asm/pgtable_64.h           |   4 +-
  arch/sparc/mm/init_64.c                       |   2 +-
  arch/sparc/mm/srmmu.c                         |   6 +-
  arch/sparc/mm/tlb.c                           |  14 +--
  arch/x86/include/asm/pgalloc.h                |  10 +-
  arch/x86/mm/pgtable.c                         |   8 +-
  arch/xtensa/include/asm/pgalloc.h             |  12 +-
  fs/dax.c                                      |  14 +--
  include/asm-generic/pgalloc.h                 |  10 +-
  include/linux/mm.h                            |  16 ++-
  include/linux/mm_types.h                      |   4 +-
  include/linux/pgtable.h                       |   6 +-
  mm/debug_vm_pgtable.c                         |   6 +-
  mm/huge_memory.c                              | 103 +++++++++---------
  mm/internal.h                                 |   2 +-
  mm/khugepaged.c                               |  14 +--
  mm/memory.c                                   |  15 +--
  mm/mremap.c                                   |   2 +-
  mm/pgtable-generic.c                          |  37 +++----
  53 files changed, 240 insertions(+), 236 deletions(-)
#21 lore raw
Quoted 19 lines · LEROY Christophe

Hi,

Le 30/07/2024 à 08:46, [email protected] a écrit :

Quoted 16 lines

From: Alex Shi <[email protected]>

We have struct ptdesc for page table descriptor a year ago, but it has no much usages in kernel, while pgtable_t is used widely.

The pgtable_t is typedefed as 'pte_t *' in sparc, s390, powerpc and m68k except SUN3, others archs are all same as 'struct page *'.

These blocks the conception and code update for page table descriptor to struct ptdesc.

So, the simple idea to push the ptdesc conception forward is to update all pgtable_t by ptdesc or pte_t pointer. But this needs widely knowledges for most all of different archs. Common code change is easy for include/ and mm/ directory, but it's hard in all archs.

Quoted 7 lines
Quoted 3 lines

Thanks for intel LKP framework, I fixed most all of build issues except a bug on powerpc which reports a "struct ptdesc *" incompatible with struct ptdesc *' pointer issue...

Can you tell more about that problem on powerpc ? Which defconfig for instance ?

Hi Leroy,

Thanks a lot for response. The detailed error and code tree is from LKP:

Quoted 14 lines · kernel test robot
tree:   https://github.com/alexshi/linux.git ptdesc
head:   3735e634f8feb67c941d17222b53906ace2fd2b8
commit: 8ac2ea5f299444d065b292043aed0a1e94996159 [14/17] mm/pgtable: use ptdesc in pgtable_trans_huge_deposit
config: powerpc-allyesconfig (https://download.01.org/0day-ci/archive/20240803/[email protected]/config)
compiler: clang version 20.0.0git (https://github.com/llvm/llvm-project 423aec6573df4424f90555468128e17073ddc69e)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240803/[email protected]/reproduce)

In file included from arch/powerpc/include/asm/book3s/64/mmu-hash.h:20:

Quoted 6 lines
Quoted 1 line

arch/powerpc/include/asm/book3s/64/pgtable.h:1372:52: error: incompatible pointer types passing 'struct ptdesc *' to parameter of type 'struct ptdesc *' [-Werror,-Wincompatible-pointer-types]

Another trouble is pmd_pgtable() conversion in the last patch. Maybe some of arch need define theirself own pmd_ptdesc()?

Quoted 7 lines
Quoted 4 lines

This patchset is immature, even except above 2 issues, I just tested virutal machine booting and kselftest mm on x86 and arm64.

Anyway any input are appreciated!

Can you tell on which tree you based this series ? Last patch doesn't apply on 6.11-rc1:

The code based on akpm/mm-unstable tree, not upstream, you can check the code on:
https://github.com/alexshi/linux.git ptdesc

Thanks a lot!
Alex

maintainer
#22 lore raw
In reply to [email protected]
Quoted 2 lines · On Tue, Jul 30, 2024 at 02:46:54PM +0800, [email protected] wrote:

We have struct ptdesc for page table descriptor a year ago, but it has no much usages in kernel, while pgtable_t is used widely.

Hum, I thought I responded to this to point out the problem, but I don't see the response anywhere, so I'll try again.

Quoted 2 lines

The pgtable_t is typedefed as 'pte_t *' in sparc, s390, powerpc and m68k except SUN3, others archs are all same as 'struct page *'.

And there's a very good reason for that. On s390 and powerpc (I cannot speak to the sparc/m68k), each page table is (potentially) smaller than PAGE_SIZE. So we cannot do what your patch purports to do, as we would not know whether we're referring to the first or subsequent page tables contained within a page.

Maybe at some point in the distant future we'll be able to allocate a ptdesc per page table instead of per page allocated for use by page tables. But we cannot do that yet.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#23 lore raw
Quoted 10 lines · Matthew Wilcox
Quoted 2 lines · On Tue, Jul 30, 2024 at 02:46:54PM +0800, [email protected] wrote:

We have struct ptdesc for page table descriptor a year ago, but it has no much usages in kernel, while pgtable_t is used widely.

Hum, I thought I responded to this to point out the problem, but I don't see the response anywhere, so I'll try again.

Quoted 2 lines

The pgtable_t is typedefed as 'pte_t *' in sparc, s390, powerpc and m68k except SUN3, others archs are all same as 'struct page *'.

Quoted 8 lines

And there's a very good reason for that. On s390 and powerpc (I cannot speak to the sparc/m68k), each page table is (potentially) smaller than PAGE_SIZE. So we cannot do what your patch purports to do, as we would not know whether we're referring to the first or subsequent page tables contained within a page.

Quoted 1 line

Maybe at some point in the distant future we'll be able to allocate

a ptdesc per page table instead of per page allocated for use by page tables. But we cannot do that yet.

Got it. Thanks a lot for the info!