Skip to contentSkip to conversation

mm/mmu_gather: define RCU version tlb_remove_table_one() in CONFIG_MMU_GATHER_RCU_TABLE_FREE

PATCH In review Wei Yang 3 participants 5 messages Feb 22 – Feb 24

Subsystems (3)
lore
Share & tools
lore

Wei Yang

author
#0 lore raw

PATCH 1/1 mm/mmu_gather: define RCU version tlb_remove_table_one() in CONFIG_MMU_GATHER_RCU_TABLE_FREE

Before commit 718b13861d22 ("x86: mm: free page table pages by RCU instead of semi RCU"), the page table pages will be freed by semi RCU, that is:

- batch table freeing: asynchronous free by RCU
- single table freeing: IPI + synchronous free

This commit introduce a RCU version for single table freeing to support PT_RECLAIM. While it is not necessary to limit the RCU version only in CONFIG_PT_RECLAIM. It is reasonable to asynchronous free single table by RCU if CONFIG_MMU_GATHER_RCU_TABLE_FREE.

This patch moves the definition of single table freeing in CONFIG_MMU_GATHER_RCU_TABLE_FREE, so we have both RCU version batch/single table freeing if CONFIG_MMU_GATHER_RCU_TABLE_FREE.

Cc Qi Zheng <[email protected]>
Cc "David Hildenbrand (Arm)" <[email protected]>
---
 mm/mmu_gather.c | 51 +++++++++++++++++++++----------------------------
 1 file changed, 22 insertions(+), 29 deletions(-)
mm/mmu_gather.c +22 -29
index fe5b6a031717..6c1cbd9c717b 100644
@@ -296,6 +296,22 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
 	call_rcu(&batch->rcu, tlb_remove_table_rcu);
 }
 
+static inline void __tlb_remove_table_one_rcu(struct rcu_head *head)
+{
+	struct ptdesc *ptdesc;
+
+	ptdesc = container_of(head, struct ptdesc, pt_rcu_head);
+	__tlb_remove_table(ptdesc);
+}
+
+static inline void tlb_remove_table_one(void *table)
+{
+	struct ptdesc *ptdesc;
+
+	ptdesc = table;
+	call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu);
+}
+
 #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
 static void tlb_remove_table_free(struct mmu_table_batch *batch)
@@ -303,6 +319,12 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
 	__tlb_remove_table_free(batch);
 }
 
+static inline void tlb_remove_table_one(void *table)
+{
+	tlb_remove_table_sync_one();
+	__tlb_remove_table(table);
+}
+
 #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
 /*
@@ -320,35 +342,6 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb)
 	}
 }
 
-#ifdef CONFIG_PT_RECLAIM
-static inline void __tlb_remove_table_one_rcu(struct rcu_head *head)
-{
-	struct ptdesc *ptdesc;
-
-	ptdesc = container_of(head, struct ptdesc, pt_rcu_head);
-	__tlb_remove_table(ptdesc);
-}
-
-static inline void __tlb_remove_table_one(void *table)
-{
-	struct ptdesc *ptdesc;
-
-	ptdesc = table;
-	call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu);
-}
-#else
-static inline void __tlb_remove_table_one(void *table)
-{
-	tlb_remove_table_sync_one();
-	__tlb_remove_table(table);
-}
-#endif /* CONFIG_PT_RECLAIM */
-
-static void tlb_remove_table_one(void *table)
-{
-	__tlb_remove_table_one(table);
-}
-
 static void tlb_table_flush(struct mmu_gather *tlb)
 {
 	struct mmu_table_batch **batch = &tlb->batch;
Signature · 1 line
2.34.1
maintainer
#1 lore raw
Quoted 7 lines · On Sun, 22 Feb 2026 00:45:12 +0000 Wei Yang <[email protected]> wrote:

Before commit 718b13861d22 ("x86: mm: free page table pages by RCU instead of semi RCU"), the page table pages will be freed by semi RCU, that is:

- batch table freeing: asynchronous free by RCU
- single table freeing: IPI + synchronous free

Quoted 8 lines

This commit introduce a RCU version for single table freeing to support PT_RECLAIM. While it is not necessary to limit the RCU version only in CONFIG_PT_RECLAIM. It is reasonable to asynchronous free single table by RCU if CONFIG_MMU_GATHER_RCU_TABLE_FREE.

This patch moves the definition of single table freeing in CONFIG_MMU_GATHER_RCU_TABLE_FREE, so we have both RCU version batch/single table freeing if CONFIG_MMU_GATHER_RCU_TABLE_FREE.

What are the expected (and measured!) benefits of this?

author
#2 lore raw
Quoted 14 lines · Andrew Morton
Quoted 12 lines · On Sun, 22 Feb 2026 00:45:12 +0000 Wei Yang <[email protected]> wrote:

Before commit 718b13861d22 ("x86: mm: free page table pages by RCU instead of semi RCU"), the page table pages will be freed by semi RCU, that is:

- batch table freeing: asynchronous free by RCU
- single table freeing: IPI + synchronous free

This commit introduce a RCU version for single table freeing to support PT_RECLAIM. While it is not necessary to limit the RCU version only in CONFIG_PT_RECLAIM. It is reasonable to asynchronous free single table by RCU if CONFIG_MMU_GATHER_RCU_TABLE_FREE.

Quoted 6 lines
Quoted 3 lines

This patch moves the definition of single table freeing in CONFIG_MMU_GATHER_RCU_TABLE_FREE, so we have both RCU version batch/single table freeing if CONFIG_MMU_GATHER_RCU_TABLE_FREE.

What are the expected (and measured!) benefits of this?

No, this is just a cleanup.

Last time checked with Qi and thought it worth a formal patch. [1]

[1]: http://lkml.kernel.org/r/[email protected]

Signature · 2 lines
Wei Yang
Help you, Help me
#3 lore raw
In reply to Wei Yang

Hi Wei,

Quoted 7 lines · Wei Yang

Before commit 718b13861d22 ("x86: mm: free page table pages by RCU instead of semi RCU"), the page table pages will be freed by semi RCU, that is:

- batch table freeing: asynchronous free by RCU
- single table freeing: IPI + synchronous free

Quoted 8 lines

This commit introduce a RCU version for single table freeing to support PT_RECLAIM. While it is not necessary to limit the RCU version only in CONFIG_PT_RECLAIM. It is reasonable to asynchronous free single table by RCU if CONFIG_MMU_GATHER_RCU_TABLE_FREE.

This patch moves the definition of single table freeing in CONFIG_MMU_GATHER_RCU_TABLE_FREE, so we have both RCU version batch/single table freeing if CONFIG_MMU_GATHER_RCU_TABLE_FREE.

Thank you for doing this, but we can't do it right now.

See the latest version: https://github.com/torvalds/linux/commit/9c8c02df3f8742f6db927e787ab971fd0b5ac08a, when the HAVE_ARCH_TLB_REMOVE_TABLE is selected, the pt_rcu_head member
cannot be used.

Thanks,
Qi

Quoted 84 lines
Cc Qi Zheng <[email protected]>
Cc "David Hildenbrand (Arm)" <[email protected]>
---
  mm/mmu_gather.c | 51 +++++++++++++++++++++----------------------------
  1 file changed, 22 insertions(+), 29 deletions(-)
mm/mmu_gather.c +22 -29
index fe5b6a031717..6c1cbd9c717b 100644
@@ -296,6 +296,22 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
  	call_rcu(&batch->rcu, tlb_remove_table_rcu);
  }
 
+static inline void __tlb_remove_table_one_rcu(struct rcu_head *head)
+{
+	struct ptdesc *ptdesc;
+
+	ptdesc = container_of(head, struct ptdesc, pt_rcu_head);
+	__tlb_remove_table(ptdesc);
+}
+
+static inline void tlb_remove_table_one(void *table)
+{
+	struct ptdesc *ptdesc;
+
+	ptdesc = table;
+	call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu);
+}
+
  #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
  static void tlb_remove_table_free(struct mmu_table_batch *batch)
@@ -303,6 +319,12 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
  	__tlb_remove_table_free(batch);
  }
 
+static inline void tlb_remove_table_one(void *table)
+{
+	tlb_remove_table_sync_one();
+	__tlb_remove_table(table);
+}
+
  #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
  /*
@@ -320,35 +342,6 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb)
  	}
  }
 
-#ifdef CONFIG_PT_RECLAIM
-static inline void __tlb_remove_table_one_rcu(struct rcu_head *head)
-{
-	struct ptdesc *ptdesc;
-
-	ptdesc = container_of(head, struct ptdesc, pt_rcu_head);
-	__tlb_remove_table(ptdesc);
-}
-
-static inline void __tlb_remove_table_one(void *table)
-{
-	struct ptdesc *ptdesc;
-
-	ptdesc = table;
-	call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu);
-}
-#else
-static inline void __tlb_remove_table_one(void *table)
-{
-	tlb_remove_table_sync_one();
-	__tlb_remove_table(table);
-}
-#endif /* CONFIG_PT_RECLAIM */
-
-static void tlb_remove_table_one(void *table)
-{
-	__tlb_remove_table_one(table);
-}
-
  static void tlb_table_flush(struct mmu_gather *tlb)
  {
  	struct mmu_table_batch **batch = &tlb->batch;
author
#4 lore raw
Quoted 19 lines · Qi Zheng

Hi Wei,

Quoted 7 lines · Wei Yang

Before commit 718b13861d22 ("x86: mm: free page table pages by RCU instead of semi RCU"), the page table pages will be freed by semi RCU, that is:

- batch table freeing: asynchronous free by RCU
- single table freeing: IPI + synchronous free

Quoted 8 lines

This commit introduce a RCU version for single table freeing to support PT_RECLAIM. While it is not necessary to limit the RCU version only in CONFIG_PT_RECLAIM. It is reasonable to asynchronous free single table by RCU if CONFIG_MMU_GATHER_RCU_TABLE_FREE.

This patch moves the definition of single table freeing in CONFIG_MMU_GATHER_RCU_TABLE_FREE, so we have both RCU version batch/single table freeing if CONFIG_MMU_GATHER_RCU_TABLE_FREE.

Quoted 7 lines

Thank you for doing this, but we can't do it right now.

See the latest version: https://github.com/torvalds/linux/commit/9c8c02df3f8742f6db927e787ab971fd0b5ac08a,
when the
HAVE_ARCH_TLB_REMOVE_TABLE is selected, the pt_rcu_head member
cannot be used.

Thanks for point out. Don't notice the latest change.

Andrew,

Sorry for the bothering. Please drop this one.

Signature · 2 lines
Wei Yang
Help you, Help me