mm/mmu_gather: define RCU version tlb_remove_table_one() in CONFIG_MMU_GATHER_RCU_TABLE_FREE
PATCH 1/1 mm/mmu_gather: define RCU version tlb_remove_table_one() in CONFIG_MMU_GATHER_RCU_TABLE_FREE
Before commit 718b13861d22 ("x86: mm: free page table pages by RCU instead of semi RCU"), the page table pages will be freed by semi RCU, that is:
- batch table freeing: asynchronous free by RCU
- single table freeing: IPI + synchronous free
This commit introduce a RCU version for single table freeing to support PT_RECLAIM. While it is not necessary to limit the RCU version only in CONFIG_PT_RECLAIM. It is reasonable to asynchronous free single table by RCU if CONFIG_MMU_GATHER_RCU_TABLE_FREE.
This patch moves the definition of single table freeing in CONFIG_MMU_GATHER_RCU_TABLE_FREE, so we have both RCU version batch/single table freeing if CONFIG_MMU_GATHER_RCU_TABLE_FREE.
--- mm/mmu_gather.c | 51 +++++++++++++++++++++---------------------------- 1 file changed, 22 insertions(+), 29 deletions(-)
mm/mmu_gather.c +22 -29
@@ -296,6 +296,22 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)call_rcu (&batch->rcu, tlb_remove_table_rcu); } +static inline void __tlb_remove_table_one_rcu (struct rcu_head *head) +{ +struct ptdesc *ptdesc; + + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); +__tlb_remove_table (ptdesc); +} + +static inline void tlb_remove_table_one (void *table) +{ +struct ptdesc *ptdesc; + + ptdesc = table; +call_rcu (&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); +} +#else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */ static void tlb_remove_table_free (struct mmu_table_batch *batch)
@@ -303,6 +319,12 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch) __tlb_remove_table_free(batch); } +static inline void tlb_remove_table_one (void *table) +{ +tlb_remove_table_sync_one (); +__tlb_remove_table (table); +} +#endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ /*
@@ -320,35 +342,6 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb) } } -#ifdef CONFIG_PT_RECLAIM -static inline void __tlb_remove_table_one_rcu (struct rcu_head *head) -{ -struct ptdesc *ptdesc; - - ptdesc = container_of(head, struct ptdesc, pt_rcu_head); -__tlb_remove_table (ptdesc); -} - -static inline void __tlb_remove_table_one (void *table) -{ -struct ptdesc *ptdesc; - - ptdesc = table; -call_rcu (&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); -} -#else -static inline void __tlb_remove_table_one (void *table) -{ -tlb_remove_table_sync_one (); -__tlb_remove_table (table); -} -#endif /* CONFIG_PT_RECLAIM */ - -static void tlb_remove_table_one (void *table) -{ -__tlb_remove_table_one (table); -} -static void tlb_table_flush (struct mmu_gather *tlb) {struct mmu_table_batch **batch = &tlb->batch;
Signature · 1 line
2.34.1
Andrew Morton
maintainerQuoted 7 lines · On Sun, 22 Feb 2026 00:45:12 +0000 Wei Yang <[email protected]> wrote:
Before commit
718b13861d22("x86: mm: free page table pages by RCU instead of semi RCU"), the page table pages will be freed by semi RCU, that is:- batch table freeing: asynchronous free by RCU
- single table freeing: IPI + synchronous free
Quoted 8 lines
This commit introduce a RCU version for single table freeing to support
PT_RECLAIM. While it is not necessary to limit the RCU version only inCONFIG_PT_RECLAIM. It is reasonable to asynchronous free single table by RCU ifCONFIG_MMU_GATHER_RCU_TABLE_FREE.This patch moves the definition of single table freeing in
CONFIG_MMU_GATHER_RCU_TABLE_FREE, so we have both RCU version batch/single table freeing ifCONFIG_MMU_GATHER_RCU_TABLE_FREE.
What are the expected (and measured!) benefits of this?
Quoted 14 lines · Andrew Morton
Quoted 12 lines · On Sun, 22 Feb 2026 00:45:12 +0000 Wei Yang <[email protected]> wrote:
Before commit
718b13861d22("x86: mm: free page table pages by RCU instead of semi RCU"), the page table pages will be freed by semi RCU, that is:- batch table freeing: asynchronous free by RCU
- single table freeing: IPI + synchronous freeThis commit introduce a RCU version for single table freeing to support
PT_RECLAIM. While it is not necessary to limit the RCU version only inCONFIG_PT_RECLAIM. It is reasonable to asynchronous free single table by RCU ifCONFIG_MMU_GATHER_RCU_TABLE_FREE.
Quoted 6 lines
Quoted 3 lines
This patch moves the definition of single table freeing in
CONFIG_MMU_GATHER_RCU_TABLE_FREE, so we have both RCU version batch/single table freeing ifCONFIG_MMU_GATHER_RCU_TABLE_FREE.What are the expected (and measured!) benefits of this?
No, this is just a cleanup.
Last time checked with Qi and thought it worth a formal patch. [1]
[1]: http://lkml.kernel.org/r/[email protected]
Signature · 2 lines
Wei Yang Help you, Help me
Hi Wei,
Quoted 7 lines · Wei Yang
Before commit
718b13861d22("x86: mm: free page table pages by RCU instead of semi RCU"), the page table pages will be freed by semi RCU, that is:- batch table freeing: asynchronous free by RCU
- single table freeing: IPI + synchronous free
Quoted 8 lines
This commit introduce a RCU version for single table freeing to support
PT_RECLAIM. While it is not necessary to limit the RCU version only inCONFIG_PT_RECLAIM. It is reasonable to asynchronous free single table by RCU ifCONFIG_MMU_GATHER_RCU_TABLE_FREE.This patch moves the definition of single table freeing in
CONFIG_MMU_GATHER_RCU_TABLE_FREE, so we have both RCU version batch/single table freeing ifCONFIG_MMU_GATHER_RCU_TABLE_FREE.
Thank you for doing this, but we can't do it right now.
See the latest version: https://github.com/torvalds/linux/commit/9c8c02df3f8742f6db927e787ab971fd0b5ac08a, when the HAVE_ARCH_TLB_REMOVE_TABLE is selected, the pt_rcu_head member
cannot be used.
Thanks,
Qi
Quoted 84 lines
Signed-off-by Wei Yang <[email protected]>Cc Qi Zheng <[email protected]>Cc "David Hildenbrand (Arm)" <[email protected]>--- mm/mmu_gather.c | 51 +++++++++++++++++++++---------------------------- 1 file changed, 22 insertions(+), 29 deletions(-)mm/mmu_gather.c +22
-29@@ -296,6 +296,22 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)call_rcu (&batch->rcu, tlb_remove_table_rcu); } +static inline void __tlb_remove_table_one_rcu (struct rcu_head *head) +{ +struct ptdesc *ptdesc; + + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); +__tlb_remove_table (ptdesc); +} + +static inline void tlb_remove_table_one (void *table) +{ +struct ptdesc *ptdesc; + + ptdesc = table; +call_rcu (&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); +} +#else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */ static void tlb_remove_table_free (struct mmu_table_batch *batch)@@ -303,6 +319,12 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch) __tlb_remove_table_free(batch); } +static inline void tlb_remove_table_one (void *table) +{ +tlb_remove_table_sync_one (); +__tlb_remove_table (table); +} +#endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ /* @@ -320,35 +342,6 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb) } } -#ifdef CONFIG_PT_RECLAIM -static inline void __tlb_remove_table_one_rcu (struct rcu_head *head) -{ -struct ptdesc *ptdesc; - - ptdesc = container_of(head, struct ptdesc, pt_rcu_head); -__tlb_remove_table (ptdesc); -} - -static inline void __tlb_remove_table_one (void *table) -{ -struct ptdesc *ptdesc; - - ptdesc = table; -call_rcu (&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); -} -#else -static inline void __tlb_remove_table_one (void *table) -{ -tlb_remove_table_sync_one (); -__tlb_remove_table (table); -} -#endif /* CONFIG_PT_RECLAIM */ - -static void tlb_remove_table_one (void *table) -{ -__tlb_remove_table_one (table); -} -static void tlb_table_flush (struct mmu_gather *tlb) {struct mmu_table_batch **batch = &tlb->batch;
Quoted 19 lines · Qi Zheng
Hi Wei,
Quoted 7 lines · Wei Yang
Before commit
718b13861d22("x86: mm: free page table pages by RCU instead of semi RCU"), the page table pages will be freed by semi RCU, that is:- batch table freeing: asynchronous free by RCU
- single table freeing: IPI + synchronous freeQuoted 8 lines
This commit introduce a RCU version for single table freeing to support
PT_RECLAIM. While it is not necessary to limit the RCU version only inCONFIG_PT_RECLAIM. It is reasonable to asynchronous free single table by RCU ifCONFIG_MMU_GATHER_RCU_TABLE_FREE.This patch moves the definition of single table freeing in
CONFIG_MMU_GATHER_RCU_TABLE_FREE, so we have both RCU version batch/single table freeing ifCONFIG_MMU_GATHER_RCU_TABLE_FREE.
Quoted 7 lines
Thank you for doing this, but we can't do it right now.
See the latest version: https://github.com/torvalds/linux/commit/9c8c02df3f8742f6db927e787ab971fd0b5ac08a,
when theHAVE_ARCH_TLB_REMOVE_TABLEis selected, thept_rcu_headmember
cannot be used.
Thanks for point out. Don't notice the latest change.
Andrew,
Sorry for the bothering. Please drop this one.
Signature · 2 lines
Wei Yang Help you, Help me