Extend and consolidate mmu_gather into new file
Subsystems (4)
v2 cover Extend and consolidate mmu_gather into new file
Hi all,
This series builds on the core changes I previously posted here:
rfc: http://lists.infradead.org/pipermail/linux-arm-kernel/2018-August/597821.html
v1: http://lists.infradead.org/pipermail/linux-arm-kernel/2018-August/598919.html
The main changes are:
* Move the mmu_gather bits out of memory.c and into their own file (looped in the mm people for this)
* Add a MAINTAINERS entry for the new file, and all tlb.h headers. If any mm developers would like to be included here as well, please just ask.
I'd like to queue these patches on their own branch in the arm64 git so that others can develop on top of them for the next merge window. Peter and Nick have both expressed an interest in that, and I already have a bunch of arm64 optimisations on top which I posted previously.
Cheers,
Will
--->8
Peter Zijlstra (2):
asm-generic/tlb: Track freeing of page-table directories in struct
mmu_gather
mm/memory: Move mmu_gather and TLB invalidation code into its own file
Will Deacon (3):
asm-generic/tlb: Guard with #ifdef CONFIG_MMU asm-generic/tlb: Track which levels of the page tables have been
cleared
MAINTAINERS: Add entry for MMU GATHER AND TLB INVALIDATION
MAINTAINERS | 12 +++ include/asm-generic/tlb.h | 86 ++++++++++++--- mm/Makefile | 6 +- mm/memory.c | 247 ------------------------------------------- mm/mmu_gather.c | 259 ++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 348 insertions(+), 262 deletions(-) create mode 100644 mm/mmu_gather.c
Signature · 1 line
2.1.4
PATCH v2 1/5 asm-generic/tlb: Guard with #ifdef CONFIG_MMU
The inner workings of the mmu_gather-based TLB invalidation mechanism are not relevant to nommu configurations, so guard them with an #ifdef. This allows us to implement future functions using static inlines without breaking the build.
--- include/asm-generic/tlb.h | 4 ++++ 1 file changed, 4 insertions(+)
include/asm-generic/tlb.h +4 -0
@@ -20,6 +20,8 @@#include <asm/pgalloc.h> #include <asm/tlbflush.h> +#ifdef CONFIG_MMU +#ifdef CONFIG_HAVE_RCU_TABLE_FREE /* * Semi RCU freeing of the page directories.
@@ -310,6 +312,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,#endif #endif +#endif /* CONFIG_MMU */ +#define tlb_migrate_finish (mm) do {} while (0)#endif /* _ASM_GENERIC__TLB_H */
Signature · 1 line
2.1.4
PATCH v2 2/5 asm-generic/tlb: Track freeing of page-table directories in struct mmu_gather
In reply to Will DeaconFrom: Peter Zijlstra <[email protected]>
Some architectures require different TLB invalidation instructions depending on whether it is only the last-level of page table being changed, or whether there are also changes to the intermediate (directory) entries higher up the tree.
Add a new bit to the flags bitfield in struct mmu_gather so that the architecture code can operate accordingly if it's the intermediate levels being invalidated.
--- include/asm-generic/tlb.h | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-)
include/asm-generic/tlb.h +23 -8
@@ -99,12 +99,22 @@ struct mmu_gather {#endif unsigned long start;unsigned long end; -/* we are in the middle of an operation to clear -* a full mm and can make some optimizations */ - unsigned int fullmm : 1, -/* we have performed an operation which -* requires a complete flush of the tlb */ - need_flush_all : 1; +/* +* we are in the middle of an operation to clear +* a full mm and can make some optimizations +*/ + unsigned int fullmm : 1; + +/* +* we have performed an operation which +* requires a complete flush of the tlb +*/ + unsigned int need_flush_all : 1; + +/* +* we have removed page directories +*/ + unsigned int freed_tables : 1;struct mmu_gather_batch *active;struct mmu_gather_batch local;
@@ -139,6 +149,7 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) tlb->start =TASK_SIZE ; tlb->end =0 ; } + tlb->freed_tables =0 ; }static inline void tlb_flush_mmu_tlbonly (struct mmu_gather *tlb)
@@ -280,6 +291,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,#define pte_free_tlb (tlb, ptep, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->freed_tables = 1; \ __pte_free_tlb(tlb, ptep, address); \ } while (0)#endif
@@ -287,7 +299,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,#ifndef pmd_free_tlb#define pmd_free_tlb (tlb, pmdp, address) \ do { \ - __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->freed_tables = 1; \ __pmd_free_tlb(tlb, pmdp, address); \ } while (0)#endif
@@ -297,6 +310,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,#define pud_free_tlb (tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->freed_tables = 1; \ __pud_free_tlb(tlb, pudp, address); \ } while (0)#endif
@@ -306,7 +320,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,#ifndef p4d_free_tlb#define p4d_free_tlb (tlb, pudp, address) \ do { \ - __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->freed_tables = 1; \ __p4d_free_tlb(tlb, pudp, address); \ } while (0)#endif
Signature · 1 line
2.1.4
PATCH v2 3/5 asm-generic/tlb: Track which levels of the page tables have been cleared
In reply to Will DeaconIt is common for architectures with hugepage support to require only a single TLB invalidation operation per hugepage during unmap(), rather than iterating through the mapping at a PAGE_SIZE increment. Currently, however, the level in the page table where the unmap() operation occurs is not stored in the mmu_gather structure, therefore forcing architectures to issue additional TLB invalidation operations or to give up and over-invalidate by e.g. invalidating the entire TLB.
Ideally, we could add an interval rbtree to the mmu_gather structure, which would allow us to associate the correct mapping granule with the various sub-mappings within the range being invalidated. However, this is costly in terms of book-keeping and memory management, so instead we approximate by keeping track of the page table levels that are cleared and provide a means to query the smallest granule required for invalidation.
--- include/asm-generic/tlb.h | 58 ++++++++++++++++++++++++++++++++++++++++------- mm/memory.c | 4 +++- 2 files changed, 53 insertions(+), 9 deletions(-)
include/asm-generic/tlb.h +50 -8
@@ -116,6 +116,14 @@ struct mmu_gather {*/ unsigned int freed_tables : 1; +/* +* at which levels have we cleared entries? +*/ + unsigned int cleared_ptes : 1; + unsigned int cleared_pmds : 1; + unsigned int cleared_puds : 1; + unsigned int cleared_p4ds : 1; +struct mmu_gather_batch *active;struct mmu_gather_batch local;struct page *__pages[MMU_GATHER_BUNDLE ];
@@ -150,6 +158,10 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) tlb->end =0 ; } tlb->freed_tables =0 ; + tlb->cleared_ptes =0 ; + tlb->cleared_pmds =0 ; + tlb->cleared_puds =0 ; + tlb->cleared_p4ds =0 ; }static inline void tlb_flush_mmu_tlbonly (struct mmu_gather *tlb)
@@ -199,6 +211,25 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, }#endif +static inline unsigned long tlb_get_unmap_shift (struct mmu_gather *tlb) +{ +if (tlb->cleared_ptes) +return PAGE_SHIFT ; +if (tlb->cleared_pmds) +return PMD_SHIFT ; +if (tlb->cleared_puds) +return PUD_SHIFT ; +if (tlb->cleared_p4ds) +return P4D_SHIFT ; + +return PAGE_SHIFT ; +} + +static inline unsigned long tlb_get_unmap_size (struct mmu_gather *tlb) +{ +return 1UL <<tlb_get_unmap_shift (tlb); +} +/* * In the case of tlb vma handling, we can optimise these away in the * case where we're doing a full MM flush. When we're doing a munmap,
@@ -232,13 +263,19 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,#define tlb_remove_tlb_entry (tlb, ptep, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->cleared_ptes = 1; \ __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) -#define tlb_remove_huge_tlb_entry (h, tlb, ptep, address) \ - do { \ - __tlb_adjust_range(tlb, address, huge_page_size(h)); \ - __tlb_remove_tlb_entry(tlb, ptep, address); \ +#define tlb_remove_huge_tlb_entry (h, tlb, ptep, address) \ + do { \ + unsigned long _sz = huge_page_size(h); \ + __tlb_adjust_range(tlb, address, _sz); \ + if (_sz == PMD_SIZE) \ + tlb->cleared_pmds = 1; \ + else if (_sz == PUD_SIZE) \ + tlb->cleared_puds = 1; \ + __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0)/**
@@ -252,6 +289,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,#define tlb_remove_pmd_tlb_entry (tlb, pmdp, address) \ do { \ __tlb_adjust_range(tlb, address, HPAGE_PMD_SIZE); \ + tlb->cleared_pmds = 1; \ __tlb_remove_pmd_tlb_entry(tlb, pmdp, address); \ } while (0)
@@ -266,6 +304,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,#define tlb_remove_pud_tlb_entry (tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, HPAGE_PUD_SIZE); \ + tlb->cleared_puds = 1; \ __tlb_remove_pud_tlb_entry(tlb, pudp, address); \ } while (0)
@@ -291,7 +330,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,#define pte_free_tlb (tlb, ptep, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ + tlb->cleared_pmds = 1; \ __pte_free_tlb(tlb, ptep, address); \ } while (0)#endif
@@ -300,7 +340,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,#define pmd_free_tlb (tlb, pmdp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ + tlb->cleared_puds = 1; \ __pmd_free_tlb(tlb, pmdp, address); \ } while (0)#endif
@@ -310,7 +351,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,#define pud_free_tlb (tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ + tlb->cleared_p4ds = 1; \ __pud_free_tlb(tlb, pudp, address); \ } while (0)#endif
@@ -321,7 +363,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,#define p4d_free_tlb (tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ __p4d_free_tlb(tlb, pudp, address); \ } while (0)#endif
mm/memory.c +3 -1
@@ -267,8 +267,10 @@ void arch_tlb_finish_mmu(struct mmu_gather *tlb, {struct mmu_gather_batch *batch, *next; -if (force) +if (force) { +__tlb_reset_range (tlb);__tlb_adjust_range (tlb, start, end - start); + }tlb_flush_mmu (tlb);
Signature · 1 line
2.1.4
PATCH v2 4/5 mm/memory: Move mmu_gather and TLB invalidation code into its own file
In reply to Will DeaconFrom: Peter Zijlstra <[email protected]>
In preparation for maintaining the mmu_gather code as its own entity, move the implementation out of memory.c and into its own file.
--- include/asm-generic/tlb.h | 1 + mm/Makefile | 6 +- mm/memory.c | 249 -------------------------------------------- mm/mmu_gather.c | 259 ++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 263 insertions(+), 252 deletions(-) create mode 100644 mm/mmu_gather.c
include/asm-generic/tlb.h +1 -0
@@ -138,6 +138,7 @@ void arch_tlb_gather_mmu(struct mmu_gather *tlb,void tlb_flush_mmu (struct mmu_gather *tlb);void arch_tlb_finish_mmu (struct mmu_gather *tlb,unsigned long start,unsigned long end,bool force); +void tlb_flush_mmu_free (struct mmu_gather *tlb);extern bool __tlb_remove_page_size (struct mmu_gather *tlb,struct page *page,int page_size);
mm/Makefile +3 -3
@@ -23,9 +23,9 @@ KCOV_INSTRUMENT_vmstat.o := n mmu-y := nommu.o mmu-$(CONFIG_MMU) := gup.o highmem.o memory.o mincore.o \ - mlock.o mmap.o mprotect.o mremap.o msync.o \ - page_vma_mapped.o pagewalk.o pgtable-generic.o \ - rmap.o vmalloc.o + mlock.o mmap.o mmu_gather.o mprotect.o mremap.o \ + msync.o page_vma_mapped.o pagewalk.o \ + pgtable-generic.o rmap.o vmalloc.o ifdef CONFIG_CROSS_MEMORY_ATTACH
mm/memory.c +0 -249
255 lines not on this page · raw message
mm/mmu_gather.c new file +259 -0
259 lines not on this page · raw message
Signature · 1 line
2.1.4
PATCH v2 5/5 MAINTAINERS: Add entry for MMU GATHER AND TLB INVALIDATION
In reply to Will DeaconWe recently had to debug a TLB invalidation problem on the munmap() path, which was made more difficult than necessary because:
(a) The MMU gather code had changed without people realising
(b) Many people subtly misunderstood the operation of the MMU gather code and its interactions with RCU and arch-specific TLB invalidation
(c) Untangling the intended behaviour involved educated guesswork and plenty of discussion
Hopefully, we can avoid getting into this mess again by designating a cross-arch group of people to look after this code. It is not intended that they will have a separate tree, but they at least provide a point of contact for anybody working in this area and can co-ordinate any proposed future changes to the internal API.
--- MAINTAINERS | 12 ++++++++++++ 1 file changed, 12 insertions(+)
MAINTAINERS +12 -0
@@ -9681,6 +9681,18 @@ S: Maintained F: arch/arm/boot/dts/mmp* F: arch/arm/mach-mmp/ +MMU GATHER AND TLB INVALIDATION +M: Will Deacon <[email protected]> +M: "Aneesh Kumar K.V" <[email protected]> +M: Nick Piggin <[email protected]> +M: Peter Zijlstra <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +F: arch/*/include/asm/tlb.h +F: include/asm-generic/tlb.h +F: mm/mmu_gather.c + MN88472 MEDIA DRIVER M: Antti Palosaari <[email protected]> L: [email protected]
Signature · 1 line
2.1.4
Andrew Morton
maintainerQuoted 10 lines · On Tue, 4 Sep 2018 12:45:28 +0100 Will Deacon <[email protected]> wrote:
This series builds on the core changes I previously posted here:
rfc: http://lists.infradead.org/pipermail/linux-arm-kernel/2018-August/597821.html
v1: http://lists.infradead.org/pipermail/linux-arm-kernel/2018-August/598919.htmlThe main changes are:
* Move the
mmu_gatherbits out ofmemory.cand into their own file (looped in the mm people for this)
Quoted 8 lines
* Add a
MAINTAINERSentry for the new file, and alltlb.hheaders. If any mm developers would like to be included here as well, please just ask.I'd like to queue these patches on their own branch in the arm64 git so that others can develop on top of them for the next merge window. Peter and Nick have both expressed an interest in that, and I already have a bunch of arm64 optimisations on top which I posted previously.
All looks good to me - please proceed that way. Please also add me to the MAINTAINERS record so I get more emails.
Quoted 16 lines · Andrew Morton
Quoted 14 lines · On Tue, 4 Sep 2018 12:45:28 +0100 Will Deacon <[email protected]> wrote:
This series builds on the core changes I previously posted here:
rfc: http://lists.infradead.org/pipermail/linux-arm-kernel/2018-August/597821.html
v1: http://lists.infradead.org/pipermail/linux-arm-kernel/2018-August/598919.htmlThe main changes are:
* Move the
mmu_gatherbits out ofmemory.cand into their own file (looped in the mm people for this)* Add a
MAINTAINERSentry for the new file, and alltlb.hheaders. If any mm developers would like to be included here as well, please just ask.
Quoted 7 lines
Quoted 4 lines
I'd like to queue these patches on their own branch in the arm64 git so that others can develop on top of them for the next merge window. Peter and Nick have both expressed an interest in that, and I already have a bunch of arm64 optimisations on top which I posted previously.
All looks good to me - please proceed that way. Please also add me to the
MAINTAINERSrecord so I get more emails.
Cheers, Andrew. I'll add you to the MAINTAINERS entry and get this lot into -next once the kbuild robot is happy that I've got all the header files right.
Will