Skip to contentSkip to conversation

Extend and consolidate mmu_gather into new file

PATCH In review Will Deacon 2 participants 8 messages Sep 4, 2018 – Sep 5, 2018

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v2 cover Extend and consolidate mmu_gather into new file

Hi all,

This series builds on the core changes I previously posted here:

rfc: http://lists.infradead.org/pipermail/linux-arm-kernel/2018-August/597821.html
v1: http://lists.infradead.org/pipermail/linux-arm-kernel/2018-August/598919.html

The main changes are:

* Move the mmu_gather bits out of memory.c and into their own file (looped in the mm people for this)

* Add a MAINTAINERS entry for the new file, and all tlb.h headers. If any mm developers would like to be included here as well, please just ask.

I'd like to queue these patches on their own branch in the arm64 git so that others can develop on top of them for the next merge window. Peter and Nick have both expressed an interest in that, and I already have a bunch of arm64 optimisations on top which I posted previously.

Cheers,

Will

--->8

Peter Zijlstra (2):
asm-generic/tlb: Track freeing of page-table directories in struct
mmu_gather
mm/memory: Move mmu_gather and TLB invalidation code into its own file

Will Deacon (3):
asm-generic/tlb: Guard with #ifdef CONFIG_MMU asm-generic/tlb: Track which levels of the page tables have been
cleared
MAINTAINERS: Add entry for MMU GATHER AND TLB INVALIDATION

 MAINTAINERS               |  12 +++
 include/asm-generic/tlb.h |  86 ++++++++++++---
 mm/Makefile               |   6 +-
 mm/memory.c               | 247 -------------------------------------------
 mm/mmu_gather.c           | 259 ++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 348 insertions(+), 262 deletions(-)
 create mode 100644 mm/mmu_gather.c
Signature · 1 line
2.1.4
#1 lore raw

PATCH v2 1/5 asm-generic/tlb: Guard with #ifdef CONFIG_MMU

The inner workings of the mmu_gather-based TLB invalidation mechanism are not relevant to nommu configurations, so guard them with an #ifdef. This allows us to implement future functions using static inlines without breaking the build.

---
 include/asm-generic/tlb.h | 4 ++++
 1 file changed, 4 insertions(+)
include/asm-generic/tlb.h +4 -0
index b3353e21f3b3..a25e236f7a7f 100644
@@ -20,6 +20,8 @@
 #include <asm/pgalloc.h>
 #include <asm/tlbflush.h>
 
+#ifdef CONFIG_MMU
+
 #ifdef CONFIG_HAVE_RCU_TABLE_FREE
 /*
  * Semi RCU freeing of the page directories.
@@ -310,6 +312,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,
 #endif
 #endif
 
+#endif /* CONFIG_MMU */
+
 #define tlb_migrate_finish(mm) do {} while (0)
 
 #endif /* _ASM_GENERIC__TLB_H */
Signature · 1 line
2.1.4
#2 lore raw

PATCH v2 2/5 asm-generic/tlb: Track freeing of page-table directories in struct mmu_gather

In reply to Will Deacon

From: Peter Zijlstra <[email protected]>

Some architectures require different TLB invalidation instructions depending on whether it is only the last-level of page table being changed, or whether there are also changes to the intermediate (directory) entries higher up the tree.

Add a new bit to the flags bitfield in struct mmu_gather so that the architecture code can operate accordingly if it's the intermediate levels being invalidated.

---
 include/asm-generic/tlb.h | 31 +++++++++++++++++++++++--------
 1 file changed, 23 insertions(+), 8 deletions(-)
include/asm-generic/tlb.h +23 -8
index a25e236f7a7f..2b444ad94566 100644
@@ -99,12 +99,22 @@ struct mmu_gather {
 #endif
 	unsigned long		start;
 	unsigned long		end;
-	/* we are in the middle of an operation to clear
-	 * a full mm and can make some optimizations */
-	unsigned int		fullmm : 1,
-	/* we have performed an operation which
-	 * requires a complete flush of the tlb */
-				need_flush_all : 1;
+	/*
+	 * we are in the middle of an operation to clear
+	 * a full mm and can make some optimizations
+	 */
+	unsigned int		fullmm : 1;
+
+	/*
+	 * we have performed an operation which
+	 * requires a complete flush of the tlb
+	 */
+	unsigned int		need_flush_all : 1;
+
+	/*
+	 * we have removed page directories
+	 */
+	unsigned int		freed_tables : 1;
 
 	struct mmu_gather_batch *active;
 	struct mmu_gather_batch	local;
@@ -139,6 +149,7 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb)
 		tlb->start = TASK_SIZE;
 		tlb->end = 0;
 	}
+	tlb->freed_tables = 0;
 }
 
 static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb)
@@ -280,6 +291,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,
 #define pte_free_tlb(tlb, ptep, address)			\
 	do {							\
 		__tlb_adjust_range(tlb, address, PAGE_SIZE);	\
+		tlb->freed_tables = 1;			\
 		__pte_free_tlb(tlb, ptep, address);		\
 	} while (0)
 #endif
@@ -287,7 +299,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,
 #ifndef pmd_free_tlb
 #define pmd_free_tlb(tlb, pmdp, address)			\
 	do {							\
-		__tlb_adjust_range(tlb, address, PAGE_SIZE);		\
+		__tlb_adjust_range(tlb, address, PAGE_SIZE);	\
+		tlb->freed_tables = 1;			\
 		__pmd_free_tlb(tlb, pmdp, address);		\
 	} while (0)
 #endif
@@ -297,6 +310,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,
 #define pud_free_tlb(tlb, pudp, address)			\
 	do {							\
 		__tlb_adjust_range(tlb, address, PAGE_SIZE);	\
+		tlb->freed_tables = 1;			\
 		__pud_free_tlb(tlb, pudp, address);		\
 	} while (0)
 #endif
@@ -306,7 +320,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,
 #ifndef p4d_free_tlb
 #define p4d_free_tlb(tlb, pudp, address)			\
 	do {							\
-		__tlb_adjust_range(tlb, address, PAGE_SIZE);		\
+		__tlb_adjust_range(tlb, address, PAGE_SIZE);	\
+		tlb->freed_tables = 1;			\
 		__p4d_free_tlb(tlb, pudp, address);		\
 	} while (0)
 #endif
Signature · 1 line
2.1.4
#3 lore raw

PATCH v2 3/5 asm-generic/tlb: Track which levels of the page tables have been cleared

In reply to Will Deacon

It is common for architectures with hugepage support to require only a single TLB invalidation operation per hugepage during unmap(), rather than iterating through the mapping at a PAGE_SIZE increment. Currently, however, the level in the page table where the unmap() operation occurs is not stored in the mmu_gather structure, therefore forcing architectures to issue additional TLB invalidation operations or to give up and over-invalidate by e.g. invalidating the entire TLB.

Ideally, we could add an interval rbtree to the mmu_gather structure, which would allow us to associate the correct mapping granule with the various sub-mappings within the range being invalidated. However, this is costly in terms of book-keeping and memory management, so instead we approximate by keeping track of the page table levels that are cleared and provide a means to query the smallest granule required for invalidation.

---
 include/asm-generic/tlb.h | 58 ++++++++++++++++++++++++++++++++++++++++-------
 mm/memory.c               |  4 +++-
 2 files changed, 53 insertions(+), 9 deletions(-)
include/asm-generic/tlb.h +50 -8
index 2b444ad94566..9791e98122a0 100644
@@ -116,6 +116,14 @@ struct mmu_gather {
 	 */
 	unsigned int		freed_tables : 1;
 
+	/*
+	 * at which levels have we cleared entries?
+	 */
+	unsigned int		cleared_ptes : 1;
+	unsigned int		cleared_pmds : 1;
+	unsigned int		cleared_puds : 1;
+	unsigned int		cleared_p4ds : 1;
+
 	struct mmu_gather_batch *active;
 	struct mmu_gather_batch	local;
 	struct page		*__pages[MMU_GATHER_BUNDLE];
@@ -150,6 +158,10 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb)
 		tlb->end = 0;
 	}
 	tlb->freed_tables = 0;
+	tlb->cleared_ptes = 0;
+	tlb->cleared_pmds = 0;
+	tlb->cleared_puds = 0;
+	tlb->cleared_p4ds = 0;
 }
 
 static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb)
@@ -199,6 +211,25 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,
 }
 #endif
 
+static inline unsigned long tlb_get_unmap_shift(struct mmu_gather *tlb)
+{
+	if (tlb->cleared_ptes)
+		return PAGE_SHIFT;
+	if (tlb->cleared_pmds)
+		return PMD_SHIFT;
+	if (tlb->cleared_puds)
+		return PUD_SHIFT;
+	if (tlb->cleared_p4ds)
+		return P4D_SHIFT;
+
+	return PAGE_SHIFT;
+}
+
+static inline unsigned long tlb_get_unmap_size(struct mmu_gather *tlb)
+{
+	return 1UL << tlb_get_unmap_shift(tlb);
+}
+
 /*
  * In the case of tlb vma handling, we can optimise these away in the
  * case where we're doing a full MM flush.  When we're doing a munmap,
@@ -232,13 +263,19 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,
 #define tlb_remove_tlb_entry(tlb, ptep, address)		\
 	do {							\
 		__tlb_adjust_range(tlb, address, PAGE_SIZE);	\
+		tlb->cleared_ptes = 1;				\
 		__tlb_remove_tlb_entry(tlb, ptep, address);	\
 	} while (0)
 
-#define tlb_remove_huge_tlb_entry(h, tlb, ptep, address)	     \
-	do {							     \
-		__tlb_adjust_range(tlb, address, huge_page_size(h)); \
-		__tlb_remove_tlb_entry(tlb, ptep, address);	     \
+#define tlb_remove_huge_tlb_entry(h, tlb, ptep, address)	\
+	do {							\
+		unsigned long _sz = huge_page_size(h);		\
+		__tlb_adjust_range(tlb, address, _sz);		\
+		if (_sz == PMD_SIZE)				\
+			tlb->cleared_pmds = 1;			\
+		else if (_sz == PUD_SIZE)			\
+			tlb->cleared_puds = 1;			\
+		__tlb_remove_tlb_entry(tlb, ptep, address);	\
 	} while (0)
 
 /**
@@ -252,6 +289,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,
 #define tlb_remove_pmd_tlb_entry(tlb, pmdp, address)			\
 	do {								\
 		__tlb_adjust_range(tlb, address, HPAGE_PMD_SIZE);	\
+		tlb->cleared_pmds = 1;					\
 		__tlb_remove_pmd_tlb_entry(tlb, pmdp, address);		\
 	} while (0)
 
@@ -266,6 +304,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,
 #define tlb_remove_pud_tlb_entry(tlb, pudp, address)			\
 	do {								\
 		__tlb_adjust_range(tlb, address, HPAGE_PUD_SIZE);	\
+		tlb->cleared_puds = 1;					\
 		__tlb_remove_pud_tlb_entry(tlb, pudp, address);		\
 	} while (0)
 
@@ -291,7 +330,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,
 #define pte_free_tlb(tlb, ptep, address)			\
 	do {							\
 		__tlb_adjust_range(tlb, address, PAGE_SIZE);	\
-		tlb->freed_tables = 1;			\
+		tlb->freed_tables = 1;				\
+		tlb->cleared_pmds = 1;				\
 		__pte_free_tlb(tlb, ptep, address);		\
 	} while (0)
 #endif
@@ -300,7 +340,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,
 #define pmd_free_tlb(tlb, pmdp, address)			\
 	do {							\
 		__tlb_adjust_range(tlb, address, PAGE_SIZE);	\
-		tlb->freed_tables = 1;			\
+		tlb->freed_tables = 1;				\
+		tlb->cleared_puds = 1;				\
 		__pmd_free_tlb(tlb, pmdp, address);		\
 	} while (0)
 #endif
@@ -310,7 +351,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,
 #define pud_free_tlb(tlb, pudp, address)			\
 	do {							\
 		__tlb_adjust_range(tlb, address, PAGE_SIZE);	\
-		tlb->freed_tables = 1;			\
+		tlb->freed_tables = 1;				\
+		tlb->cleared_p4ds = 1;				\
 		__pud_free_tlb(tlb, pudp, address);		\
 	} while (0)
 #endif
@@ -321,7 +363,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb,
 #define p4d_free_tlb(tlb, pudp, address)			\
 	do {							\
 		__tlb_adjust_range(tlb, address, PAGE_SIZE);	\
-		tlb->freed_tables = 1;			\
+		tlb->freed_tables = 1;				\
 		__p4d_free_tlb(tlb, pudp, address);		\
 	} while (0)
 #endif
mm/memory.c +3 -1
index c467102a5cbc..9135f48e8d84 100644
@@ -267,8 +267,10 @@ void arch_tlb_finish_mmu(struct mmu_gather *tlb,
 {
 	struct mmu_gather_batch *batch, *next;
 
-	if (force)
+	if (force) {
+		__tlb_reset_range(tlb);
 		__tlb_adjust_range(tlb, start, end - start);
+	}
 
 	tlb_flush_mmu(tlb);
 
Signature · 1 line
2.1.4
#4 lore raw

PATCH v2 4/5 mm/memory: Move mmu_gather and TLB invalidation code into its own file

In reply to Will Deacon

From: Peter Zijlstra <[email protected]>

In preparation for maintaining the mmu_gather code as its own entity, move the implementation out of memory.c and into its own file.

Cc "Kirill A. Shutemov" <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Michal Hocko <[email protected]>
---
 include/asm-generic/tlb.h |   1 +
 mm/Makefile               |   6 +-
 mm/memory.c               | 249 --------------------------------------------
 mm/mmu_gather.c           | 259 ++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 263 insertions(+), 252 deletions(-)
 create mode 100644 mm/mmu_gather.c
include/asm-generic/tlb.h +1 -0
index 9791e98122a0..6be86c1c5c58 100644
@@ -138,6 +138,7 @@ void arch_tlb_gather_mmu(struct mmu_gather *tlb,
 void tlb_flush_mmu(struct mmu_gather *tlb);
 void arch_tlb_finish_mmu(struct mmu_gather *tlb,
 			 unsigned long start, unsigned long end, bool force);
+void tlb_flush_mmu_free(struct mmu_gather *tlb);
 extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page,
 				   int page_size);
 
mm/Makefile +3 -3
index 8716bdabe1e6..7c48e0d3d8ab 100644
@@ -23,9 +23,9 @@ KCOV_INSTRUMENT_vmstat.o := n
 
 mmu-y			:= nommu.o
 mmu-$(CONFIG_MMU)	:= gup.o highmem.o memory.o mincore.o \
-			   mlock.o mmap.o mprotect.o mremap.o msync.o \
-			   page_vma_mapped.o pagewalk.o pgtable-generic.o \
-			   rmap.o vmalloc.o
+			   mlock.o mmap.o mmu_gather.o mprotect.o mremap.o \
+			   msync.o page_vma_mapped.o pagewalk.o \
+			   pgtable-generic.o rmap.o vmalloc.o
 
 
 ifdef CONFIG_CROSS_MEMORY_ATTACH
mm/memory.c +0 -249

255 lines not on this page · raw message

mm/mmu_gather.c new file +259 -0

259 lines not on this page · raw message

Signature · 1 line
2.1.4
#5 lore raw

PATCH v2 5/5 MAINTAINERS: Add entry for MMU GATHER AND TLB INVALIDATION

In reply to Will Deacon

We recently had to debug a TLB invalidation problem on the munmap() path, which was made more difficult than necessary because:

(a) The MMU gather code had changed without people realising
(b) Many people subtly misunderstood the operation of the MMU gather code and its interactions with RCU and arch-specific TLB invalidation
(c) Untangling the intended behaviour involved educated guesswork and plenty of discussion

Hopefully, we can avoid getting into this mess again by designating a cross-arch group of people to look after this code. It is not intended that they will have a separate tree, but they at least provide a point of contact for anybody working in this area and can co-ordinate any proposed future changes to the internal API.

Cc Peter Zijlstra <[email protected]>
Cc Nicholas Piggin <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Michal Hocko <[email protected]>
---
 MAINTAINERS | 12 ++++++++++++
 1 file changed, 12 insertions(+)
MAINTAINERS +12 -0
index 9ad052aeac39..e490a0a0605a 100644
@@ -9681,6 +9681,18 @@ S:	Maintained
 F:	arch/arm/boot/dts/mmp*
 F:	arch/arm/mach-mmp/
 
+MMU GATHER AND TLB INVALIDATION
+M:	Will Deacon <[email protected]>
+M:	"Aneesh Kumar K.V" <[email protected]>
+M:	Nick Piggin <[email protected]>
+M:	Peter Zijlstra <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Maintained
+F:	arch/*/include/asm/tlb.h
+F:	include/asm-generic/tlb.h
+F:	mm/mmu_gather.c
+
 MN88472 MEDIA DRIVER
 M:	Antti Palosaari <[email protected]>
 L:	[email protected]
Signature · 1 line
2.1.4
maintainer
#6 lore raw
In reply to Will Deacon
Quoted 10 lines · On Tue, 4 Sep 2018 12:45:28 +0100 Will Deacon <[email protected]> wrote:

This series builds on the core changes I previously posted here:

rfc: http://lists.infradead.org/pipermail/linux-arm-kernel/2018-August/597821.html
v1: http://lists.infradead.org/pipermail/linux-arm-kernel/2018-August/598919.html

The main changes are:

* Move the mmu_gather bits out of memory.c and into their own file (looped in the mm people for this)

Quoted 8 lines

* Add a MAINTAINERS entry for the new file, and all tlb.h headers. If any mm developers would like to be included here as well, please just ask.

I'd like to queue these patches on their own branch in the arm64 git so that others can develop on top of them for the next merge window. Peter and Nick have both expressed an interest in that, and I already have a bunch of arm64 optimisations on top which I posted previously.

All looks good to me - please proceed that way. Please also add me to the MAINTAINERS record so I get more emails.

#7 lore raw
Quoted 16 lines · Andrew Morton
Quoted 14 lines · On Tue, 4 Sep 2018 12:45:28 +0100 Will Deacon <[email protected]> wrote:

This series builds on the core changes I previously posted here:

rfc: http://lists.infradead.org/pipermail/linux-arm-kernel/2018-August/597821.html
v1: http://lists.infradead.org/pipermail/linux-arm-kernel/2018-August/598919.html

The main changes are:

* Move the mmu_gather bits out of memory.c and into their own file (looped in the mm people for this)

* Add a MAINTAINERS entry for the new file, and all tlb.h headers. If any mm developers would like to be included here as well, please just ask.

Quoted 7 lines
Quoted 4 lines

I'd like to queue these patches on their own branch in the arm64 git so that others can develop on top of them for the next merge window. Peter and Nick have both expressed an interest in that, and I already have a bunch of arm64 optimisations on top which I posted previously.

All looks good to me - please proceed that way. Please also add me to the MAINTAINERS record so I get more emails.

Cheers, Andrew. I'll add you to the MAINTAINERS entry and get this lot into -next once the kbuild robot is happy that I've got all the header files right.

Will