Skip to contentSkip to conversation

mremap: properly flush TLB before releasing the page

PATCH In review Will Deacon 5 participants 6 messages Nov 2, 2018

Subsystems (4)
Stable branch Memory management Memory mapping THP
lore
Share & tools
lore

Will Deacon

author
#0 lore raw

PATCH 1/1 mremap: properly flush TLB before releasing the page

From: Linus Torvalds <[email protected]>

Commit eb66ae030829605d61fbef1909ce310e29f78821 upstream.

This is a backport to stable 4.4.y.

Jann Horn points out that our TLB flushing was subtly wrong for the mremap() case. What makes mremap() special is that we don't follow the usual "add page to list of pages to be freed, then flush tlb, and then free pages". No, mremap() obviously just _moves_ the page from one page table location to another.

That matters, because mremap() thus doesn't directly control the lifetime of the moved page with a freelist: instead, the lifetime of the page is controlled by the page table locking, that serializes access to the entry.

As a result, we need to flush the TLB not just before releasing the lock for the source location (to avoid any concurrent accesses to the entry), but also before we release the destination page table lock (to avoid the TLB being flushed after somebody else has already done something to that page).

This also makes the whole "need_flush" logic unnecessary, since we now always end up flushing the TLB for every valid entry.

Reported-and-tested-by Jann Horn <[email protected]>
Acked-by Will Deacon <[email protected]>
Tested-by Ingo Molnar <[email protected]>
Acked-by Peter Zijlstra (Intel) <[email protected]>
Signed-off-by Linus Torvalds <[email protected]>
Signed-off-by Greg Kroah-Hartman <[email protected]>
[will: backport to 4.4 stable]
Signed-off-by Will Deacon <[email protected]>
---
 mm/huge_memory.c |  6 +++++-
 mm/mremap.c      | 21 ++++++++++++++++-----
 2 files changed, 21 insertions(+), 6 deletions(-)
mm/huge_memory.c +5 -1
index c4ea57ee2fd1..465786cd6490 100644
@@ -1511,7 +1511,7 @@ int move_huge_pmd(struct vm_area_struct *vma, struct vm_area_struct *new_vma,
 	spinlock_t *old_ptl, *new_ptl;
 	int ret = 0;
 	pmd_t pmd;
-
+	bool force_flush = false;
 	struct mm_struct *mm = vma->vm_mm;
 
 	if ((old_addr & ~HPAGE_PMD_MASK) ||

@@ -1539,6 +1539,8 @@ int move_huge_pmd(struct vm_area_struct *vma, struct vm_area_struct *new_vma,
 		if (new_ptl != old_ptl)
 			spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
 		pmd = pmdp_huge_get_and_clear(mm, old_addr, old_pmd);
+		if (pmd_present(pmd))
+			force_flush = true;
 		VM_BUG_ON(!pmd_none(*new_pmd));
 
 		if (pmd_move_must_withdraw(new_ptl, old_ptl)) {

@@ -1547,6 +1549,8 @@ int move_huge_pmd(struct vm_area_struct *vma, struct vm_area_struct *new_vma,
 			pgtable_trans_huge_deposit(mm, new_pmd, pgtable);
 		}
 		set_pmd_at(mm, new_addr, new_pmd, pmd_mksoft_dirty(pmd));
+		if (force_flush)
+			flush_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
 		if (new_ptl != old_ptl)
 			spin_unlock(new_ptl);
 		spin_unlock(old_ptl);
mm/mremap.c +16 -5
index fe7b7f65f4f4..450b306d473e 100644
@@ -96,6 +96,8 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
 	struct mm_struct *mm = vma->vm_mm;
 	pte_t *old_pte, *new_pte, pte;
 	spinlock_t *old_ptl, *new_ptl;
+	bool force_flush = false;
+	unsigned long len = old_end - old_addr;
 
 	/*
 	 * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma

@@ -143,12 +145,26 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
 		if (pte_none(*old_pte))
 			continue;
 		pte = ptep_get_and_clear(mm, old_addr, old_pte);
+		/*
+		 * If we are remapping a valid PTE, make sure
+		 * to flush TLB before we drop the PTL for the PTE.
+		 *
+		 * NOTE! Both old and new PTL matter: the old one
+		 * for racing with page_mkclean(), the new one to
+		 * make sure the physical page stays valid until
+		 * the TLB entry for the old mapping has been
+		 * flushed.
+		 */
+		if (pte_present(pte))
+			force_flush = true;
 		pte = move_pte(pte, new_vma->vm_page_prot, old_addr, new_addr);
 		pte = move_soft_dirty_pte(pte);
 		set_pte_at(mm, new_addr, new_pte, pte);
 	}
 
 	arch_leave_lazy_mmu_mode();
+	if (force_flush)
+		flush_tlb_range(vma, old_end - len, old_end);
 	if (new_ptl != old_ptl)
 		spin_unlock(new_ptl);
 	pte_unmap(new_pte - 1);

@@ -168,7 +184,6 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
 {
 	unsigned long extent, next, old_end;
 	pmd_t *old_pmd, *new_pmd;
-	bool need_flush = false;
 	unsigned long mmun_start;	/* For mmu_notifiers */
 	unsigned long mmun_end;		/* For mmu_notifiers */
 

@@ -207,7 +222,6 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
 					anon_vma_unlock_write(vma->anon_vma);
 			}
 			if (err > 0) {
-				need_flush = true;
 				continue;
 			} else if (!err) {
 				split_huge_page_pmd(vma, old_addr, old_pmd);

@@ -224,10 +238,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
 			extent = LATENCY_LIMIT;
 		move_ptes(vma, old_pmd, old_addr, old_addr + extent,
 			  new_vma, new_pmd, new_addr, need_rmap_locks);
-		need_flush = true;
 	}
-	if (likely(need_flush))
-		flush_tlb_range(vma, old_end-len, old_addr);
 
 	mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
Signature · 1 line
2.1.4
#1 lore raw
Quoted 3 lines · Will Deacon

From: Linus Torvalds <[email protected]>

Commit eb66ae030829605d61fbef1909ce310e29f78821 upstream.

I have never seen the original patch on mailing lists, so I'll reply to the backport.

Quoted 9 lines

This is a backport to stable 4.4.y.

Jann Horn points out that our TLB flushing was subtly wrong for the mremap() case. What makes mremap() special is that we don't follow the usual "add page to list of pages to be freed, then flush tlb, and then free pages". No, mremap() obviously just _moves_ the page from one page table location to another.

Quoted 4 lines

That matters, because mremap() thus doesn't directly control the lifetime of the moved page with a freelist: instead, the lifetime of the page is controlled by the page table locking, that serializes access to the entry.

I believe we do control the lifetime of the page with mmap_sem, don't we?

I mean any shoot down of the page from a mapping would require at least down_read(mmap_sem) and we hold down_write(mmap_sem). Hm?

Quoted 70 lines

As a result, we need to flush the TLB not just before releasing the lock for the source location (to avoid any concurrent accesses to the entry), but also before we release the destination page table lock (to avoid the TLB being flushed after somebody else has already done something to that page).

This also makes the whole "need_flush" logic unnecessary, since we now always end up flushing the TLB for every valid entry.

Reported-and-tested-by Jann Horn <[email protected]>
Acked-by Will Deacon <[email protected]>
Tested-by Ingo Molnar <[email protected]>
Acked-by Peter Zijlstra (Intel) <[email protected]>
Signed-off-by Linus Torvalds <[email protected]>
Signed-off-by Greg Kroah-Hartman <[email protected]>
[will: backport to 4.4 stable]
Signed-off-by Will Deacon <[email protected]>
---
 mm/huge_memory.c |  6 +++++-
 mm/mremap.c      | 21 ++++++++++++++++-----
 2 files changed, 21 insertions(+), 6 deletions(-)
mm/huge_memory.c +5 -1
index c4ea57ee2fd1..465786cd6490 100644
@@ -1511,7 +1511,7 @@ int move_huge_pmd(struct vm_area_struct *vma, struct vm_area_struct *new_vma,
 	spinlock_t *old_ptl, *new_ptl;
 	int ret = 0;
 	pmd_t pmd;
-
+	bool force_flush = false;
 	struct mm_struct *mm = vma->vm_mm;
 
 	if ((old_addr & ~HPAGE_PMD_MASK) ||

@@ -1539,6 +1539,8 @@ int move_huge_pmd(struct vm_area_struct *vma, struct vm_area_struct *new_vma,
 		if (new_ptl != old_ptl)
 			spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
 		pmd = pmdp_huge_get_and_clear(mm, old_addr, old_pmd);
+		if (pmd_present(pmd))
+			force_flush = true;
 		VM_BUG_ON(!pmd_none(*new_pmd));
 
 		if (pmd_move_must_withdraw(new_ptl, old_ptl)) {

@@ -1547,6 +1549,8 @@ int move_huge_pmd(struct vm_area_struct *vma, struct vm_area_struct *new_vma,
 			pgtable_trans_huge_deposit(mm, new_pmd, pgtable);
 		}
 		set_pmd_at(mm, new_addr, new_pmd, pmd_mksoft_dirty(pmd));
+		if (force_flush)
+			flush_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
 		if (new_ptl != old_ptl)
 			spin_unlock(new_ptl);
 		spin_unlock(old_ptl);
mm/mremap.c +2 -0
index fe7b7f65f4f4..450b306d473e 100644
@@ -96,6 +96,8 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
 	struct mm_struct *mm = vma->vm_mm;
 	pte_t *old_pte, *new_pte, pte;
 	spinlock_t *old_ptl, *new_ptl;
+	bool force_flush = false;
+	unsigned long len = old_end - old_addr;
 
 	/*
 	 * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma

@@ -143,12 +145,26 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
 		if (pte_none(*old_pte))
 			continue;
 		pte = ptep_get_and_clear(mm, old_addr, old_pte);

mm/mremap.c static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, · line 157

Quoted 10 lines
mm/mremap.c +10 -0
+		/*
+		 * If we are remapping a valid PTE, make sure
+		 * to flush TLB before we drop the PTL for the PTE.
+		 *
+		 * NOTE! Both old and new PTL matter: the old one
+		 * for racing with page_mkclean(), the new one to
+		 * make sure the physical page stays valid until
+		 * the TLB entry for the old mapping has been
+		 * flushed.
+		 */

Could you elaborate on the race with page_mkclean()?

I think the new logic is unnecessary strict (and slow).

Any barely sane userspace must not access the old mapping after mremap(MREMAP_MAYMOVE) called and must not access the new mapping before the mremap() returns.

The old logic *should* be safe if this argument valid.

Do I miss something?

Quoted 43 lines
mm/mremap.c +4 -5
+		if (pte_present(pte))
+			force_flush = true;
 		pte = move_pte(pte, new_vma->vm_page_prot, old_addr, new_addr);
 		pte = move_soft_dirty_pte(pte);
 		set_pte_at(mm, new_addr, new_pte, pte);
 	}
 
 	arch_leave_lazy_mmu_mode();
+	if (force_flush)
+		flush_tlb_range(vma, old_end - len, old_end);
 	if (new_ptl != old_ptl)
 		spin_unlock(new_ptl);
 	pte_unmap(new_pte - 1);

@@ -168,7 +184,6 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
 {
 	unsigned long extent, next, old_end;
 	pmd_t *old_pmd, *new_pmd;
-	bool need_flush = false;
 	unsigned long mmun_start;	/* For mmu_notifiers */
 	unsigned long mmun_end;		/* For mmu_notifiers */
 

@@ -207,7 +222,6 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
 					anon_vma_unlock_write(vma->anon_vma);
 			}
 			if (err > 0) {
-				need_flush = true;
 				continue;
 			} else if (!err) {
 				split_huge_page_pmd(vma, old_addr, old_pmd);

@@ -224,10 +238,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
 			extent = LATENCY_LIMIT;
 		move_ptes(vma, old_pmd, old_addr, old_addr + extent,
 			  new_vma, new_pmd, new_addr, need_rmap_locks);
-		need_flush = true;
 	}
-	if (likely(need_flush))
-		flush_tlb_range(vma, old_end-len, old_addr);
 
 	mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
Signature · 1 line
2.1.4
Signature · 1 line
 Kirill A. Shutemov
reviewer
#2 lore raw
Quoted 7 lines · Kirill A. Shutemov
Quoted 3 lines · Will Deacon

From: Linus Torvalds <[email protected]>

Commit eb66ae030829605d61fbef1909ce310e29f78821 upstream.

I have never seen the original patch on mailing lists, so I'll reply to the backport.

For context, the original bug report is public at
https://bugs.chromium.org/p/project-zero/issues/detail?id=1695 .

Quoted 8 lines
Quoted 8 lines

This is a backport to stable 4.4.y.

Jann Horn points out that our TLB flushing was subtly wrong for the mremap() case. What makes mremap() special is that we don't follow the usual "add page to list of pages to be freed, then flush tlb, and then free pages". No, mremap() obviously just _moves_ the page from one page table location to another.

Quoted 6 lines
Quoted 4 lines

That matters, because mremap() thus doesn't directly control the lifetime of the moved page with a freelist: instead, the lifetime of the page is controlled by the page table locking, that serializes access to the entry.

I believe we do control the lifetime of the page with mmap_sem, don't we?

Nope. For file-backed pages, someone can come through the file mapping and free our pages, e.g. through the ftruncate() syscall.

Quoted 142 lines

I mean any shoot down of the page from a mapping would require at least down_read(mmap_sem) and we hold down_write(mmap_sem). Hm?

Quoted 70 lines

As a result, we need to flush the TLB not just before releasing the lock for the source location (to avoid any concurrent accesses to the entry), but also before we release the destination page table lock (to avoid the TLB being flushed after somebody else has already done something to that page).

This also makes the whole "need_flush" logic unnecessary, since we now always end up flushing the TLB for every valid entry.

Reported-and-tested-by Jann Horn <[email protected]>
Acked-by Will Deacon <[email protected]>
Tested-by Ingo Molnar <[email protected]>
Acked-by Peter Zijlstra (Intel) <[email protected]>
Signed-off-by Linus Torvalds <[email protected]>
Signed-off-by Greg Kroah-Hartman <[email protected]>
[will: backport to 4.4 stable]
Signed-off-by Will Deacon <[email protected]>
---
 mm/huge_memory.c |  6 +++++-
 mm/mremap.c      | 21 ++++++++++++++++-----
 2 files changed, 21 insertions(+), 6 deletions(-)
mm/huge_memory.c +5 -1
index c4ea57ee2fd1..465786cd6490 100644
@@ -1511,7 +1511,7 @@ int move_huge_pmd(struct vm_area_struct *vma, struct vm_area_struct *new_vma,
      spinlock_t *old_ptl, *new_ptl;
      int ret = 0;
      pmd_t pmd;
-
+     bool force_flush = false;
      struct mm_struct *mm = vma->vm_mm;
 
      if ((old_addr & ~HPAGE_PMD_MASK) ||

@@ -1539,6 +1539,8 @@ int move_huge_pmd(struct vm_area_struct *vma, struct vm_area_struct *new_vma,
              if (new_ptl != old_ptl)
                      spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
              pmd = pmdp_huge_get_and_clear(mm, old_addr, old_pmd);
+             if (pmd_present(pmd))
+                     force_flush = true;
              VM_BUG_ON(!pmd_none(*new_pmd));
 
              if (pmd_move_must_withdraw(new_ptl, old_ptl)) {

@@ -1547,6 +1549,8 @@ int move_huge_pmd(struct vm_area_struct *vma, struct vm_area_struct *new_vma,
                      pgtable_trans_huge_deposit(mm, new_pmd, pgtable);
              }
              set_pmd_at(mm, new_addr, new_pmd, pmd_mksoft_dirty(pmd));
+             if (force_flush)
+                     flush_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
              if (new_ptl != old_ptl)
                      spin_unlock(new_ptl);
              spin_unlock(old_ptl);
mm/mremap.c +2 -0
index fe7b7f65f4f4..450b306d473e 100644
@@ -96,6 +96,8 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
      struct mm_struct *mm = vma->vm_mm;
      pte_t *old_pte, *new_pte, pte;
      spinlock_t *old_ptl, *new_ptl;
+     bool force_flush = false;
+     unsigned long len = old_end - old_addr;
 
      /*
       * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma

@@ -143,12 +145,26 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
              if (pte_none(*old_pte))
                      continue;
              pte = ptep_get_and_clear(mm, old_addr, old_pte);

mm/mremap.c static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, · line 157

Quoted 10 lines
mm/mremap.c +10 -0
+             /*
+              * If we are remapping a valid PTE, make sure
+              * to flush TLB before we drop the PTL for the PTE.
+              *
+              * NOTE! Both old and new PTL matter: the old one
+              * for racing with page_mkclean(), the new one to
+              * make sure the physical page stays valid until
+              * the TLB entry for the old mapping has been
+              * flushed.
+              */

Could you elaborate on the race with page_mkclean()?

I think the new logic is unnecessary strict (and slow).

Any barely sane userspace must not access the old mapping after mremap(MREMAP_MAYMOVE) called and must not access the new mapping before the mremap() returns.

The old logic *should* be safe if this argument valid.

Do I miss something?

Quoted 41 lines
mm/mremap.c +4 -6
+             if (pte_present(pte))
+                     force_flush = true;
              pte = move_pte(pte, new_vma->vm_page_prot, old_addr, new_addr);
              pte = move_soft_dirty_pte(pte);
              set_pte_at(mm, new_addr, new_pte, pte);
      }
 
      arch_leave_lazy_mmu_mode();
+     if (force_flush)
+             flush_tlb_range(vma, old_end - len, old_end);
      if (new_ptl != old_ptl)
              spin_unlock(new_ptl);
      pte_unmap(new_pte - 1);

@@ -168,7 +184,6 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
 {
      unsigned long extent, next, old_end;
      pmd_t *old_pmd, *new_pmd;
-     bool need_flush = false;
      unsigned long mmun_start;       /* For mmu_notifiers */
      unsigned long mmun_end;         /* For mmu_notifiers */
 

@@ -207,7 +222,6 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
                                      anon_vma_unlock_write(vma->anon_vma);
                      }
                      if (err > 0) {
-                             need_flush = true;
                              continue;
                      } else if (!err) {
                              split_huge_page_pmd(vma, old_addr, old_pmd);

@@ -224,10 +238,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
                      extent = LATENCY_LIMIT;
              move_ptes(vma, old_pmd, old_addr, old_addr + extent,
                        new_vma, new_pmd, new_addr, need_rmap_locks);
-             need_flush = true;
      }
-     if (likely(need_flush))
-             flush_tlb_range(vma, old_end-len, old_addr);
 
      mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
 
--
Quoted 2 lines

2.1.4

--
Kirill A. Shutemov

#3 lore raw
Quoted 11 lines · Jann Horn
Quoted 7 lines · Kirill A. Shutemov
Quoted 3 lines · Will Deacon

From: Linus Torvalds <[email protected]>

Commit eb66ae030829605d61fbef1909ce310e29f78821 upstream.

I have never seen the original patch on mailing lists, so I'll reply to the backport.

For context, the original bug report is public at
https://bugs.chromium.org/p/project-zero/issues/detail?id=1695 .

Okay. I see.

I wounder if it would be cheaper to fix this by taking i_mmap_lock_write() unconditionally in mremap() path rather than do a lot of flushing. We take the lock now only to remap to lower addresses.

Signature · 1 line
 Kirill A. Shutemov
#4 lore raw
Quoted 3 lines · Kirill A. Shutemov

I wounder if it would be cheaper to fix this by taking i_mmap_lock_write() unconditionally in mremap() path rather than do a lot of flushing.

That wouldn't help. Think anonymous pages and try_to_free() rmap walk. So then I think we'd have to take the anonvma lock or something.

And it's not like we are likely to even do any more flushes, really. We don't flush for each page, only for each page table. So every 512 pages or so.

Linus