Skip to contentSkip to conversation

mm/vmalloc: fix up vmap error handling

PATCH In review Hao Ge 3 participants 16 messages Nov 16 – Nov 17

Subsystems (6)
lore
Share & tools
lore

Hao Ge

author
#0 lore raw

v5 cover mm/vmalloc: fix up vmap error handling

__vmap_pages_range_noflush() and friends can install some PTEs before failing and leave them mapped, and the kernel callers do not agree on who cleans them up. pcpu_map_pages() and kmsan_ioremap_page_range() roll back what they mapped before the failure, while vm_module_tags_populate() and the __GFP_NOFAIL retry loop in __vmalloc_area_node() rely on the mapping functions cleaning up and do not call anything like vunmap_range() themselves. When the same range is mapped again, the attempt hits the leftovers and fails with a warning in vmap_pages_pte_range(). A leftover huge pmd is worse. During the retry, vmap_try_huge_pmd() calls pmd_free_pte_page() on it, and the x86 implementation cannot tell a leaf entry from a page table and frees the mapped page.

After discussing with Suren and Ulad [1], we move the rollback into the mapping functions. A failed vmap leaves nothing mapped, and the callers no longer clean up after it.

Patch 1 gives __vunmap_range_noflush() a can_resched argument, the failure paths can be reached with a non-blocking gfp mask and the undo must not schedule there. Patch 2 is the KMSAN part, the shadow and the origin of a range are mapped by two separate calls, so the rollback there has to cover both. Patches 3 and 4 reject an unaligned range at the entries of vmap_page_range() and __vunmap_range_noflush(). Patch 5 only flushes the cache once the mapping succeeded. Patches 6 to 9 add the rollback to the mapping functions, one entry per patch, vmap_page_range(), __vmap_pages_range_noflush(), the KMSAN metadata and the batched vmap() path. Patches 10 and 11 drop the powerpc and percpu cleanups that are not needed anymore.

Changes since v4:
- new patch 1, the unmap paths can run under a non-blocking gfp mask and must skip cond_resched(); it carries a Fixes: tag so it travels to stable together with patch 2
- drop v4 patch 2, the KMSAN-disabled vunmap skip. !kmsan_enabled does not mean no metadata was installed, KMSAN_WARN_ON() can clear the flag at runtime, on Suren's review
- v4 patch 3 now warns and returns -EINVAL at vmap_page_range(), instead of BUG_ON()ing inside vmap_range_noflush()
- drop v4 patch 4, the walk check. No in-tree caller passes an unaligned range to vmap_pages_range_noflush_walk(), the vmap allocator guarantees the alignment for vmalloc()/vmap()/ vm_map_ram()
- v4 patch 5 warns and bails out instead of BUG_ON(), following Ulad's review of v4
- v4 patch 7 is split into patches 6 to 9, one mapping entry per patch. The batched vmap() path is covered now too, on Suren's request

[1] https://lore.kernel.org/all/[email protected]/

v4 link: https://lore.kernel.org/all/[email protected]/
v3 link: https://lore.kernel.org/all/[email protected]/
v2 link: https://lore.kernel.org/all/[email protected]/
v1 link: https://lore.kernel.org/all/[email protected]/

Hao Ge (11):
mm/vmalloc: skip cond_resched() in the vmap unmap paths when atomic mm/kmsan: undo the shadow mapping when the origin mapping fails mm/vmalloc: reject an unaligned range in vmap_page_range() mm/vmalloc: reject an unaligned range in __vunmap_range_noflush() mm/vmalloc: only flush the cache after the vmap succeeded mm/vmalloc: undo what vmap_page_range() mapped on failure mm/vmalloc: undo what __vmap_pages_range_noflush() mapped on failure mm/vmalloc: drop the KMSAN metadata in vmap_pages_range_noflush() on
failure
mm/vmalloc: undo what vmap_pages_range_batched() mapped on failure
powerpc: drop redundant unmaps of failed vmap mappings mm/percpu: stop unmapping the CPU that failed to map

 arch/powerpc/kernel/isa-bridge.c |   5 +-
 arch/powerpc/kernel/pci_64.c     |   4 +-
 include/linux/kmsan.h            |   8 +-
 mm/kmsan/hooks.c                 |  18 ++--
 mm/kmsan/shadow.c                |   9 +-
 mm/percpu-vm.c                   |   5 +-
 mm/vmalloc.c                     | 154 +++++++++++++++++++++++--------
 mm/vmalloc.h                     |   6 +-
 8 files changed, 151 insertions(+), 58 deletions(-)
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH v5 1/11 mm/vmalloc: skip cond_resched() in the vmap unmap paths when atomic

The vmap failure paths can be reached with GFP_ATOMIC or GFP_NOWAIT since commit b186a94227b7 ("kmsan: remove hard-coded GFP_KERNEL flags") and commit 8da89ba18ed4 ("mm/vmalloc: handle non-blocking GFP in __vmalloc_area_node()"), so an undo of a partially installed mapping may have to run in atomic context, where the cond_resched() in vunmap_pmd_range() cannot schedule.

Give __vunmap_range_noflush() and kmsan_vunmap_range_noflush() a can_resched argument and skip the cond_resched() when it is false. The existing callers are on sleepable paths and pass true, so nothing changes for them.

Fixes 47ebd0310e89 ("mm: kmsan: handle alloc failures in kmsan_vmap_pages_range_noflush()")
---
 include/linux/kmsan.h |  8 ++++++--
 mm/kmsan/hooks.c      | 14 +++++++++-----
 mm/vmalloc.c          | 24 ++++++++++++++----------
 mm/vmalloc.h          |  3 ++-
 4 files changed, 31 insertions(+), 18 deletions(-)
include/linux/kmsan.h +6 -2
index 7da9fd506b39..0016c032d1e7 100644
@@ -150,11 +150,14 @@ int __must_check kmsan_vmap_pages_range_noflush(unsigned long start,
  * kmsan_vunmap_kernel_range_noflush() - Notify KMSAN about a vunmap.
  * @start: start of vunmapped range.
  * @end:   end of vunmapped range.
+ * @can_resched: whether the unmap walk may call cond_resched(); pass
+ * false from atomic context.
  *
  * KMSAN unmaps the contiguous metadata ranges created by
  * kmsan_map_kernel_range_noflush().
  */
-void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end);
+void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end,
+				bool can_resched);
 
 /**
  * kmsan_ioremap_page_range() - Notify KMSAN about a ioremap_page_range() call.
@@ -355,7 +358,8 @@ static inline int __must_check kmsan_vmap_pages_range_noflush(
 }
 
 static inline void kmsan_vunmap_range_noflush(unsigned long start,
-					      unsigned long end)
+					      unsigned long end,
+					      bool can_resched)
 {
 }
 
mm/kmsan/hooks.c +9 -5
index a706666db097..69008b085658 100644
@@ -136,10 +136,13 @@ static unsigned long vmalloc_origin(unsigned long addr)
 						 KMSAN_META_ORIGIN);
 }
 
-void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end)
+void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end,
+				bool can_resched)
 {
-	__vunmap_range_noflush(vmalloc_shadow(start), vmalloc_shadow(end));
-	__vunmap_range_noflush(vmalloc_origin(start), vmalloc_origin(end));
+	__vunmap_range_noflush(vmalloc_shadow(start), vmalloc_shadow(end),
+			       can_resched);
+	__vunmap_range_noflush(vmalloc_origin(start), vmalloc_origin(end),
+			       can_resched);
 	flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end));
 	flush_cache_vmap(vmalloc_origin(start), vmalloc_origin(end));
 }
@@ -168,8 +171,9 @@ static void kmsan_iounmap_pages(unsigned long start, unsigned long end)
 		if (origin)
 			__free_pages(origin, KMSAN_IOREMAP_META_ORDER);
 	}
-	__vunmap_range_noflush(shadow_start, shadow_end);
-	__vunmap_range_noflush(origin_start, origin_end);
+	/* ioremap metadata mappings are always GFP_KERNEL, so resched is safe. */
+	__vunmap_range_noflush(shadow_start, shadow_end, true);
+	__vunmap_range_noflush(origin_start, origin_end, true);
 	flush_tlb_kernel_range(shadow_start, shadow_end);
 	flush_tlb_kernel_range(origin_start, origin_end);
 }
mm/vmalloc.c +14 -10
index ccaf158c5b7f..3c0d9e69b886 100644
@@ -429,7 +429,7 @@ static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 }
 
 static void vunmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,
-			     pgtbl_mod_mask *mask)
+			     pgtbl_mod_mask *mask, bool can_resched)
 {
 	pmd_t *pmd;
 	unsigned long next;
@@ -451,12 +451,14 @@ static void vunmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,
 			continue;
 		vunmap_pte_range(pmd, addr, next, mask);
 
-		cond_resched();
+		/* The undo path may run in atomic context. */
+		if (can_resched)
+			cond_resched();
 	} while (pmd++, addr = next, addr != end);
 }
 
 static void vunmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,
-			     pgtbl_mod_mask *mask)
+			     pgtbl_mod_mask *mask, bool can_resched)
 {
 	pud_t *pud;
 	unsigned long next;
@@ -476,12 +478,12 @@ static void vunmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,
 		}
 		if (pud_none_or_clear_bad(pud))
 			continue;
-		vunmap_pmd_range(pud, addr, next, mask);
+		vunmap_pmd_range(pud, addr, next, mask, can_resched);
 	} while (pud++, addr = next, addr != end);
 }
 
 static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
-			     pgtbl_mod_mask *mask)
+			     pgtbl_mod_mask *mask, bool can_resched)
 {
 	p4d_t *p4d;
 	unsigned long next;
@@ -496,7 +498,7 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
 
 		if (p4d_none_or_clear_bad(p4d))
 			continue;
-		vunmap_pud_range(p4d, addr, next, mask);
+		vunmap_pud_range(p4d, addr, next, mask, can_resched);
 	} while (p4d++, addr = next, addr != end);
 }
 
@@ -512,7 +514,8 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
  *
  * This is an internal function only. Do not use outside mm/.
  */
-void __vunmap_range_noflush(unsigned long start, unsigned long end)
+void __vunmap_range_noflush(unsigned long start, unsigned long end,
+			    bool can_resched)
 {
 	unsigned long next;
 	pgd_t *pgd;
@@ -527,7 +530,7 @@ void __vunmap_range_noflush(unsigned long start, unsigned long end)
 			mask |= PGTBL_PGD_MODIFIED;
 		if (pgd_none_or_clear_bad(pgd))
 			continue;
-		vunmap_p4d_range(pgd, addr, next, &mask);
+		vunmap_p4d_range(pgd, addr, next, &mask, can_resched);
 	} while (pgd++, addr = next, addr != end);
 
 	if (mask & ARCH_PAGE_TABLE_SYNC_MASK)
@@ -536,8 +539,9 @@ void __vunmap_range_noflush(unsigned long start, unsigned long end)
 
 void vunmap_range_noflush(unsigned long start, unsigned long end)
 {
-	kmsan_vunmap_range_noflush(start, end);
-	__vunmap_range_noflush(start, end);
+	/* The unmap paths are sleepable, keep the resched point. */
+	kmsan_vunmap_range_noflush(start, end, true);
+	__vunmap_range_noflush(start, end, true);
 }
 
 /**
mm/vmalloc.h +2 -1
index d8372a4f65c4..4ec114a75f04 100644
@@ -44,6 +44,7 @@ int __must_check __vmap_pages_range_noflush(unsigned long addr,
 
 void vunmap_range_noflush(unsigned long start, unsigned long end);
 
-void __vunmap_range_noflush(unsigned long start, unsigned long end);
+void __vunmap_range_noflush(unsigned long start, unsigned long end,
+			    bool can_resched);
 
 #endif /* __MM_VMALLOC_H */
Signature · 1 line
2.25.1
author
#2 lore raw

PATCH v5 2/11 mm/kmsan: undo the shadow mapping when the origin mapping fails

In reply to Hao Ge

kmsan_vmap_pages_range_noflush() first maps the shadow pages and then the origin pages. If the second mapping fails, the first one is left mapped, and the callers do not roll it back. The next vmap of the same metadata range hits the stale PTEs again and fails with a WARN_ON() and -EBUSY in vmap_pages_pte_range().

Undo the shadow mapping on that error path, the same way kmsan_ioremap_page_range() cleans up after a partial failure. __vunmap_range_noflush() only clears the PTEs; nothing has touched the shadow mapping, so no TLB flush is needed.

Fixes 47ebd0310e89 ("mm: kmsan: handle alloc failures in kmsan_vmap_pages_range_noflush()")
---
 mm/kmsan/shadow.c | 5 +++++
 1 file changed, 5 insertions(+)
mm/kmsan/shadow.c +5 -0
index 0c88d89bf0d6..212449db34a1 100644
@@ -258,6 +258,11 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end,
 					    o_pages, page_shift);
 	kmsan_leave_runtime();
 	if (mapped) {
+		/* Undo the shadow mapping set up above. */
+		kmsan_enter_runtime();
+		__vunmap_range_noflush(shadow_start, shadow_end,
+				       gfpflags_allow_blocking(gfp_mask));
+		kmsan_leave_runtime();
 		err = mapped;
 		goto ret;
 	}
Signature · 1 line
2.25.1
author
#3 lore raw

PATCH v5 3/11 mm/vmalloc: reject an unaligned range in vmap_page_range()

In reply to Hao Ge

commit a739e6b557af ("mm: vmalloc: WARN_ON if mapping size is not PAGE_SIZE aligned") put the check in vmap_pte_range(), which is too late, because the upper levels may already have installed page tables for a range that is rejected anyway.

The pte loop maps whole pages and stops only when addr lands exactly on end. An unaligned end lets addr jump over it and the loop keeps mapping past the range. An unaligned start maps the page below the range and usually misses end too. Such a range is never a legitimate request.

vmap_range_noflush() has a single caller, vmap_page_range(). Warn on addr and end there and bail out with -EINVAL before any page table is installed, and drop the check from vmap_pte_range().

---
 mm/vmalloc.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
mm/vmalloc.c +3 -3
index 3c0d9e69b886..a168f6f20f4f 100644
@@ -125,9 +125,6 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 	unsigned long size;
 	unsigned int nr_pages;
 
-	if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr)))
-		return -EINVAL;
-
 	pfn = phys_addr >> PAGE_SHIFT;
 	pte = pte_alloc_kernel_track(pmd, addr, mask);
 	if (!pte)
@@ -369,6 +366,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,
 {
 	int err;
 
+	if (WARN_ON_ONCE(!PAGE_ALIGNED(addr) || !PAGE_ALIGNED(end)))
+		return -EINVAL;
+
 	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
 				 ioremap_max_page_shift);
 	flush_cache_vmap(addr, end);
Signature · 1 line
2.25.1
Reviewed-by on v5 3/11
#4 lore raw

Re: [PATCH v5 03/11] mm/vmalloc: reject an unaligned range in vmap_page_range()

Quoted 11 lines · Hao Ge

commit a739e6b557af ("mm: vmalloc: WARN_ON if mapping size is not PAGE_SIZE aligned") put the check in vmap_pte_range(), which is too late, because the upper levels may already have installed page tables for a range that is rejected anyway.

The pte loop maps whole pages and stops only when addr lands exactly on end. An unaligned end lets addr jump over it and the loop keeps mapping past the range. An unaligned start maps the page below the range and usually misses end too. Such a range is never a legitimate request.

Quoted 8 lines

vmap_range_noflush() has a single caller, vmap_page_range(). Warn on addr and end there and bail out with -EINVAL before any page table is installed, and drop the check from vmap_pte_range().

---

Quoted 27 lines
 mm/vmalloc.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
mm/vmalloc.c +3 -3
index 3c0d9e69b886..a168f6f20f4f 100644
@@ -125,9 +125,6 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 	unsigned long size;
 	unsigned int nr_pages;
 
-	if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr)))
-		return -EINVAL;
-
 	pfn = phys_addr >> PAGE_SHIFT;
 	pte = pte_alloc_kernel_track(pmd, addr, mask);
 	if (!pte)
@@ -369,6 +366,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,
 {
 	int err;
 
+	if (WARN_ON_ONCE(!PAGE_ALIGNED(addr) || !PAGE_ALIGNED(end)))
+		return -EINVAL;
+
 	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
 				 ioremap_max_page_shift);
 	flush_cache_vmap(addr, end);
author
#5 lore raw

PATCH v5 4/11 mm/vmalloc: reject an unaligned range in __vunmap_range_noflush()

In reply to Hao Ge

The PTE loop stops only at addr == end. An unaligned end is never reached, so the loop keeps clearing past the range. An unaligned start clears the page below the range first. Both are caller bugs, and the damage is hard to attribute afterwards.

vmap only installs page aligned mappings, so an unaligned unmap request is never a legitimate teardown. Warn and bail out at the entry of __vunmap_range_noflush(), before any PTE is cleared.

---
 mm/vmalloc.c | 3 +++
 1 file changed, 3 insertions(+)
mm/vmalloc.c +3 -0
index a168f6f20f4f..e2b63cb1592b 100644
@@ -523,6 +523,9 @@ void __vunmap_range_noflush(unsigned long start, unsigned long end,
 	pgtbl_mod_mask mask = 0;
 
 	BUG_ON(addr >= end);
+	if (WARN_ON_ONCE(!PAGE_ALIGNED(addr) || !PAGE_ALIGNED(end)))
+		return;
+
 	pgd = pgd_offset_k(addr);
 	do {
 		next = pgd_addr_end(addr, end);
Signature · 1 line
2.25.1
Reviewed-by on v5 4/11
#6 lore raw

Re: [PATCH v5 04/11] mm/vmalloc: reject an unaligned range in __vunmap_range_noflush()

Quoted 21 lines · Hao Ge

The PTE loop stops only at addr == end. An unaligned end is never reached, so the loop keeps clearing past the range. An unaligned start clears the page below the range first. Both are caller bugs, and the damage is hard to attribute afterwards.

vmap only installs page aligned mappings, so an unaligned unmap request is never a legitimate teardown. Warn and bail out at the entry of __vunmap_range_noflush(), before any PTE is cleared.

---
 mm/vmalloc.c | 3 +++
 1 file changed, 3 insertions(+)
mm/vmalloc.c +0 -0
index a168f6f20f4f..e2b63cb1592b 100644

mm/vmalloc.c void __vunmap_range_noflush(unsigned long start, unsigned long end, · line 525

Quoted 3 lines
mm/vmalloc.c +0 -0
@@ -523,6 +523,9 @@ void __vunmap_range_noflush(unsigned long start, unsigned long end,
 	pgtbl_mod_mask mask = 0;
 
 	BUG_ON(addr >= end);

Nit: It is usually a practice to change the existing BUG_ON's to WARN_ON's too, if you are changing around that part.

mm/vmalloc.c void __vunmap_range_noflush(unsigned long start, unsigned long end, · line 527

Quoted 2 lines
mm/vmalloc.c +2 -0
+	if (WARN_ON_ONCE(!PAGE_ALIGNED(addr) || !PAGE_ALIGNED(end)))
+		return;
Quoted 4 lines
mm/vmalloc.c +1 -0
+
 	pgd = pgd_offset_k(addr);
 	do {
 		next = pgd_addr_end(addr, end);
author
#7 lore raw

PATCH v5 5/11 mm/vmalloc: only flush the cache after the vmap succeeded

In reply to Hao Ge

vmap_page_range(), __vmap_pages_range() and vmap_pages_range_batched() call flush_cache_vmap() before they even look at the error code. A failed mapping is never used, so the flush is pointless at best.

According to the Sashiko review it is worse than pointless on UML:
flush_cache_vmap() there is flush_tlb_kernel_range(), which syncs the PTEs into the host OS. Flushing a half-installed mapping therefore leaves host mappings behind for a range that was just freed again.

Only flush when the mapping, and for vmap_page_range() also the KMSAN metadata, is fully set up.

---
 mm/vmalloc.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
mm/vmalloc.c +8 -3
index e2b63cb1592b..d99d7710e1ba 100644
@@ -371,10 +371,12 @@ int vmap_page_range(unsigned long addr, unsigned long end,
 
 	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
 				 ioremap_max_page_shift);
-	flush_cache_vmap(addr, end);
 	if (!err)
 		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
 					       ioremap_max_page_shift);
+	if (!err)
+		flush_cache_vmap(addr, end);
+
 	return err;
 }
 
@@ -763,7 +765,9 @@ static int __vmap_pages_range(unsigned long addr, unsigned long end,
 	int err;
 
 	err = vmap_pages_range_noflush(addr, end, prot, pages, page_shift, gfp_mask);
-	flush_cache_vmap(addr, end);
+	if (!err)
+		flush_cache_vmap(addr, end);
+
 	return err;
 }
 
@@ -3729,7 +3733,8 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
 				pages + batch_idx, prev_shift);
 
 out:
-	flush_cache_vmap(addr, end);
+	if (!err)
+		flush_cache_vmap(addr, end);
 	return err;
 }
 
Signature · 1 line
2.25.1
author
#8 lore raw

PATCH v5 6/11 mm/vmalloc: undo what vmap_page_range() mapped on failure

In reply to Hao Ge

ioremap and arch setup code map their ranges with vmap_page_range(). When the walk fails half way, or the KMSAN metadata fails after the walk, the ptes installed so far stay behind. The caller frees the vm_area on the error path, so the next mapping that lands on the same range trips the BUG() in vmap_pte_range().

vmap_range_noflush() now takes a mapped_end argument that the walk levels advance as they install, and vmap_pte_range() BUG()s before it installs over a pte that is already set. The ptes below mapped_end are therefore all ours, while a pte above it can only have been put there by somebody else, for example when the caller's range overlaps an existing mapping. On a walk failure the undo is limited to [addr, mapped_end), going further would clear ptes that are not ours. If the walk completed and only the KMSAN metadata failed, the whole range was installed by this call and all of it is undone.

All callers are sleepable, the teardown passes can_resched = true.

---
 mm/vmalloc.c | 52 +++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 39 insertions(+), 13 deletions(-)
mm/vmalloc.c +39 -13
index d99d7710e1ba..5c1eacc2a0ac 100644
@@ -117,7 +117,8 @@ static __always_inline unsigned long vmap_set_ptes(pte_t *ptep,
 
 static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 			phys_addr_t phys_addr, pgprot_t prot,
-			unsigned int max_page_shift, pgtbl_mod_mask *mask)
+			unsigned int max_page_shift, pgtbl_mod_mask *mask,
+			unsigned long *mapped_end)
 {
 	pte_t *pte;
 	u64 pfn;
@@ -143,6 +144,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 
 		size = vmap_set_ptes(pte, addr, end, pfn, prot, max_page_shift);
 		nr_pages = PFN_DOWN(size);
+		*mapped_end = addr + size;
 	} while (pte += nr_pages, pfn += nr_pages, addr += size, addr != end);
 
 	lazy_mmu_mode_disable();
@@ -188,7 +190,8 @@ static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end,
 
 static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,
 			phys_addr_t phys_addr, pgprot_t prot,
-			unsigned int max_page_shift, pgtbl_mod_mask *mask)
+			unsigned int max_page_shift, pgtbl_mod_mask *mask,
+			unsigned long *mapped_end)
 {
 	pmd_t *pmd;
 	unsigned long next;
@@ -203,10 +206,12 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,
 		if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot,
 					max_page_shift)) {
 			*mask |= PGTBL_PMD_MODIFIED;
+			*mapped_end = next;
 			continue;
 		}
 
-		err = vmap_pte_range(pmd, addr, next, phys_addr, prot, max_page_shift, mask);
+		err = vmap_pte_range(pmd, addr, next, phys_addr, prot,
+				     max_page_shift, mask, mapped_end);
 		if (err)
 			break;
 	} while (pmd++, phys_addr += (next - addr), addr = next, addr != end);
@@ -245,7 +250,8 @@ static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end,
 
 static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,
 			phys_addr_t phys_addr, pgprot_t prot,
-			unsigned int max_page_shift, pgtbl_mod_mask *mask)
+			unsigned int max_page_shift, pgtbl_mod_mask *mask,
+			unsigned long *mapped_end)
 {
 	pud_t *pud;
 	unsigned long next;
@@ -260,10 +266,12 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,
 		if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot,
 					max_page_shift)) {
 			*mask |= PGTBL_PUD_MODIFIED;
+			*mapped_end = next;
 			continue;
 		}
 
-		err = vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask);
+		err = vmap_pmd_range(pud, addr, next, phys_addr, prot,
+				     max_page_shift, mask, mapped_end);
 		if (err)
 			break;
 	} while (pud++, phys_addr += (next - addr), addr = next, addr != end);
@@ -302,7 +310,8 @@ static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end,
 
 static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
 			phys_addr_t phys_addr, pgprot_t prot,
-			unsigned int max_page_shift, pgtbl_mod_mask *mask)
+			unsigned int max_page_shift, pgtbl_mod_mask *mask,
+			unsigned long *mapped_end)
 {
 	p4d_t *p4d;
 	unsigned long next;
@@ -317,10 +326,12 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
 		if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot,
 					max_page_shift)) {
 			*mask |= PGTBL_P4D_MODIFIED;
+			*mapped_end = next;
 			continue;
 		}
 
-		err = vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask);
+		err = vmap_pud_range(p4d, addr, next, phys_addr, prot,
+				     max_page_shift, mask, mapped_end);
 		if (err)
 			break;
 	} while (p4d++, phys_addr += (next - addr), addr = next, addr != end);
@@ -329,7 +340,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
 
 static int vmap_range_noflush(unsigned long addr, unsigned long end,
 			phys_addr_t phys_addr, pgprot_t prot,
-			unsigned int max_page_shift)
+			unsigned int max_page_shift, unsigned long *mapped_end)
 {
 	pgd_t *pgd;
 	unsigned long start;
@@ -346,11 +357,12 @@ static int vmap_range_noflush(unsigned long addr, unsigned long end,
 	BUG_ON(addr >= end);
 
 	start = addr;
+	*mapped_end = addr;
 	pgd = pgd_offset_k(addr);
 	do {
 		next = pgd_addr_end(addr, end);
 		err = vmap_p4d_range(pgd, addr, next, phys_addr, prot,
-					max_page_shift, &mask);
+					max_page_shift, &mask, mapped_end);
 		if (err)
 			break;
 	} while (pgd++, phys_addr += (next - addr), addr = next, addr != end);
@@ -364,18 +376,32 @@ static int vmap_range_noflush(unsigned long addr, unsigned long end,
 int vmap_page_range(unsigned long addr, unsigned long end,
 		    phys_addr_t phys_addr, pgprot_t prot)
 {
+	unsigned long mapped_end = addr;
 	int err;
 
 	if (WARN_ON_ONCE(!PAGE_ALIGNED(addr) || !PAGE_ALIGNED(end)))
 		return -EINVAL;
 
 	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
-				 ioremap_max_page_shift);
-	if (!err)
+				 ioremap_max_page_shift, &mapped_end);
+	if (err) {
+		/* Undo the part the walk installed before it failed. */
+		if (mapped_end > addr)
+			__vunmap_range_noflush(addr, mapped_end, true);
+	} else {
 		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
 					       ioremap_max_page_shift);
-	if (!err)
-		flush_cache_vmap(addr, end);
+		if (err) {
+			/*
+			 * Only the KMSAN metadata failed, the walk mapped
+			 * the whole range. Undo it all. The callers are
+			 * ioremap and arch setup code, always sleepable.
+			 */
+			__vunmap_range_noflush(addr, end, true);
+		} else {
+			flush_cache_vmap(addr, end);
+		}
+	}
 
 	return err;
 }
Signature · 1 line
2.25.1
author
#9 lore raw

PATCH v5 7/11 mm/vmalloc: undo what __vmap_pages_range_noflush() mapped on failure

In reply to Hao Ge

__vmap_pages_range_noflush() maps a struct page array and may fail half way. The ptes installed so far stay behind, so a retry of the same range, the __GFP_NOFAIL loop in __vmalloc_area_node() or the module tags populate, trips the WARN_ON(!pte_none()) in vmap_pages_pte_range() and fails with -EBUSY.

vmap_pages_range_noflush_walk() now reports how many pages it installed through its running index into the pages array, and __vmap_pages_range_noflush() undoes [addr, addr + mapped_pages * PAGE_SIZE) on failure. vmap_pages_pte_range() fails with -EBUSY instead of installing over a pte that is already set, so the ptes below that point are all ours, while a pte above it can only have been put there by somebody else, for example when the caller's range overlaps an existing mapping. Going further would clear ptes that are not ours.

This path can run with a non-blocking gfp mask, the undo passes gfpflags_allow_blocking() as can_resched.

---
 mm/kmsan/hooks.c  |  4 ++--
 mm/kmsan/shadow.c |  4 ++--
 mm/vmalloc.c      | 36 +++++++++++++++++++++++++++++-------
 mm/vmalloc.h      |  3 ++-
 4 files changed, 35 insertions(+), 12 deletions(-)
mm/kmsan/hooks.c +2 -2
index 69008b085658..d830a4e649f1 100644
@@ -208,7 +208,7 @@ int kmsan_ioremap_page_range(unsigned long start, unsigned long end,
 		mapped = __vmap_pages_range_noflush(
 			vmalloc_shadow(start + off),
 			vmalloc_shadow(start + off + PAGE_SIZE), prot, &shadow,
-			PAGE_SHIFT);
+			PAGE_SHIFT, gfp_mask);
 		if (mapped) {
 			err = mapped;
 			goto ret;
@@ -218,7 +218,7 @@ int kmsan_ioremap_page_range(unsigned long start, unsigned long end,
 		mapped = __vmap_pages_range_noflush(
 			vmalloc_origin(start + off),
 			vmalloc_origin(start + off + PAGE_SIZE), prot, &origin,
-			PAGE_SHIFT);
+			PAGE_SHIFT, gfp_mask);
 		if (mapped) {
 			err = mapped;
 			goto ret;
mm/kmsan/shadow.c +2 -2
index 212449db34a1..452760a333e4 100644
@@ -247,7 +247,7 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end,
 	origin_end = vmalloc_meta((void *)end, KMSAN_META_ORIGIN);
 	kmsan_enter_runtime();
 	mapped = __vmap_pages_range_noflush(shadow_start, shadow_end, prot,
-					    s_pages, page_shift);
+					    s_pages, page_shift, gfp_mask);
 	kmsan_leave_runtime();
 	if (mapped) {
 		err = mapped;
@@ -255,7 +255,7 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end,
 	}
 	kmsan_enter_runtime();
 	mapped = __vmap_pages_range_noflush(origin_start, origin_end, prot,
-					    o_pages, page_shift);
+					    o_pages, page_shift, gfp_mask);
 	kmsan_leave_runtime();
 	if (mapped) {
 		/* Undo the shadow mapping set up above. */
mm/vmalloc.c +29 -7
index 5c1eacc2a0ac..4f3c320a4849 100644
@@ -725,7 +725,8 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr,
  * may have contiguous chunks, as hinted by @shift.
  */
 static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end,
-		pgprot_t prot, struct page **pages, unsigned int shift)
+		pgprot_t prot, struct page **pages, unsigned int shift,
+		unsigned int *mapped_pages)
 {
 	unsigned long start = addr;
 	pgd_t *pgd;
@@ -748,6 +749,7 @@ static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end,
 	if (mask & ARCH_PAGE_TABLE_SYNC_MASK)
 		arch_sync_kernel_mappings(start, end);
 
+	*mapped_pages = nr;
 	return err;
 }
 
@@ -758,18 +760,36 @@ static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end,
  * The caller is responsible for calling flush_cache_vmap() after this
  * function returns successfully and before the addresses are accessed.
  *
+ * On failure the PTEs installed by this call are torn down before
+ * returning, so a retry sees a clean range; PTEs that were already
+ * there when the walk failed are left alone. When @gfp_mask cannot
+ * block, the teardown skips cond_resched(), it may run in atomic
+ * context.
+ *
  * This is an internal function only. Do not use outside mm/.
  */
 int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
-		pgprot_t prot, struct page **pages, unsigned int page_shift)
+		pgprot_t prot, struct page **pages, unsigned int page_shift,
+		gfp_t gfp_mask)
 {
+	int err;
+	unsigned int mapped_pages = 0;
+
 	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
 		return -EINVAL;
 
 	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC))
 		page_shift = PAGE_SHIFT;
 
-	return vmap_pages_range_noflush_walk(addr, end, prot, pages, page_shift);
+	err = vmap_pages_range_noflush_walk(addr, end, prot, pages, page_shift,
+					    &mapped_pages);
+
+	/* Undo the PTEs installed before the failure. */
+	if (err && mapped_pages)
+		__vunmap_range_noflush(addr, addr + (mapped_pages * PAGE_SIZE),
+				       gfpflags_allow_blocking(gfp_mask));
+
+	return err;
 }
 
 int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
@@ -781,7 +801,8 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
 
 	if (ret)
 		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift,
+					  gfp_mask);
 }
 
 static int __vmap_pages_range(unsigned long addr, unsigned long end,
@@ -3715,7 +3736,7 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
 		pgprot_t prot, struct page **pages)
 {
 	const unsigned int nr_pages = (end - addr) >> PAGE_SHIFT;
-	unsigned int prev_shift = 0, batch_idx = 0;
+	unsigned int prev_shift = 0, batch_idx = 0, nr_mapped;
 	unsigned long batch_start = addr, batch_end = addr;
 	int err;
 
@@ -3734,7 +3755,8 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
 
 		if (shift != prev_shift) {
 			err = vmap_pages_range_noflush_walk(batch_start, batch_end,
-					prot, pages + batch_idx, prev_shift);
+					prot, pages + batch_idx, prev_shift,
+					&nr_mapped);
 			if (err)
 				goto out;
 			prev_shift = shift;
@@ -3756,7 +3778,7 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
 	/* Remaining */
 	if (batch_start < end)
 		err = vmap_pages_range_noflush_walk(batch_start, end, prot,
-				pages + batch_idx, prev_shift);
+				pages + batch_idx, prev_shift, &nr_mapped);
 
 out:
 	if (!err)
mm/vmalloc.h +2 -1
index 4ec114a75f04..9ea28845f713 100644
@@ -40,7 +40,8 @@ void clear_vm_uninitialized_flag(struct vm_struct *vm);
 
 int __must_check __vmap_pages_range_noflush(unsigned long addr,
 		unsigned long end, pgprot_t prot,
-		struct page **pages, unsigned int page_shift);
+		struct page **pages, unsigned int page_shift,
+		gfp_t gfp_mask);
 
 void vunmap_range_noflush(unsigned long start, unsigned long end);
 
Signature · 1 line
2.25.1
#10 lore raw

Re: [PATCH v5 07/11] mm/vmalloc: undo what __vmap_pages_range_noflush() mapped on failure

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] mm/vmalloc: vmap_pages_range_noflush_walk() truncates nr into unsigned int --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm/vmalloc: vmap_pages_range_noflush_walk() truncates nr into unsigned int
--

commit f06c1baff5ffda8b5f9a1dce600b9d716bd0c648
Author: Hao Ge <[email protected]>

mm/vmalloc: undo what __vmap_pages_range_noflush() mapped on failure

This commit tracks the number of pages installed by vmap_pages_range_noflush_walk() and tears down those PTEs in __vmap_pages_range_noflush() when a mapping fails midway.

Quoted 49 lines
mm/vmalloc.c +14 -3
index 5c1eacc2a0ac3..4f3c320a48499 100644
@@ -725,7 +725,8 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr,
  * may have contiguous chunks, as hinted by @shift.
  */
 static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end,
-		pgprot_t prot, struct page **pages, unsigned int shift)
+		pgprot_t prot, struct page **pages, unsigned int shift,
+		unsigned int *mapped_pages)
 {
 	unsigned long start = addr;
 	pgd_t *pgd;
@@ -748,6 +749,7 @@ static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end,
 	if (mask & ARCH_PAGE_TABLE_SYNC_MASK)
 		arch_sync_kernel_mappings(start, end);
 
+	*mapped_pages = nr;
 	return err;
 }
 
@@ -758,18 +760,36 @@ static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end,
  * The caller is responsible for calling flush_cache_vmap() after this
  * function returns successfully and before the addresses are accessed.
  *
+ * On failure the PTEs installed by this call are torn down before
+ * returning, so a retry sees a clean range; PTEs that were already
+ * there when the walk failed are left alone. When @gfp_mask cannot
+ * block, the teardown skips cond_resched(), it may run in atomic
+ * context.
+ *
  * This is an internal function only. Do not use outside mm/.
  */
 int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
-		pgprot_t prot, struct page **pages, unsigned int page_shift)
+		pgprot_t prot, struct page **pages, unsigned int page_shift,
+		gfp_t gfp_mask)
 {
+	int err;
+	unsigned int mapped_pages = 0;
+
 	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
 		return -EINVAL;
 
 	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC))
 		page_shift = PAGE_SHIFT;
 
-	return vmap_pages_range_noflush_walk(addr, end, prot, pages, page_shift);

mm/vmalloc.c static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end, · line 793

Quoted 10 lines
mm/vmalloc.c +9 -0
+	err = vmap_pages_range_noflush_walk(addr, end, prot, pages, page_shift,
+					    &mapped_pages);
+
+	/* Undo the PTEs installed before the failure. */
+	if (err && mapped_pages)
+		__vunmap_range_noflush(addr, addr + (mapped_pages * PAGE_SIZE),
+				       gfpflags_allow_blocking(gfp_mask));
+
+	return err;
 }

[Severity: Low]
Should mapped_pages (and nr_mapped in vmap_pages_range_batched()) be unsigned long instead of unsigned int?

vmap_pages_range_noflush_walk() tracks nr as an unsigned long across the page-table walk, so storing it into an unsigned int truncates the count to 32 bits and would leave the unmap range incomplete if a walk ever failed past UINT_MAX pages.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
author
#11 lore raw

Re: [PATCH v5 07/11] mm/vmalloc: undo what __vmap_pages_range_noflush() mapped on failure

Quoted 79 lines · On 2026/11/17 15:15, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm/vmalloc: vmap_pages_range_noflush_walk() truncates nr into unsigned int
--

commit f06c1baff5ffda8b5f9a1dce600b9d716bd0c648
Author: Hao Ge <[email protected]>

mm/vmalloc: undo what __vmap_pages_range_noflush() mapped on failure

This commit tracks the number of pages installed by vmap_pages_range_noflush_walk() and tears down those PTEs in __vmap_pages_range_noflush() when a mapping fails midway.

Quoted 49 lines
mm/vmalloc.c +14 -3
index 5c1eacc2a0ac3..4f3c320a48499 100644
@@ -725,7 +725,8 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr,
  * may have contiguous chunks, as hinted by @shift.
  */
 static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end,
-		pgprot_t prot, struct page **pages, unsigned int shift)
+		pgprot_t prot, struct page **pages, unsigned int shift,
+		unsigned int *mapped_pages)
 {
 	unsigned long start = addr;
 	pgd_t *pgd;
@@ -748,6 +749,7 @@ static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end,
 	if (mask & ARCH_PAGE_TABLE_SYNC_MASK)
 		arch_sync_kernel_mappings(start, end);
 
+	*mapped_pages = nr;
 	return err;
 }
 
@@ -758,18 +760,36 @@ static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end,
  * The caller is responsible for calling flush_cache_vmap() after this
  * function returns successfully and before the addresses are accessed.
  *
+ * On failure the PTEs installed by this call are torn down before
+ * returning, so a retry sees a clean range; PTEs that were already
+ * there when the walk failed are left alone. When @gfp_mask cannot
+ * block, the teardown skips cond_resched(), it may run in atomic
+ * context.
+ *
  * This is an internal function only. Do not use outside mm/.
  */
 int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
-		pgprot_t prot, struct page **pages, unsigned int page_shift)
+		pgprot_t prot, struct page **pages, unsigned int page_shift,
+		gfp_t gfp_mask)
 {
+	int err;
+	unsigned int mapped_pages = 0;
+
 	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
 		return -EINVAL;
 
 	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC))
 		page_shift = PAGE_SHIFT;
 
-	return vmap_pages_range_noflush_walk(addr, end, prot, pages, page_shift);

mm/vmalloc.c static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end, · line 793

Quoted 10 lines
mm/vmalloc.c +9 -0
+	err = vmap_pages_range_noflush_walk(addr, end, prot, pages, page_shift,
+					    &mapped_pages);
+
+	/* Undo the PTEs installed before the failure. */
+	if (err && mapped_pages)
+		__vunmap_range_noflush(addr, addr + (mapped_pages * PAGE_SIZE),
+				       gfpflags_allow_blocking(gfp_mask));
+
+	return err;
 }

[Severity: Low]
Should mapped_pages (and nr_mapped in vmap_pages_range_batched()) be unsigned long instead of unsigned int?

Quoted 5 lines

vmap_pages_range_noflush_walk() tracks nr as an unsigned long across the page-table walk, so storing it into an unsigned int truncates the count to 32 bits and would leave the unmap range incomplete if a walk ever failed past UINT_MAX pages.

Sure, keeping them consistent does make sense.
I will hold this change and wait for more review comments on the rest of the patches.

author
#12 lore raw

PATCH v5 8/11 mm/vmalloc: drop the KMSAN metadata in vmap_pages_range_noflush() on failure

In reply to Hao Ge

vmap_pages_range_noflush() sets up the KMSAN metadata for the whole range before it starts mapping. If the mapping fails, the page tables undo themselves now, but the metadata stays behind, and the next attempt to map the same range hits the leftover shadow PTEs.

Tear the metadata down on the error path, it was fully set up before the mapping failed. The teardown follows the page tables and skips cond_resched() when the caller's gfp mask cannot block.

---
 mm/vmalloc.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
mm/vmalloc.c +12 -2
index 4f3c320a4849..28e65073624a 100644
@@ -801,8 +801,18 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
 
 	if (ret)
 		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift,
-					  gfp_mask);
+
+	ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift,
+					 gfp_mask);
+	/*
+	 * The page tables undo themselves on failure. Tear down the
+	 * metadata that was fully set up before the mapping failed.
+	 */
+	if (ret)
+		kmsan_vunmap_range_noflush(addr, end,
+					   gfpflags_allow_blocking(gfp_mask));
+
+	return ret;
 }
 
 static int __vmap_pages_range(unsigned long addr, unsigned long end,
Signature · 1 line
2.25.1
author
#13 lore raw

PATCH v5 9/11 mm/vmalloc: undo what vmap_pages_range_batched() mapped on failure

In reply to Hao Ge

vmap_pages_range_batched() maps the range batch by batch. When a batch fails, the batches before it are installed, and the failing one may have installed some of its pages. mapped_end follows the end of the last installed page, and the error path undoes [addr, mapped_end), never past the point where the walk stopped.

The KMSAN metadata is set up for the whole range up front, so it is torn down as a whole when the mapping fails, like in vmap_pages_range_noflush(). The only caller, vmap(), also calls vunmap() on its error path; the double teardown only walks over PTEs that are already cleared.

---
 mm/vmalloc.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)
mm/vmalloc.c +16 -6
index 28e65073624a..aa80bc273ec0 100644
@@ -3748,13 +3748,14 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
 	const unsigned int nr_pages = (end - addr) >> PAGE_SHIFT;
 	unsigned int prev_shift = 0, batch_idx = 0, nr_mapped;
 	unsigned long batch_start = addr, batch_end = addr;
+	unsigned long mapped_end = addr;
 	int err;
 
 	err = kmsan_vmap_pages_range_noflush(addr, end, prot, pages,
 					     PAGE_SHIFT, GFP_KERNEL);
 
 	if (err)
-		goto out;
+		return err;
 
 	for (unsigned int i = 0; i < nr_pages; ) {
 		unsigned int shift = PAGE_SHIFT +
@@ -3764,9 +3765,10 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
 			prev_shift = shift;
 
 		if (shift != prev_shift) {
-			err = vmap_pages_range_noflush_walk(batch_start, batch_end,
-					prot, pages + batch_idx, prev_shift,
-					&nr_mapped);
+			err = vmap_pages_range_noflush_walk(batch_start,
+					batch_end, prot, pages + batch_idx,
+					prev_shift, &nr_mapped);
+			mapped_end = batch_start + (nr_mapped * PAGE_SIZE);
 			if (err)
 				goto out;
 			prev_shift = shift;
@@ -3786,13 +3788,21 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
 	}
 
 	/* Remaining */
-	if (batch_start < end)
+	if (batch_start < end) {
 		err = vmap_pages_range_noflush_walk(batch_start, end, prot,
 				pages + batch_idx, prev_shift, &nr_mapped);
+		mapped_end = batch_start + (nr_mapped * PAGE_SIZE);
+	}
 
 out:
-	if (!err)
+	if (err) {
+		/* Undo the part the batches installed before they failed. */
+		if (mapped_end > addr)
+			__vunmap_range_noflush(addr, mapped_end, true);
+		kmsan_vunmap_range_noflush(addr, end, true);
+	} else {
 		flush_cache_vmap(addr, end);
+	}
 	return err;
 }
 
Signature · 1 line
2.25.1
author
#14 lore raw

PATCH v5 10/11 powerpc: drop redundant unmaps of failed vmap mappings

In reply to Hao Ge

vmap_page_range() and ioremap_page_range() clear the PTEs they installed when they fail, so the vunmap_range() in remap_isa_base() and ioremap_phb() has nothing left to remove.

---
 arch/powerpc/kernel/isa-bridge.c | 5 ++---
 arch/powerpc/kernel/pci_64.c     | 4 +---
 2 files changed, 3 insertions(+), 6 deletions(-)
arch/powerpc/kernel/isa-bridge.c +2 -3
index 5c064485197a..93029d85e6ea 100644
@@ -46,9 +46,8 @@ static void remap_isa_base(phys_addr_t pa, unsigned long size)
 	WARN_ON_ONCE(size & ~PAGE_MASK);
 
 	if (slab_is_available()) {
-		if (vmap_page_range(ISA_IO_BASE, ISA_IO_BASE + size, pa,
-				    pgprot_noncached(PAGE_KERNEL)))
-			vunmap_range(ISA_IO_BASE, ISA_IO_BASE + size);
+		vmap_page_range(ISA_IO_BASE, ISA_IO_BASE + size, pa,
+				pgprot_noncached(PAGE_KERNEL));
 	} else {
 		early_ioremap_range(ISA_IO_BASE, pa, size,
 				pgprot_noncached(PAGE_KERNEL));
arch/powerpc/kernel/pci_64.c +1 -3
index e27342ef128b..f74269636d5e 100644
@@ -139,10 +139,8 @@ void __iomem *ioremap_phb(phys_addr_t paddr, unsigned long size)
 
 	addr = (unsigned long)area->addr;
 	if (ioremap_page_range(addr, addr + size, paddr,
-			pgprot_noncached(PAGE_KERNEL))) {
-		vunmap_range(addr, addr + size);
+			pgprot_noncached(PAGE_KERNEL)))
 		return NULL;
-	}
 
 	return (void __iomem *)addr;
 }
Signature · 1 line
2.25.1
author
#15 lore raw

PATCH v5 11/11 mm/percpu: stop unmapping the CPU that failed to map

In reply to Hao Ge

__pcpu_map_pages() clears the PTEs it installed when it fails, so pcpu_map_pages() only has to tear down the CPUs it mapped before the failing one.

---
 mm/percpu-vm.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/percpu-vm.c +3 -2
index 509d8901835c..59d3a336a6bd 100644
@@ -252,10 +252,11 @@ static int pcpu_map_pages(struct pcpu_chunk *chunk, struct page **pages,
 	return 0;
 err:
 	for_each_possible_cpu(tcpu) {
-		__pcpu_unmap_pages(pcpu_chunk_addr(chunk, tcpu, page_start),
-				   page_end - page_start);
+		/* The failing CPU's partial mapping undoes itself */
 		if (tcpu == cpu)
 			break;
+		__pcpu_unmap_pages(pcpu_chunk_addr(chunk, tcpu, page_start),
+				   page_end - page_start);
 	}
 	pcpu_post_unmap_tlb_flush(chunk, page_start, page_end);
 	return err;
Signature · 1 line
2.25.1