Skip to contentSkip to conversation

mm/vmalloc: make the mapping functions undo their partial mappings

PATCH Superseded Hao Ge 4 participants 21 messages Oct 19 – Oct 23

Subsystems (6)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Hao Ge

author
#0 lore raw

v2 cover mm/vmalloc: make the mapping functions undo their partial mappings

__vmap_pages_range_noflush() and friends can install some PTEs before failing and leave them mapped, and the kernel callers do not agree on who cleans them up. pcpu_map_pages() and kmsan_ioremap_page_range() roll back what they mapped before the failure, while vm_module_tags_populate() and the __GFP_NOFAIL retry loop in __vmalloc_area_node() rely on the mapping functions cleaning up and do not call anything like vunmap_range() themselves. When the same range is mapped again, the attempt hits the leftovers and fails, with a BUG() in vmap_pte_range() for huge mappings and a warning on the small-page path.

After discussing with Suren and Ulad [1], we move the rollback into the mapping functions. A failed vmap leaves nothing mapped, and the callers no longer clean up after it.

[1] https://lore.kernel.org/all/[email protected]/

Patch 1 is the KMSAN part. The shadow and the origin of a range are mapped by two separate calls, so the rollback there has to cover both. Patch 2 is its disabled-KMSAN counterpart. Patch 3 adds the rollback to __vmap_pages_range_noflush() and vmap_page_range(). Patches 4 and 5 drop the powerpc and percpu cleanups that are not needed anymore.

Changes since v1:
- Add patch 2. Sashiko reported that an empty range that hits patch 3's WARN_ON_ONCE() trips __vunmap_range_noflush()'s BUG_ON() through kmsan_vunmap_range_noflush() when kmsan=0. Patch 2 adds the kmsan_enabled check the unmap side was missing.
- Pick up Reviewed-by tags from Suren and Ulad on patches 1, 4 and 5.
- Add Cc: stable to patch 3.

v1 link: https://lore.kernel.org/all/[email protected]/

Hao Ge (5):
mm/kmsan: undo the shadow mapping when the origin mapping fails mm/kmsan: skip vunmap_range_noflush when KMSAN is disabled mm/vmalloc: undo partial mappings inside the mapping functions
powerpc: drop redundant unmaps of failed vmap mappings mm/percpu: stop unmapping the CPU that failed to map

 arch/powerpc/kernel/isa-bridge.c |  5 ++--
 arch/powerpc/kernel/pci_64.c     |  4 +--
 mm/kmsan/hooks.c                 |  3 +++
 mm/kmsan/shadow.c                |  4 +++
 mm/percpu-vm.c                   |  5 ++--
 mm/vmalloc.c                     | 44 +++++++++++++++++++++++---------
 6 files changed, 45 insertions(+), 20 deletions(-)
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH v2 1/5 mm/kmsan: undo the shadow mapping when the origin mapping fails

kmsan_vmap_pages_range_noflush() first maps the shadow pages and then the origin pages. If the second mapping fails, the first one is left mapped, and the callers do not roll it back. The next vmap of the same metadata range hits the stale PTEs again and BUG()s on the huge mapping path, or gets -EBUSY with a WARN_ON() on the small page one.

Undo the shadow mapping on that error path, the same way kmsan_ioremap_page_range() cleans up after a partial failure. __vunmap_range_noflush() only clears the PTEs; nothing has touched the shadow mapping, so no TLB flush is needed.

Fixes 47ebd0310e89 ("mm: kmsan: handle alloc failures in kmsan_vmap_pages_range_noflush()")
---
 mm/kmsan/shadow.c | 4 ++++
 1 file changed, 4 insertions(+)
mm/kmsan/shadow.c +4 -0
index 0c88d89bf0d6..2166086d3dc3 100644
@@ -258,6 +258,10 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end,
 					    o_pages, page_shift);
 	kmsan_leave_runtime();
 	if (mapped) {
+		/* Undo the shadow mapping set up above. */
+		kmsan_enter_runtime();
+		__vunmap_range_noflush(shadow_start, shadow_end);
+		kmsan_leave_runtime();
 		err = mapped;
 		goto ret;
 	}
Signature · 1 line
2.25.1
author
#3 lore raw

PATCH v2 2/5 mm/kmsan: skip vunmap_range_noflush when KMSAN is disabled

In reply to Hao Ge

kmsan_vmap_pages_range_noflush() returns 0 when kmsan_enabled is false. The unmap side does not, so it still walks the shadow and origin page tables even though KMSAN never installed any metadata there.

Add the matching early return.

---
 mm/kmsan/hooks.c | 3 +++
 1 file changed, 3 insertions(+)
mm/kmsan/hooks.c +3 -0
index a706666db097..107f66841274 100644
@@ -138,6 +138,9 @@ static unsigned long vmalloc_origin(unsigned long addr)
 
 void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end)
 {
+	if (!kmsan_enabled)
+		return;
+
 	__vunmap_range_noflush(vmalloc_shadow(start), vmalloc_shadow(end));
 	__vunmap_range_noflush(vmalloc_origin(start), vmalloc_origin(end));
 	flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end));
Signature · 1 line
2.25.1
author
#7 lore raw

PATCH v2 3/5 mm/vmalloc: undo partial mappings inside the mapping functions

In reply to Hao Ge

__vmap_pages_range_noflush() and friends can install some PTEs before failing and leave them mapped, and the kernel callers do not agree on who cleans them up. pcpu_map_pages() and kmsan_ioremap_page_range() roll back what they mapped before the failure, while vm_module_tags_populate() and the __GFP_NOFAIL retry loop in __vmalloc_area_node() rely on the mapping functions cleaning up and do not call anything like vunmap_range() themselves. When the same range is mapped again, the attempt hits the leftovers and fails, with a BUG() in vmap_pte_range() for huge mappings and a warning on the small-page path.

After discussing with Suren and Ulad, we decided to put the rollback into the entry points of the vmap API rather than into every low-level helper. __vmap_pages_range_noflush() undoes the whole range it was asked to map when it fails, and vmap_page_range() undoes its range for the ioremap-style mappings.

The rollback is __vunmap_range_noflush(), it only clears the PTEs, no TLB flush, nothing has touched these mappings. vmap_pages_range_noflush() drops the KMSAN metadata when the data mapping fails, and an empty range bails out with -EINVAL now, which used to BUG_ON() on the small-page path and map nothing at all on the huge-page one.

---
 mm/vmalloc.c | 44 ++++++++++++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 12 deletions(-)
mm/vmalloc.c +32 -12
index 44e0eb32708f..75aa5acf5566 100644
@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,
 	if (!err)
 		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
 					       ioremap_max_page_shift);
+	if (err)
+		__vunmap_range_noflush(addr, end);
+
 	return err;
 }
 
@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
 		pgprot_t prot, struct page **pages, unsigned int page_shift)
 {
 	unsigned int i, nr = (end - addr) >> PAGE_SHIFT;
+	unsigned long start = addr;
+	int err = 0;
+
+	if (WARN_ON_ONCE(addr >= end))
+		return -EINVAL;
 
 	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
 		return -EINVAL;
 
 	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
-			page_shift == PAGE_SHIFT)
-		return vmap_small_pages_range_noflush(addr, end, prot, pages);
-
-	for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
-		int err;
-
-		err = vmap_range_noflush(addr, addr + (1UL << page_shift),
+			page_shift == PAGE_SHIFT) {
+		err = vmap_small_pages_range_noflush(addr, end, prot, pages);
+	} else {
+		for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
+			err = vmap_range_noflush(addr, addr + (1UL << page_shift),
 					page_to_phys(pages[i]), prot,
 					page_shift);
-		if (err)
-			return err;
+			if (err)
+				break;
 
-		addr += 1UL << page_shift;
+			addr += 1UL << page_shift;
+		}
 	}
 
-	return 0;
+	/* Undo the PTEs installed before the failure. */
+	if (err)
+		__vunmap_range_noflush(start, end);
+
+	return err;
 }
 
 int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
 
 	if (ret)
 		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+
+	ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+	/*
+	 * The page tables undo themselves on failure. Tear down the
+	 * metadata that was fully set up before the mapping failed.
+	 */
+	if (ret)
+		kmsan_vunmap_range_noflush(addr, end);
+
+	return ret;
 }
 
 static int __vmap_pages_range(unsigned long addr, unsigned long end,
Signature · 1 line
2.25.1
author
#18 lore raw

PATCH v2 4/5 powerpc: drop redundant unmaps of failed vmap mappings

In reply to Hao Ge

vmap_page_range() and ioremap_page_range() clear the PTEs they installed when they fail, so the vunmap_range() in remap_isa_base() and ioremap_phb() has nothing left to remove.

---
 arch/powerpc/kernel/isa-bridge.c | 5 ++---
 arch/powerpc/kernel/pci_64.c     | 4 +---
 2 files changed, 3 insertions(+), 6 deletions(-)
arch/powerpc/kernel/isa-bridge.c +2 -3
index 5c064485197a..93029d85e6ea 100644
@@ -46,9 +46,8 @@ static void remap_isa_base(phys_addr_t pa, unsigned long size)
 	WARN_ON_ONCE(size & ~PAGE_MASK);
 
 	if (slab_is_available()) {
-		if (vmap_page_range(ISA_IO_BASE, ISA_IO_BASE + size, pa,
-				    pgprot_noncached(PAGE_KERNEL)))
-			vunmap_range(ISA_IO_BASE, ISA_IO_BASE + size);
+		vmap_page_range(ISA_IO_BASE, ISA_IO_BASE + size, pa,
+				pgprot_noncached(PAGE_KERNEL));
 	} else {
 		early_ioremap_range(ISA_IO_BASE, pa, size,
 				pgprot_noncached(PAGE_KERNEL));
arch/powerpc/kernel/pci_64.c +1 -3
index e27342ef128b..f74269636d5e 100644
@@ -139,10 +139,8 @@ void __iomem *ioremap_phb(phys_addr_t paddr, unsigned long size)
 
 	addr = (unsigned long)area->addr;
 	if (ioremap_page_range(addr, addr + size, paddr,
-			pgprot_noncached(PAGE_KERNEL))) {
-		vunmap_range(addr, addr + size);
+			pgprot_noncached(PAGE_KERNEL)))
 		return NULL;
-	}
 
 	return (void __iomem *)addr;
 }
Signature · 1 line
2.25.1
author
#20 lore raw

PATCH v2 5/5 mm/percpu: stop unmapping the CPU that failed to map

In reply to Hao Ge

__pcpu_map_pages() clears the PTEs it installed when it fails, so pcpu_map_pages() only has to tear down the CPUs it mapped before the failing one.

---
 mm/percpu-vm.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/percpu-vm.c +3 -2
index 509d8901835c..59d3a336a6bd 100644
@@ -252,10 +252,11 @@ static int pcpu_map_pages(struct pcpu_chunk *chunk, struct page **pages,
 	return 0;
 err:
 	for_each_possible_cpu(tcpu) {
-		__pcpu_unmap_pages(pcpu_chunk_addr(chunk, tcpu, page_start),
-				   page_end - page_start);
+		/* The failing CPU's partial mapping undoes itself */
 		if (tcpu == cpu)
 			break;
+		__pcpu_unmap_pages(pcpu_chunk_addr(chunk, tcpu, page_start),
+				   page_end - page_start);
 	}
 	pcpu_post_unmap_tlb_flush(chunk, page_start, page_end);
 	return err;
Signature · 1 line
2.25.1
Reviewed-by on v2 1/5
#2 lore raw

Re: [PATCH v2 1/5] mm/kmsan: undo the shadow mapping when the origin mapping fails

In reply to Hao Ge
Quoted 11 lines · Hao Ge

kmsan_vmap_pages_range_noflush() first maps the shadow pages and then the origin pages. If the second mapping fails, the first one is left mapped, and the callers do not roll it back. The next vmap of the same metadata range hits the stale PTEs again and BUG()s on the huge mapping path, or gets -EBUSY with a WARN_ON() on the small page one.

Undo the shadow mapping on that error path, the same way kmsan_ioremap_page_range() cleans up after a partial failure. __vunmap_range_noflush() only clears the PTEs; nothing has touched the shadow mapping, so no TLB flush is needed.

Quoted 5 lines
Fixes 47ebd0310e89 ("mm: kmsan: handle alloc failures in kmsan_vmap_pages_range_noflush()")

---

Quoted 18 lines
 mm/kmsan/shadow.c | 4 ++++
 1 file changed, 4 insertions(+)
mm/kmsan/shadow.c +4 -0
index 0c88d89bf0d6..2166086d3dc3 100644
@@ -258,6 +258,10 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end,
 					    o_pages, page_shift);
 	kmsan_leave_runtime();
 	if (mapped) {
+		/* Undo the shadow mapping set up above. */
+		kmsan_enter_runtime();
+		__vunmap_range_noflush(shadow_start, shadow_end);
+		kmsan_leave_runtime();
 		err = mapped;
 		goto ret;
 	}
#4 lore raw

Re: [PATCH v2 2/5] mm/kmsan: skip vunmap_range_noflush when KMSAN is disabled

In reply to Hao Ge
Quoted 9 lines · Hao Ge

kmsan_vmap_pages_range_noflush() returns 0 when kmsan_enabled is false. The unmap side does not, so it still walks the shadow and origin page tables even though KMSAN never installed any metadata there.

Add the matching early return.

Cc stable without fixes?

I guess without the fix nothing is actually wrong, except doing wasteful work?

Quoted 19 lines
---
 mm/kmsan/hooks.c | 3 +++
 1 file changed, 3 insertions(+)
mm/kmsan/hooks.c +3 -0
index a706666db097..107f66841274 100644
@@ -138,6 +138,9 @@ static unsigned long vmalloc_origin(unsigned long addr)
 
 void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end)
 {
+	if (!kmsan_enabled)
+		return;
+
 	__vunmap_range_noflush(vmalloc_shadow(start), vmalloc_shadow(end));
 	__vunmap_range_noflush(vmalloc_origin(start), vmalloc_origin(end));
 	flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end));
author
#5 lore raw

Re: [PATCH v2 2/5] mm/kmsan: skip vunmap_range_noflush when KMSAN is disabled

Hi Dev

Quoted 8 lines · Dev Jain
Quoted 5 lines · Hao Ge

kmsan_vmap_pages_range_noflush() returns 0 when kmsan_enabled is false. The unmap side does not, so it still walks the shadow and origin page tables even though KMSAN never installed any metadata there.

Quoted 6 lines
Quoted 4 lines

Add the matching early return.

Cc stable without fixes?

Oops, my bad. The Cc stable was added mistakenly.

Quoted 5 lines

I guess without the fix nothing is actually wrong, except doing wasteful work?

That was true before, but not anymore after Patch 3 (mm/vmalloc: undo partial mappings inside the mapping functions). vmap_pages_range_noflush() now does:

int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
pgprot_t prot, struct page **pages, unsigned int page_shift, gfp_t gfp_mask)

{
        int ret = kmsan_vmap_pages_range_noflush(addr, end, prot, pages,
                                                page_shift, gfp_mask);

        if (ret)
                return ret;

        ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
        /*
         * The page tables undo themselves on failure. Tear down the
         * metadata that was fully set up before the mapping failed.
         */
        if (ret)
                kmsan_vunmap_range_noflush(addr, end);

        return ret;
}

Sashiko report: with !kmsan_enabled and a caller passing an empty range (addr >= end) the flow is:

1. kmsan_vmap_pages_range_noflush() returns 0 at its first check, it never looks at the range.

2. __vmap_pages_range_noflush() hits the new WARN_ON_ONCE(addr >= end) and returns -EINVAL.

3. ret is non-zero now, so the rollback calls kmsan_vunmap_range_noflush(addr, end), which unconditionally does

	__vunmap_range_noflush(vmalloc_shadow(start),vmalloc_shadow(end));

vmalloc_shadow() is a fixed offset, so the shadow range is empty too, and __vunmap_range_noflush() starts with BUG_ON(addr >= end). Panic.

So the guard is not about the empty range itself: a caller passing an empty range is buggy, and on today's kernels it already BUG_ON() in vmap_small_pages_range_noflush() on the small-page path. Patch 3 deliberately turns that into WARN_ON_ONCE() + -EINVAL, and this patch makes sure the new metadata rollback does not turn that recoverable error back into a panic when KMSAN is disabled.

But it's only theoretical. We haven't hit this case in real usage. We add this guard merely to reduce unnecessary work, and incidentally harden Patch 3.

Thanks for pointing this out.

Best Regards
Hao

Quoted 20 lines
Quoted 19 lines
---
 mm/kmsan/hooks.c | 3 +++
 1 file changed, 3 insertions(+)
mm/kmsan/hooks.c +3 -0
index a706666db097..107f66841274 100644
@@ -138,6 +138,9 @@ static unsigned long vmalloc_origin(unsigned long addr)
 
 void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end)
 {
+	if (!kmsan_enabled)
+		return;
+
 	__vunmap_range_noflush(vmalloc_shadow(start), vmalloc_shadow(end));
 	__vunmap_range_noflush(vmalloc_origin(start), vmalloc_origin(end));
 	flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end));
Reviewed-by on v2 2/5
#6 lore raw

Re: [PATCH v2 2/5] mm/kmsan: skip vunmap_range_noflush when KMSAN is disabled

In reply to Hao Ge
Quoted 11 lines · Hao Ge

kmsan_vmap_pages_range_noflush() returns 0 when kmsan_enabled is false. The unmap side does not, so it still walks the shadow and origin page tables even though KMSAN never installed any metadata there.

Add the matching early return.

---

Quoted 17 lines
 mm/kmsan/hooks.c | 3 +++
 1 file changed, 3 insertions(+)
mm/kmsan/hooks.c +3 -0
index a706666db097..107f66841274 100644
@@ -138,6 +138,9 @@ static unsigned long vmalloc_origin(unsigned long addr)
 
 void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end)
 {
+	if (!kmsan_enabled)
+		return;
+
 	__vunmap_range_noflush(vmalloc_shadow(start), vmalloc_shadow(end));
 	__vunmap_range_noflush(vmalloc_origin(start), vmalloc_origin(end));
 	flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end));
#8 lore raw

Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions

In reply to Hao Ge
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] mm/vmalloc: infinite loop in vunmap_pte_range() on unaligned error rollback - [High] mm/vmalloc…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop in vunmap_pte_range() on unaligned error rollback
- [High] mm/vmalloc: __vmap_pages_range_noflush() calls cond_resched() in atomic context
- [Medium] mm/vmalloc: vmap_page_range() calls flush_cache_vmap() before error rollback --

commit 043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>

mm/vmalloc: undo partial mappings inside the mapping functions

This commit adds rollback logic using __vunmap_range_noflush() and kmsan_vunmap_range_noflush() into vmap_page_range(), __vmap_pages_range_noflush(), and vmap_pages_range_noflush() when page table installation or KMSAN metadata setup fails. It also adds range and page_shift validation in __vmap_pages_range_noflush().

mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370

Quoted 13 lines
mm/vmalloc.c +3 -0
index 44e0eb32708f1..75aa5acf55660 100644
@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,
 	if (!err)
 		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
 					       ioremap_max_page_shift);
+	if (err)
+		__vunmap_range_noflush(addr, end);
+
 	return err;
 }

[Severity: High]
What happens here when vmap_range_noflush() returns -EINVAL for a range where end - addr is not PAGE_SIZE-aligned?

In vmap_pte_range(), commit a739e6b557af09 added a check to bail out on unaligned ranges specifically to avoid an infinite loop:

mm/vmalloc.c:vmap_pte_range() {
	...
	if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr)))
		return -EINVAL;
}

This can be reached on 64KB page kernels via pci_remap_iospace() -> vmap_page_range() with 16KB/32KB/48KB PCI I/O ranges (for example via loongson_pci_probe() or acpi_pci_root_remap_iospace()).

When vmap_pte_range() returns -EINVAL, vmap_page_range() now calls __vunmap_range_noflush(addr, end) with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as the PCI_IOBASE PMD populated during early boot by reserve_pio_range(), or an earlier PMD mapped in the same call), __vunmap_range_noflush() descends into vunmap_pte_range(), which has no alignment check:

mm/vmalloc.c:vunmap_pte_range() {
	...
	do {
		...
			ptent = ptep_get_and_clear(&init_mm, addr, pte);
		WARN_ON(!pte_none(ptent) && !pte_present(ptent));
	} while (pte += (size >> PAGE_SHIFT), addr += size, addr != end);
}

Because end - addr is not a multiple of PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?

[Severity: Medium]
Also in vmap_page_range(), flush_cache_vmap(addr, end) is called unconditionally before checking err and before rolling back with __vunmap_range_noflush(addr, end):

mm/vmalloc.c:vmap_page_range() {
	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
				 ioremap_max_page_shift);
	flush_cache_vmap(addr, end);
	if (!err)
		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
					       ioremap_max_page_shift);
	if (err)
		__vunmap_range_noflush(addr, end);

	return err;
}

On UML, flush_cache_vmap is defined as flush_tlb_kernel_range:

arch/um/include/asm/cacheflush.h:
#define flush_cache_vmap flush_tlb_kernel_range

When vmap_range_noflush() fails partway through or kmsan_ioremap_page_range() fails, flush_cache_vmap() runs while the partially installed PTEs are still pte_present(), causing update_pte_range() in arch/um/kernel/tlb.c to map them into the host OS via ops->mmap() and clear _PAGE_NEEDSYNC:

arch/um/kernel/tlb.c:update_pte_range() {
	...
		if (pte_present(*pte)) {
			...
			ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE,
					prot, fd, offset);
		} else
			ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE);

		*pte = pte_mkuptodate(*pte);
}

When __vunmap_range_noflush(addr, end) then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Should flush_cache_vmap(addr, end) only be called after both vmap_range_noflush() and kmsan_ioremap_page_range() succeed?

Quoted 35 lines
mm/vmalloc.c +13 -10
@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
 		pgprot_t prot, struct page **pages, unsigned int page_shift)
 {
 	unsigned int i, nr = (end - addr) >> PAGE_SHIFT;
+	unsigned long start = addr;
+	int err = 0;
+
+	if (WARN_ON_ONCE(addr >= end))
+		return -EINVAL;
 
 	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
 		return -EINVAL;
 
 	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
-			page_shift == PAGE_SHIFT)
-		return vmap_small_pages_range_noflush(addr, end, prot, pages);
-
-	for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
-		int err;
-
-		err = vmap_range_noflush(addr, addr + (1UL << page_shift),
+			page_shift == PAGE_SHIFT) {
+		err = vmap_small_pages_range_noflush(addr, end, prot, pages);
+	} else {
+		for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
+			err = vmap_range_noflush(addr, addr + (1UL << page_shift),
 					page_to_phys(pages[i]), prot,
 					page_shift);
-		if (err)
-			return err;
+			if (err)
+				break;
 
-		addr += 1UL << page_shift;
+			addr += 1UL << page_shift;

mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 718

Quoted 10 lines
mm/vmalloc.c +6 -1
+		}
 	}
 
-	return 0;
+	/* Undo the PTEs installed before the failure. */
+	if (err)
+		__vunmap_range_noflush(start, end);
+
+	return err;
 }

[Severity: High]
Similarly here, __vmap_pages_range_noflush() checks addr >= end, but does not check PAGE_ALIGNED(addr) or PAGE_ALIGNED(end).

If a caller passes a sub-page or unaligned range (for example, 0 < end - addr < PAGE_SIZE), nr = (end - addr) >> PAGE_SHIFT evaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely in vmap_pages_pte_range(). And if the huge-page path fails on an unaligned range, __vunmap_range_noflush(start, end) will hit the same infinite loop in vunmap_pte_range().

Should both vmap_page_range() and __vmap_pages_range_noflush() validate PAGE_ALIGNED(addr) && PAGE_ALIGNED(end) upfront before attempting to map or unmap?

Quoted 8 lines
mm/vmalloc.c +1 -1
 int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
 
 	if (ret)
 		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+

mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 739

Quoted 10 lines
mm/vmalloc.c +9 -0
+	ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+	/*
+	 * The page tables undo themselves on failure. Tear down the
+	 * metadata that was fully set up before the mapping failed.
+	 */
+	if (ret)
+		kmsan_vunmap_range_noflush(addr, end);
+
+	return ret;
 }

[Severity: High]
Can calling __vunmap_range_noflush() and kmsan_vunmap_range_noflush() here trigger a sleeping-in-atomic-context bug when __vmalloc() or kvmalloc() is called with GFP_ATOMIC or GFP_NOWAIT?

__vmalloc_area_node() supports non-blocking allocations by wrapping __vmap_pages_range() in memalloc_apply_gfp_scope(gfp_mask):

mm/vmalloc.c:__vmalloc_area_node() {
	...
	flags = memalloc_apply_gfp_scope(gfp_mask);
	do {
		ret = __vmap_pages_range(addr, addr + size, prot, area->pages,
				page_shift, nested_gfp);
	} while ((gfp_mask & __GFP_NOFAIL) && (ret < 0));
	memalloc_restore_gfp_scope(flags);
}

When a GFP_ATOMIC or GFP_NOWAIT allocation runs in atomic context (such as under a spinlock, preempt_disable(), or rcu_read_lock()) and page table allocation fails in vmap_small_pages_range_noflush() after at least one PMD in [start, end) is populated (or after kmsan_vmap_pages_range_noflush() populated the KMSAN shadow/origin ranges), __vmap_pages_range_noflush() and kmsan_vunmap_range_noflush() synchronously call __vunmap_range_noflush().

__vunmap_range_noflush() walks the populated PMDs via vunmap_pmd_range(), which unconditionally calls cond_resched():

mm/vmalloc.c:vunmap_pmd_range() {
	...
	do {
		next = pmd_addr_end(addr, end);
		if (pmd_none_or_clear_bad(pmd))
			continue;
		vunmap_pte_range(pmd, addr, next, mask);

		cond_resched();
	} while (pmd++, addr = next, addr != end);
}

Commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct") deferred error-path cleanup in __vmalloc_area_node() to a workqueue specifically because unmapping on failure can sleep. Doesn't calling __vunmap_range_noflush() synchronously inside __vmap_pages_range_noflush() and vmap_pages_range_noflush() reintroduce cond_resched() in atomic context?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#19 lore raw

Re: [PATCH v2 4/5] powerpc: drop redundant unmaps of failed vmap mappings

In reply to Hao Ge
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] powerpc/pci: leak of vm_struct in ioremap_phb() when ioremap_page_range() fails --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] powerpc/pci: leak of vm_struct in ioremap_phb() when ioremap_page_range() fails --

commit 293efb97e53f2ab49d30288da6e26fcbf75e032f
Author: Hao Ge <[email protected]>

powerpc: drop redundant unmaps of failed vmap mappings

This commit removes vunmap_range() calls from the error paths of remap_isa_base() and ioremap_phb(), since vmap_page_range() and ioremap_page_range() already clean up their installed PTEs upon failure.

Quoted 6 lines
arch/powerpc/kernel/pci_64.c +0 -0
index e27342ef128b8..f74269636d5ec 100644
@@ -139,10 +139,8 @@ void __iomem *ioremap_phb(phys_addr_t paddr, unsigned long size)
 

arch/powerpc/kernel/pci_64.c void __iomem *ioremap_phb(phys_addr_t paddr, unsigned long size) · line 146

Quoted 10 lines
arch/powerpc/kernel/pci_64.c +1 -3
 	addr = (unsigned long)area->addr;
 	if (ioremap_page_range(addr, addr + size, paddr,
-			pgprot_noncached(PAGE_KERNEL))) {
-		vunmap_range(addr, addr + size);
+			pgprot_noncached(PAGE_KERNEL)))
 		return NULL;
-	}
 
 	return (void __iomem *)addr;
 }

[Severity: Medium]
Does ioremap_phb() still leak area and its reserved virtual address range in [PHB_IO_BASE, PHB_IO_END) when ioremap_page_range() fails?

Prior to calling ioremap_page_range(), ioremap_phb() allocates a virtual memory area:

arch/powerpc/kernel/pci_64.c:ioremap_phb() {
    ...
	area = __get_vm_area_caller(size, VM_IOREMAP, PHB_IO_BASE, PHB_IO_END,
				    __builtin_return_address(0));
	if (!area)
		return NULL;
    ...
}

When ioremap_page_range() fails (for example, -ENOMEM when allocating page tables in vmap_range_noflush()), vmap_page_range() rolls back the PTEs via __vunmap_range_noflush(), but it does not free or unlink the struct vm_struct and struct vmap_area allocated by __get_vm_area_caller().

Should this error path call free_vm_area(area) before returning NULL, matching generic_ioremap_prot() in mm/ioremap.c?

mm/ioremap.c:generic_ioremap_prot() {
    ...
	if (ioremap_page_range(vaddr, vaddr + size, phys_addr, prot)) {
		free_vm_area(area);
		return NULL;
	}
    ...
}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
#16 lore raw

Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions

In reply to Hao Ge
Quoted 47 lines · Hao Ge

__vmap_pages_range_noflush() and friends can install some PTEs before failing and leave them mapped, and the kernel callers do not agree on who cleans them up. pcpu_map_pages() and kmsan_ioremap_page_range() roll back what they mapped before the failure, while vm_module_tags_populate() and the __GFP_NOFAIL retry loop in __vmalloc_area_node() rely on the mapping functions cleaning up and do not call anything like vunmap_range() themselves. When the same range is mapped again, the attempt hits the leftovers and fails, with a BUG() in vmap_pte_range() for huge mappings and a warning on the small-page path.

After discussing with Suren and Ulad, we decided to put the rollback into the entry points of the vmap API rather than into every low-level helper. __vmap_pages_range_noflush() undoes the whole range it was asked to map when it fails, and vmap_page_range() undoes its range for the ioremap-style mappings.

The rollback is __vunmap_range_noflush(), it only clears the PTEs, no TLB flush, nothing has touched these mappings. vmap_pages_range_noflush() drops the KMSAN metadata when the data mapping fails, and an empty range bails out with -EINVAL now, which used to BUG_ON() on the small-page path and map nothing at all on the huge-page one.

---
 mm/vmalloc.c | 44 ++++++++++++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 12 deletions(-)
mm/vmalloc.c +3 -0
index 44e0eb32708f..75aa5acf5566 100644
@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,
 	if (!err)
 		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
 					       ioremap_max_page_shift);
+	if (err)
+		__vunmap_range_noflush(addr, end);
+
 	return err;
 }
 

mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 693

Quoted 8 lines
mm/vmalloc.c +5 -0
@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
 		pgprot_t prot, struct page **pages, unsigned int page_shift)
 {
 	unsigned int i, nr = (end - addr) >> PAGE_SHIFT;
+	unsigned long start = addr;
+	int err = 0;
+
+	if (WARN_ON_ONCE(addr >= end))
+		return -EINVAL;

IIUC this is a general change independent of this patch? Caller could pass addr >= end and if we take the huge mapping path, we lose the sanity check.

So this could become a separate patch.

Quoted 56 lines
mm/vmalloc.c +24 -12
 	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
 		return -EINVAL;
 
 	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
-			page_shift == PAGE_SHIFT)
-		return vmap_small_pages_range_noflush(addr, end, prot, pages);
-
-	for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
-		int err;
-
-		err = vmap_range_noflush(addr, addr + (1UL << page_shift),
+			page_shift == PAGE_SHIFT) {
+		err = vmap_small_pages_range_noflush(addr, end, prot, pages);
+	} else {
+		for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
+			err = vmap_range_noflush(addr, addr + (1UL << page_shift),
 					page_to_phys(pages[i]), prot,
 					page_shift);
-		if (err)
-			return err;
+			if (err)
+				break;
 
-		addr += 1UL << page_shift;
+			addr += 1UL << page_shift;
+		}
 	}
 
-	return 0;
+	/* Undo the PTEs installed before the failure. */
+	if (err)
+		__vunmap_range_noflush(start, end);
+
+	return err;
 }
 
 int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
 
 	if (ret)
 		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+
+	ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+	/*
+	 * The page tables undo themselves on failure. Tear down the
+	 * metadata that was fully set up before the mapping failed.
+	 */
+	if (ret)
+		kmsan_vunmap_range_noflush(addr, end);
+
+	return ret;
 }
 
 static int __vmap_pages_range(unsigned long addr, unsigned long end,
author
#9 lore raw

Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions

In reply to [email protected]

Hi Suren and Ulad

Sashiko reported several issues, and they all look real. I have some ideas for fixing the first few;
of course, please correct me if I get anything wrong.

I'd like to discuss the last issue with you all.

Quoted 64 lines · On 2026/10/20 13:45, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop in vunmap_pte_range() on unaligned error rollback
- [High] mm/vmalloc: __vmap_pages_range_noflush() calls cond_resched() in atomic context
- [Medium] mm/vmalloc: vmap_page_range() calls flush_cache_vmap() before error rollback --

commit 043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>

mm/vmalloc: undo partial mappings inside the mapping functions

This commit adds rollback logic using __vunmap_range_noflush() and kmsan_vunmap_range_noflush() into vmap_page_range(), __vmap_pages_range_noflush(), and vmap_pages_range_noflush() when page table installation or KMSAN metadata setup fails. It also adds range and page_shift validation in __vmap_pages_range_noflush().

mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370

Quoted 13 lines
mm/vmalloc.c +3 -0
index 44e0eb32708f1..75aa5acf55660 100644
@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,
 	if (!err)
 		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
 					       ioremap_max_page_shift);
+	if (err)
+		__vunmap_range_noflush(addr, end);
+
 	return err;
 }

[Severity: High]
What happens here when vmap_range_noflush() returns -EINVAL for a range where end - addr is not PAGE_SIZE-aligned?

In vmap_pte_range(), commit a739e6b557af09 added a check to bail out on unaligned ranges specifically to avoid an infinite loop:

mm/vmalloc.c:vmap_pte_range() {
	...
	if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr)))
		return -EINVAL;
}

This can be reached on 64KB page kernels via pci_remap_iospace() -> vmap_page_range() with 16KB/32KB/48KB PCI I/O ranges (for example via loongson_pci_probe() or acpi_pci_root_remap_iospace()).

When vmap_pte_range() returns -EINVAL, vmap_page_range() now calls __vunmap_range_noflush(addr, end) with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as the PCI_IOBASE PMD populated during early boot by reserve_pio_range(), or an earlier PMD mapped in the same call), __vunmap_range_noflush() descends into vunmap_pte_range(), which has no alignment check:

mm/vmalloc.c:vunmap_pte_range() {
	...
	do {
		...
			ptent = ptep_get_and_clear(&init_mm, addr, pte);
		WARN_ON(!pte_none(ptent) && !pte_present(ptent));
	} while (pte += (size >> PAGE_SHIFT), addr += size, addr != end);
}
Quoted 4 lines

Because end - addr is not a multiple of PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?

That sounds reasonable. I will add a separate patch in the next version to fix this problem.

Quoted 41 lines

[Severity: Medium]
Also in vmap_page_range(), flush_cache_vmap(addr, end) is called unconditionally before checking err and before rolling back with __vunmap_range_noflush(addr, end):

mm/vmalloc.c:vmap_page_range() {
	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
				 ioremap_max_page_shift);
	flush_cache_vmap(addr, end);
	if (!err)
		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
					       ioremap_max_page_shift);
	if (err)
		__vunmap_range_noflush(addr, end);

	return err;
}

On UML, flush_cache_vmap is defined as flush_tlb_kernel_range:

arch/um/include/asm/cacheflush.h:
#define flush_cache_vmap flush_tlb_kernel_range

When vmap_range_noflush() fails partway through or kmsan_ioremap_page_range() fails, flush_cache_vmap() runs while the partially installed PTEs are still pte_present(), causing update_pte_range() in arch/um/kernel/tlb.c to map them into the host OS via ops->mmap() and clear _PAGE_NEEDSYNC:

arch/um/kernel/tlb.c:update_pte_range() {
	...
		if (pte_present(*pte)) {
			...
			ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE,
					prot, fd, offset);
		} else
			ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE);

		*pte = pte_mkuptodate(*pte);
}
Quoted 6 lines

When __vunmap_range_noflush(addr, end) then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Should flush_cache_vmap(addr, end) only be called after both vmap_range_noflush() and kmsan_ioremap_page_range() succeed?

That sounds reasonable. I will add this logic in the next version.

Quoted 57 lines
Quoted 35 lines
mm/vmalloc.c +13 -10
@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
 		pgprot_t prot, struct page **pages, unsigned int page_shift)
 {
 	unsigned int i, nr = (end - addr) >> PAGE_SHIFT;
+	unsigned long start = addr;
+	int err = 0;
+
+	if (WARN_ON_ONCE(addr >= end))
+		return -EINVAL;
 
 	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
 		return -EINVAL;
 
 	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
-			page_shift == PAGE_SHIFT)
-		return vmap_small_pages_range_noflush(addr, end, prot, pages);
-
-	for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
-		int err;
-
-		err = vmap_range_noflush(addr, addr + (1UL << page_shift),
+			page_shift == PAGE_SHIFT) {
+		err = vmap_small_pages_range_noflush(addr, end, prot, pages);
+	} else {
+		for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
+			err = vmap_range_noflush(addr, addr + (1UL << page_shift),
 					page_to_phys(pages[i]), prot,
 					page_shift);
-		if (err)
-			return err;
+			if (err)
+				break;
 
-		addr += 1UL << page_shift;
+			addr += 1UL << page_shift;

mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 718

Quoted 10 lines
mm/vmalloc.c +6 -1
+		}
 	}
 
-	return 0;
+	/* Undo the PTEs installed before the failure. */
+	if (err)
+		__vunmap_range_noflush(start, end);
+
+	return err;
 }

[Severity: High]
Similarly here, __vmap_pages_range_noflush() checks addr >= end, but does not check PAGE_ALIGNED(addr) or PAGE_ALIGNED(end).

If a caller passes a sub-page or unaligned range (for example, 0 < end - addr < PAGE_SIZE), nr = (end - addr) >> PAGE_SHIFT evaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely in vmap_pages_pte_range(). And if the huge-page path fails on an unaligned range, __vunmap_range_noflush(start, end) will hit the same infinite loop in vunmap_pte_range().

Quoted 4 lines

Should both vmap_page_range() and __vmap_pages_range_noflush() validate PAGE_ALIGNED(addr) && PAGE_ALIGNED(end) upfront before attempting to map or unmap?

I think We need to check both PAGE_ALIGNED(addr) and PAGE_ALIGNED(end). As an example: if addr is PAGE-aligned, end is not, and end - addr < PAGE_SIZE, then nr becomes zero. On the huge page path, we simply return 0 with no work performed.

At the same time, vmap_pages_pte_range() needs the same guard as commit a739e6b557af ("mm: vmalloc: WARN_ON if mapping size is not PAGE_SIZE aligned"), to protect against unaligned ranges.

So will do next version.

Quoted 59 lines
Quoted 8 lines
mm/vmalloc.c +1 -1
 int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
 
 	if (ret)
 		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+

mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 739

Quoted 10 lines
mm/vmalloc.c +9 -0
+	ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+	/*
+	 * The page tables undo themselves on failure. Tear down the
+	 * metadata that was fully set up before the mapping failed.
+	 */
+	if (ret)
+		kmsan_vunmap_range_noflush(addr, end);
+
+	return ret;
 }

[Severity: High]
Can calling __vunmap_range_noflush() and kmsan_vunmap_range_noflush() here trigger a sleeping-in-atomic-context bug when __vmalloc() or kvmalloc() is called with GFP_ATOMIC or GFP_NOWAIT?

__vmalloc_area_node() supports non-blocking allocations by wrapping __vmap_pages_range() in memalloc_apply_gfp_scope(gfp_mask):

mm/vmalloc.c:__vmalloc_area_node() {
	...
	flags = memalloc_apply_gfp_scope(gfp_mask);
	do {
		ret = __vmap_pages_range(addr, addr + size, prot, area->pages,
				page_shift, nested_gfp);
	} while ((gfp_mask & __GFP_NOFAIL) && (ret < 0));
	memalloc_restore_gfp_scope(flags);
}

When a GFP_ATOMIC or GFP_NOWAIT allocation runs in atomic context (such as under a spinlock, preempt_disable(), or rcu_read_lock()) and page table allocation fails in vmap_small_pages_range_noflush() after at least one PMD in [start, end) is populated (or after kmsan_vmap_pages_range_noflush() populated the KMSAN shadow/origin ranges), __vmap_pages_range_noflush() and kmsan_vunmap_range_noflush() synchronously call __vunmap_range_noflush().

__vunmap_range_noflush() walks the populated PMDs via vunmap_pmd_range(), which unconditionally calls cond_resched():

mm/vmalloc.c:vunmap_pmd_range() {
	...
	do {
		next = pmd_addr_end(addr, end);
		if (pmd_none_or_clear_bad(pmd))
			continue;
		vunmap_pte_range(pmd, addr, next, mask);

		cond_resched();
	} while (pmd++, addr = next, addr != end);
}
Quoted 7 lines

Commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct") deferred error-path cleanup in __vmalloc_area_node() to a workqueue specifically because unmapping on failure can sleep. Doesn't calling __vunmap_range_noflush() synchronously inside __vmap_pages_range_noflush() and vmap_pages_range_noflush() reintroduce cond_resched() in atomic context?

This issue also appears real. One possible approach is to plumb the caller's gfp_mask into vunmap_pmd_range() to conditionally skip cond_resched().
Suren, Ulad, do you have any preferred or cleaner way to handle this?

Thanks
Best Regards
Hao

author
#17 lore raw

Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions

In reply to Dev Jain

Hi Dev

Quoted 60 lines · Dev Jain
Quoted 47 lines · Hao Ge

__vmap_pages_range_noflush() and friends can install some PTEs before failing and leave them mapped, and the kernel callers do not agree on who cleans them up. pcpu_map_pages() and kmsan_ioremap_page_range() roll back what they mapped before the failure, while vm_module_tags_populate() and the __GFP_NOFAIL retry loop in __vmalloc_area_node() rely on the mapping functions cleaning up and do not call anything like vunmap_range() themselves. When the same range is mapped again, the attempt hits the leftovers and fails, with a BUG() in vmap_pte_range() for huge mappings and a warning on the small-page path.

After discussing with Suren and Ulad, we decided to put the rollback into the entry points of the vmap API rather than into every low-level helper. __vmap_pages_range_noflush() undoes the whole range it was asked to map when it fails, and vmap_page_range() undoes its range for the ioremap-style mappings.

The rollback is __vunmap_range_noflush(), it only clears the PTEs, no TLB flush, nothing has touched these mappings. vmap_pages_range_noflush() drops the KMSAN metadata when the data mapping fails, and an empty range bails out with -EINVAL now, which used to BUG_ON() on the small-page path and map nothing at all on the huge-page one.

---
  mm/vmalloc.c | 44 ++++++++++++++++++++++++++++++++------------
  1 file changed, 32 insertions(+), 12 deletions(-)
mm/vmalloc.c +3 -0
index 44e0eb32708f..75aa5acf5566 100644
@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,
  	if (!err)
  		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
  					       ioremap_max_page_shift);
+	if (err)
+		__vunmap_range_noflush(addr, end);
+
  	return err;
  }
 

mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 693

Quoted 8 lines
mm/vmalloc.c +5 -0
@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
  		pgprot_t prot, struct page **pages, unsigned int page_shift)
  {
  	unsigned int i, nr = (end - addr) >> PAGE_SHIFT;
+	unsigned long start = addr;
+	int err = 0;
+
+	if (WARN_ON_ONCE(addr >= end))
+		return -EINVAL;

IIUC this is a general change independent of this patch? Caller could pass addr >= end and if we take the huge mapping path, we lose the sanity check.

Quoted 1 line

So this could become a separate patch.

OK,  and Sashiko has flagged another case:

when 0 < end - addr < PAGE_SIZE, nr = (end - addr) >> PAGE_SHIFT evaluates to 0,

so the huge-page path still maps nothing and returns 0.

So this separate patch would cover this case as well.

Thanks

Best Regards

Hao

Quoted 58 lines
Quoted 58 lines
mm/vmalloc.c +24 -12
  	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
  		return -EINVAL;
 
  	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
-			page_shift == PAGE_SHIFT)
-		return vmap_small_pages_range_noflush(addr, end, prot, pages);
-
-	for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
-		int err;
-
-		err = vmap_range_noflush(addr, addr + (1UL << page_shift),
+			page_shift == PAGE_SHIFT) {
+		err = vmap_small_pages_range_noflush(addr, end, prot, pages);
+	} else {
+		for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
+			err = vmap_range_noflush(addr, addr + (1UL << page_shift),
  					page_to_phys(pages[i]), prot,
  					page_shift);
-		if (err)
-			return err;
+			if (err)
+				break;
 
-		addr += 1UL << page_shift;
+			addr += 1UL << page_shift;
+		}
  	}
 
-	return 0;
+	/* Undo the PTEs installed before the failure. */
+	if (err)
+		__vunmap_range_noflush(start, end);
+
+	return err;
  }
 
  int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
 
  	if (ret)
  		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+
+	ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+	/*
+	 * The page tables undo themselves on failure. Tear down the
+	 * metadata that was fully set up before the mapping failed.
+	 */
+	if (ret)
+		kmsan_vunmap_range_noflush(addr, end);
+
+	return ret;
  }
 
  static int __vmap_pages_range(unsigned long addr, unsigned long end,
author
#10 lore raw

Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions

In reply to Hao Ge
Quoted 264 lines · Hao Ge

Hi Suren and Ulad

Sashiko reported several issues, and they all look real. I have some ideas for fixing the first few;
of course, please correct me if I get anything wrong.

I'd like to discuss the last issue with you all.

Quoted 63 lines · On 2026/10/20 13:45, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop in vunmap_pte_range() on unaligned error rollback
- [High] mm/vmalloc: __vmap_pages_range_noflush() calls cond_resched() in atomic context
- [Medium] mm/vmalloc: vmap_page_range() calls flush_cache_vmap() before error rollback --

commit 043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>

mm/vmalloc: undo partial mappings inside the mapping functions

This commit adds rollback logic using __vunmap_range_noflush() and kmsan_vunmap_range_noflush() into vmap_page_range(), __vmap_pages_range_noflush(), and vmap_pages_range_noflush() when page table installation or KMSAN metadata setup fails. It also adds range and page_shift validation in __vmap_pages_range_noflush().

mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370

Quoted 13 lines
mm/vmalloc.c +3 -0
index 44e0eb32708f1..75aa5acf55660 100644
@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,
  	if (!err)
  		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
  					       ioremap_max_page_shift);
+	if (err)
+		__vunmap_range_noflush(addr, end);
+
  	return err;
  }

[Severity: High]
What happens here when vmap_range_noflush() returns -EINVAL for a range where end - addr is not PAGE_SIZE-aligned?

In vmap_pte_range(), commit a739e6b557af09 added a check to bail out on unaligned ranges specifically to avoid an infinite loop:

mm/vmalloc.c:vmap_pte_range() {
	...
	if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr)))
		return -EINVAL;
}

This can be reached on 64KB page kernels via pci_remap_iospace() -> vmap_page_range() with 16KB/32KB/48KB PCI I/O ranges (for example via loongson_pci_probe() or acpi_pci_root_remap_iospace()).

When vmap_pte_range() returns -EINVAL, vmap_page_range() now calls __vunmap_range_noflush(addr, end) with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as the PCI_IOBASE PMD populated during early boot by reserve_pio_range(), or an earlier PMD mapped in the same call), __vunmap_range_noflush() descends into vunmap_pte_range(), which has no alignment check:

mm/vmalloc.c:vunmap_pte_range() {
	...
	do {
		...
			ptent = ptep_get_and_clear(&init_mm, addr, pte);
		WARN_ON(!pte_none(ptent) && !pte_present(ptent));
	} while (pte += (size >> PAGE_SHIFT), addr += size, addr != end);
}
Quoted 4 lines

Because end - addr is not a multiple of PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?

That sounds reasonable. I will add a separate patch in the next version to fix this problem.

Quoted 41 lines

[Severity: Medium]
Also in vmap_page_range(), flush_cache_vmap(addr, end) is called unconditionally before checking err and before rolling back with __vunmap_range_noflush(addr, end):

mm/vmalloc.c:vmap_page_range() {
	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
				 ioremap_max_page_shift);
	flush_cache_vmap(addr, end);
	if (!err)
		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
					       ioremap_max_page_shift);
	if (err)
		__vunmap_range_noflush(addr, end);

	return err;
}

On UML, flush_cache_vmap is defined as flush_tlb_kernel_range:

arch/um/include/asm/cacheflush.h:
#define flush_cache_vmap flush_tlb_kernel_range

When vmap_range_noflush() fails partway through or kmsan_ioremap_page_range() fails, flush_cache_vmap() runs while the partially installed PTEs are still pte_present(), causing update_pte_range() in arch/um/kernel/tlb.c to map them into the host OS via ops->mmap() and clear _PAGE_NEEDSYNC:

arch/um/kernel/tlb.c:update_pte_range() {
	...
		if (pte_present(*pte)) {
			...
			ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE,
					prot, fd, offset);
		} else
			ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE);

		*pte = pte_mkuptodate(*pte);
}
Quoted 6 lines

When __vunmap_range_noflush(addr, end) then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Should flush_cache_vmap(addr, end) only be called after both vmap_range_noflush() and kmsan_ioremap_page_range() succeed?

That sounds reasonable. I will add this logic in the next version.

Quoted 56 lines
Quoted 35 lines
mm/vmalloc.c +13 -10
@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
  		pgprot_t prot, struct page **pages, unsigned int page_shift)
  {
  	unsigned int i, nr = (end - addr) >> PAGE_SHIFT;
+	unsigned long start = addr;
+	int err = 0;
+
+	if (WARN_ON_ONCE(addr >= end))
+		return -EINVAL;
 
  	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
  		return -EINVAL;
 
  	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
-			page_shift == PAGE_SHIFT)
-		return vmap_small_pages_range_noflush(addr, end, prot, pages);
-
-	for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
-		int err;
-
-		err = vmap_range_noflush(addr, addr + (1UL << page_shift),
+			page_shift == PAGE_SHIFT) {
+		err = vmap_small_pages_range_noflush(addr, end, prot, pages);
+	} else {
+		for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
+			err = vmap_range_noflush(addr, addr + (1UL << page_shift),
  					page_to_phys(pages[i]), prot,
  					page_shift);
-		if (err)
-			return err;
+			if (err)
+				break;
 
-		addr += 1UL << page_shift;
+			addr += 1UL << page_shift;

mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 718

Quoted 10 lines
mm/vmalloc.c +6 -1
+		}
  	}
 
-	return 0;
+	/* Undo the PTEs installed before the failure. */
+	if (err)
+		__vunmap_range_noflush(start, end);
+
+	return err;
  }

[Severity: High]
Similarly here, __vmap_pages_range_noflush() checks addr >= end, but does not check PAGE_ALIGNED(addr) or PAGE_ALIGNED(end).

If a caller passes a sub-page or unaligned range (for example, 0 < end - addr < PAGE_SIZE), nr = (end - addr) >> PAGE_SHIFT evaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely in vmap_pages_pte_range(). And if the huge-page path fails on an unaligned range, __vunmap_range_noflush(start, end) will hit the same infinite loop in vunmap_pte_range().

Quoted 4 lines

Should both vmap_page_range() and __vmap_pages_range_noflush() validate PAGE_ALIGNED(addr) && PAGE_ALIGNED(end) upfront before attempting to map or unmap?

I think We need to check both PAGE_ALIGNED(addr) and PAGE_ALIGNED(end). As an example: if addr is PAGE-aligned, end is not, and end - addr < PAGE_SIZE, then nr becomes zero. On the huge page path, we simply return 0 with no work performed.

At the same time, vmap_pages_pte_range() needs the same guard as commit a739e6b557af ("mm: vmalloc: WARN_ON if mapping size is not PAGE_SIZE aligned"), to protect against unaligned ranges.

So will do next version.

Quoted 58 lines
Quoted 8 lines
mm/vmalloc.c +1 -1
  int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
 
  	if (ret)
  		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+

mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 739

Quoted 10 lines
mm/vmalloc.c +9 -0
+	ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+	/*
+	 * The page tables undo themselves on failure. Tear down the
+	 * metadata that was fully set up before the mapping failed.
+	 */
+	if (ret)
+		kmsan_vunmap_range_noflush(addr, end);
+
+	return ret;
  }

[Severity: High]
Can calling __vunmap_range_noflush() and kmsan_vunmap_range_noflush() here trigger a sleeping-in-atomic-context bug when __vmalloc() or kvmalloc() is called with GFP_ATOMIC or GFP_NOWAIT?

__vmalloc_area_node() supports non-blocking allocations by wrapping __vmap_pages_range() in memalloc_apply_gfp_scope(gfp_mask):

mm/vmalloc.c:__vmalloc_area_node() {
	...
	flags = memalloc_apply_gfp_scope(gfp_mask);
	do {
		ret = __vmap_pages_range(addr, addr + size, prot, area->pages,
				page_shift, nested_gfp);
	} while ((gfp_mask & __GFP_NOFAIL) && (ret < 0));
	memalloc_restore_gfp_scope(flags);
}

When a GFP_ATOMIC or GFP_NOWAIT allocation runs in atomic context (such as under a spinlock, preempt_disable(), or rcu_read_lock()) and page table allocation fails in vmap_small_pages_range_noflush() after at least one PMD in [start, end) is populated (or after kmsan_vmap_pages_range_noflush() populated the KMSAN shadow/origin ranges), __vmap_pages_range_noflush() and kmsan_vunmap_range_noflush() synchronously call __vunmap_range_noflush().

__vunmap_range_noflush() walks the populated PMDs via vunmap_pmd_range(), which unconditionally calls cond_resched():

mm/vmalloc.c:vunmap_pmd_range() {
	...
	do {
		next = pmd_addr_end(addr, end);
		if (pmd_none_or_clear_bad(pmd))
			continue;
		vunmap_pte_range(pmd, addr, next, mask);

		cond_resched();
	} while (pmd++, addr = next, addr != end);
}
Quoted 7 lines

Commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct") deferred error-path cleanup in __vmalloc_area_node() to a workqueue specifically because unmapping on failure can sleep. Doesn't calling __vunmap_range_noflush() synchronously inside __vmap_pages_range_noflush() and vmap_pages_range_noflush() reintroduce cond_resched() in atomic context?

Quoted 3 lines

This issue also appears real. One possible approach is to plumb the caller's gfp_mask into vunmap_pmd_range() to conditionally skip cond_resched().
Suren, Ulad, do you have any preferred or cleaner way to handle this?

Alternatively, we can follow the approach from commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct"):

defer the cleanup to a workqueue when |gfp_mask| has GFP_ATOMIC or GFP_NOWAIT,and perform cleanup synchronously

if we are already in process context. Anyway, I will look into this further tomorrow.

Quoted 5 lines

Thanks
Best Regards
Hao

maintainer
#11 lore raw

Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions

Quoted 265 lines · Hao Ge
Quoted 258 lines · Hao Ge

Hi Suren and Ulad

Sashiko reported several issues, and they all look real. I have some ideas for fixing the first few;
of course, please correct me if I get anything wrong.

I'd like to discuss the last issue with you all.

Quoted 63 lines · On 2026/10/20 13:45, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop in vunmap_pte_range() on unaligned error rollback
- [High] mm/vmalloc: __vmap_pages_range_noflush() calls cond_resched() in atomic context
- [Medium] mm/vmalloc: vmap_page_range() calls flush_cache_vmap() before error rollback --

commit 043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>

mm/vmalloc: undo partial mappings inside the mapping functions

This commit adds rollback logic using __vunmap_range_noflush() and kmsan_vunmap_range_noflush() into vmap_page_range(), __vmap_pages_range_noflush(), and vmap_pages_range_noflush() when page table installation or KMSAN metadata setup fails. It also adds range and page_shift validation in __vmap_pages_range_noflush().

mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370

Quoted 13 lines
mm/vmalloc.c +3 -0
index 44e0eb32708f1..75aa5acf55660 100644
@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,
  	if (!err)
  		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
  					       ioremap_max_page_shift);
+	if (err)
+		__vunmap_range_noflush(addr, end);
+
  	return err;
  }

[Severity: High]
What happens here when vmap_range_noflush() returns -EINVAL for a range where end - addr is not PAGE_SIZE-aligned?

In vmap_pte_range(), commit a739e6b557af09 added a check to bail out on unaligned ranges specifically to avoid an infinite loop:

mm/vmalloc.c:vmap_pte_range() {
	...
	if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr)))
		return -EINVAL;
}

This can be reached on 64KB page kernels via pci_remap_iospace() -> vmap_page_range() with 16KB/32KB/48KB PCI I/O ranges (for example via loongson_pci_probe() or acpi_pci_root_remap_iospace()).

When vmap_pte_range() returns -EINVAL, vmap_page_range() now calls __vunmap_range_noflush(addr, end) with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as the PCI_IOBASE PMD populated during early boot by reserve_pio_range(), or an earlier PMD mapped in the same call), __vunmap_range_noflush() descends into vunmap_pte_range(), which has no alignment check:

mm/vmalloc.c:vunmap_pte_range() {
	...
	do {
		...
			ptent = ptep_get_and_clear(&init_mm, addr, pte);
		WARN_ON(!pte_none(ptent) && !pte_present(ptent));
	} while (pte += (size >> PAGE_SHIFT), addr += size, addr != end);
}
Quoted 4 lines

Because end - addr is not a multiple of PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?

That sounds reasonable. I will add a separate patch in the next version to fix this problem.

Quoted 41 lines

[Severity: Medium]
Also in vmap_page_range(), flush_cache_vmap(addr, end) is called unconditionally before checking err and before rolling back with __vunmap_range_noflush(addr, end):

mm/vmalloc.c:vmap_page_range() {
	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
				 ioremap_max_page_shift);
	flush_cache_vmap(addr, end);
	if (!err)
		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
					       ioremap_max_page_shift);
	if (err)
		__vunmap_range_noflush(addr, end);

	return err;
}

On UML, flush_cache_vmap is defined as flush_tlb_kernel_range:

arch/um/include/asm/cacheflush.h:
#define flush_cache_vmap flush_tlb_kernel_range

When vmap_range_noflush() fails partway through or kmsan_ioremap_page_range() fails, flush_cache_vmap() runs while the partially installed PTEs are still pte_present(), causing update_pte_range() in arch/um/kernel/tlb.c to map them into the host OS via ops->mmap() and clear _PAGE_NEEDSYNC:

arch/um/kernel/tlb.c:update_pte_range() {
	...
		if (pte_present(*pte)) {
			...
			ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE,
					prot, fd, offset);
		} else
			ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE);

		*pte = pte_mkuptodate(*pte);
}
Quoted 6 lines

When __vunmap_range_noflush(addr, end) then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Should flush_cache_vmap(addr, end) only be called after both vmap_range_noflush() and kmsan_ioremap_page_range() succeed?

That sounds reasonable. I will add this logic in the next version.

Quoted 52 lines
Quoted 31 lines
mm/vmalloc.c +12 -10
@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
  		pgprot_t prot, struct page **pages, unsigned int page_shift)
  {
  	unsigned int i, nr = (end - addr) >> PAGE_SHIFT;
+	unsigned long start = addr;
+	int err = 0;
+
+	if (WARN_ON_ONCE(addr >= end))
+		return -EINVAL;
  	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
  		return -EINVAL;
  	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
-			page_shift == PAGE_SHIFT)
-		return vmap_small_pages_range_noflush(addr, end, prot, pages);
-
-	for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
-		int err;
-
-		err = vmap_range_noflush(addr, addr + (1UL << page_shift),
+			page_shift == PAGE_SHIFT) {
+		err = vmap_small_pages_range_noflush(addr, end, prot, pages);
+	} else {
+		for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
+			err = vmap_range_noflush(addr, addr + (1UL << page_shift),
  					page_to_phys(pages[i]), prot,
  					page_shift);
-		if (err)
-			return err;
+			if (err)
+				break;
-		addr += 1UL << page_shift;

mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 714

Quoted 10 lines
mm/vmalloc.c +7 -1
+			addr += 1UL << page_shift;
+		}
  	}
-	return 0;
+	/* Undo the PTEs installed before the failure. */
+	if (err)
+		__vunmap_range_noflush(start, end);
+
+	return err;
  }

[Severity: High]
Similarly here, __vmap_pages_range_noflush() checks addr >= end, but does not check PAGE_ALIGNED(addr) or PAGE_ALIGNED(end).

If a caller passes a sub-page or unaligned range (for example, 0 < end - addr < PAGE_SIZE), nr = (end - addr) >> PAGE_SHIFT evaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely in vmap_pages_pte_range(). And if the huge-page path fails on an unaligned range, __vunmap_range_noflush(start, end) will hit the same infinite loop in vunmap_pte_range().

Quoted 4 lines

Should both vmap_page_range() and __vmap_pages_range_noflush() validate PAGE_ALIGNED(addr) && PAGE_ALIGNED(end) upfront before attempting to map or unmap?

I think We need to check both PAGE_ALIGNED(addr) and PAGE_ALIGNED(end). As an example: if addr is PAGE-aligned, end is not, and end - addr < PAGE_SIZE, then nr becomes zero. On the huge page path, we simply return 0 with no work performed.

At the same time, vmap_pages_pte_range() needs the same guard as commit a739e6b557af ("mm: vmalloc: WARN_ON if mapping size is not PAGE_SIZE aligned"), to protect against unaligned ranges.

So will do next version.

Quoted 56 lines
Quoted 6 lines
mm/vmalloc.c +1 -1
  int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
  	if (ret)
  		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+

mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 738

Quoted 10 lines
mm/vmalloc.c +9 -0
+	ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+	/*
+	 * The page tables undo themselves on failure. Tear down the
+	 * metadata that was fully set up before the mapping failed.
+	 */
+	if (ret)
+		kmsan_vunmap_range_noflush(addr, end);
+
+	return ret;
  }

[Severity: High]
Can calling __vunmap_range_noflush() and kmsan_vunmap_range_noflush() here trigger a sleeping-in-atomic-context bug when __vmalloc() or kvmalloc() is called with GFP_ATOMIC or GFP_NOWAIT?

__vmalloc_area_node() supports non-blocking allocations by wrapping __vmap_pages_range() in memalloc_apply_gfp_scope(gfp_mask):

mm/vmalloc.c:__vmalloc_area_node() {
	...
	flags = memalloc_apply_gfp_scope(gfp_mask);
	do {
		ret = __vmap_pages_range(addr, addr + size, prot, area->pages,
				page_shift, nested_gfp);
	} while ((gfp_mask & __GFP_NOFAIL) && (ret < 0));
	memalloc_restore_gfp_scope(flags);
}

When a GFP_ATOMIC or GFP_NOWAIT allocation runs in atomic context (such as under a spinlock, preempt_disable(), or rcu_read_lock()) and page table allocation fails in vmap_small_pages_range_noflush() after at least one PMD in [start, end) is populated (or after kmsan_vmap_pages_range_noflush() populated the KMSAN shadow/origin ranges), __vmap_pages_range_noflush() and kmsan_vunmap_range_noflush() synchronously call __vunmap_range_noflush().

__vunmap_range_noflush() walks the populated PMDs via vunmap_pmd_range(), which unconditionally calls cond_resched():

mm/vmalloc.c:vunmap_pmd_range() {
	...
	do {
		next = pmd_addr_end(addr, end);
		if (pmd_none_or_clear_bad(pmd))
			continue;
		vunmap_pte_range(pmd, addr, next, mask);

		cond_resched();
	} while (pmd++, addr = next, addr != end);
}
Quoted 7 lines

Commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct") deferred error-path cleanup in __vmalloc_area_node() to a workqueue specifically because unmapping on failure can sleep. Doesn't calling __vunmap_range_noflush() synchronously inside __vmap_pages_range_noflush() and vmap_pages_range_noflush() reintroduce cond_resched() in atomic context?

Quoted 3 lines

This issue also appears real. One possible approach is to plumb the caller's gfp_mask into vunmap_pmd_range() to conditionally skip cond_resched().
Suren, Ulad, do you have any preferred or cleaner way to handle this?

Quoted 9 lines

Alternatively, we can follow the approach from commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct"):

defer the cleanup to a workqueue when |gfp_mask| has GFP_ATOMIC or GFP_NOWAIT,and perform cleanup synchronously

if we are already in process context. Anyway, I will look into this further tomorrow.

I will check the AI concern.

--
Uladzislau Rezki

author
#12 lore raw

Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions

Hi Uladzislau and Suren

Quoted 269 lines · Uladzislau Rezki
Quoted 268 lines · Hao Ge
Quoted 258 lines · Hao Ge

Hi Suren and Ulad

Sashiko reported several issues, and they all look real. I have some ideas for fixing the first few;
of course, please correct me if I get anything wrong.

I'd like to discuss the last issue with you all.

Quoted 63 lines · On 2026/10/20 13:45, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop in vunmap_pte_range() on unaligned error rollback
- [High] mm/vmalloc: __vmap_pages_range_noflush() calls cond_resched() in atomic context
- [Medium] mm/vmalloc: vmap_page_range() calls flush_cache_vmap() before error rollback --

commit 043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>

mm/vmalloc: undo partial mappings inside the mapping functions

This commit adds rollback logic using __vunmap_range_noflush() and kmsan_vunmap_range_noflush() into vmap_page_range(), __vmap_pages_range_noflush(), and vmap_pages_range_noflush() when page table installation or KMSAN metadata setup fails. It also adds range and page_shift validation in __vmap_pages_range_noflush().

mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370

Quoted 13 lines
mm/vmalloc.c +3 -0
index 44e0eb32708f1..75aa5acf55660 100644
@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,
  	if (!err)
  		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
  					       ioremap_max_page_shift);
+	if (err)
+		__vunmap_range_noflush(addr, end);
+
  	return err;
  }

[Severity: High]
What happens here when vmap_range_noflush() returns -EINVAL for a range where end - addr is not PAGE_SIZE-aligned?

In vmap_pte_range(), commit a739e6b557af09 added a check to bail out on unaligned ranges specifically to avoid an infinite loop:

mm/vmalloc.c:vmap_pte_range() {
	...
	if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr)))
		return -EINVAL;
}

This can be reached on 64KB page kernels via pci_remap_iospace() -> vmap_page_range() with 16KB/32KB/48KB PCI I/O ranges (for example via loongson_pci_probe() or acpi_pci_root_remap_iospace()).

When vmap_pte_range() returns -EINVAL, vmap_page_range() now calls __vunmap_range_noflush(addr, end) with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as the PCI_IOBASE PMD populated during early boot by reserve_pio_range(), or an earlier PMD mapped in the same call), __vunmap_range_noflush() descends into vunmap_pte_range(), which has no alignment check:

mm/vmalloc.c:vunmap_pte_range() {
	...
	do {
		...
			ptent = ptep_get_and_clear(&init_mm, addr, pte);
		WARN_ON(!pte_none(ptent) && !pte_present(ptent));
	} while (pte += (size >> PAGE_SHIFT), addr += size, addr != end);
}
Quoted 4 lines

Because end - addr is not a multiple of PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?

That sounds reasonable. I will add a separate patch in the next version to fix this problem.

Quoted 41 lines

[Severity: Medium]
Also in vmap_page_range(), flush_cache_vmap(addr, end) is called unconditionally before checking err and before rolling back with __vunmap_range_noflush(addr, end):

mm/vmalloc.c:vmap_page_range() {
	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
				 ioremap_max_page_shift);
	flush_cache_vmap(addr, end);
	if (!err)
		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
					       ioremap_max_page_shift);
	if (err)
		__vunmap_range_noflush(addr, end);

	return err;
}

On UML, flush_cache_vmap is defined as flush_tlb_kernel_range:

arch/um/include/asm/cacheflush.h:
#define flush_cache_vmap flush_tlb_kernel_range

When vmap_range_noflush() fails partway through or kmsan_ioremap_page_range() fails, flush_cache_vmap() runs while the partially installed PTEs are still pte_present(), causing update_pte_range() in arch/um/kernel/tlb.c to map them into the host OS via ops->mmap() and clear _PAGE_NEEDSYNC:

arch/um/kernel/tlb.c:update_pte_range() {
	...
		if (pte_present(*pte)) {
			...
			ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE,
					prot, fd, offset);
		} else
			ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE);

		*pte = pte_mkuptodate(*pte);
}
Quoted 6 lines

When __vunmap_range_noflush(addr, end) then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Should flush_cache_vmap(addr, end) only be called after both vmap_range_noflush() and kmsan_ioremap_page_range() succeed?

That sounds reasonable. I will add this logic in the next version.

Quoted 52 lines
Quoted 31 lines
mm/vmalloc.c +12 -10
@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
  		pgprot_t prot, struct page **pages, unsigned int page_shift)
  {
  	unsigned int i, nr = (end - addr) >> PAGE_SHIFT;
+	unsigned long start = addr;
+	int err = 0;
+
+	if (WARN_ON_ONCE(addr >= end))
+		return -EINVAL;
  	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
  		return -EINVAL;
  	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
-			page_shift == PAGE_SHIFT)
-		return vmap_small_pages_range_noflush(addr, end, prot, pages);
-
-	for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
-		int err;
-
-		err = vmap_range_noflush(addr, addr + (1UL << page_shift),
+			page_shift == PAGE_SHIFT) {
+		err = vmap_small_pages_range_noflush(addr, end, prot, pages);
+	} else {
+		for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
+			err = vmap_range_noflush(addr, addr + (1UL << page_shift),
  					page_to_phys(pages[i]), prot,
  					page_shift);
-		if (err)
-			return err;
+			if (err)
+				break;
-		addr += 1UL << page_shift;

mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 714

Quoted 10 lines
mm/vmalloc.c +7 -1
+			addr += 1UL << page_shift;
+		}
  	}
-	return 0;
+	/* Undo the PTEs installed before the failure. */
+	if (err)
+		__vunmap_range_noflush(start, end);
+
+	return err;
  }

[Severity: High]
Similarly here, __vmap_pages_range_noflush() checks addr >= end, but does not check PAGE_ALIGNED(addr) or PAGE_ALIGNED(end).

If a caller passes a sub-page or unaligned range (for example, 0 < end - addr < PAGE_SIZE), nr = (end - addr) >> PAGE_SHIFT evaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely in vmap_pages_pte_range(). And if the huge-page path fails on an unaligned range, __vunmap_range_noflush(start, end) will hit the same infinite loop in vunmap_pte_range().

Quoted 4 lines

Should both vmap_page_range() and __vmap_pages_range_noflush() validate PAGE_ALIGNED(addr) && PAGE_ALIGNED(end) upfront before attempting to map or unmap?

I think We need to check both PAGE_ALIGNED(addr) and PAGE_ALIGNED(end). As an example: if addr is PAGE-aligned, end is not, and end - addr < PAGE_SIZE, then nr becomes zero. On the huge page path, we simply return 0 with no work performed.

At the same time, vmap_pages_pte_range() needs the same guard as commit a739e6b557af ("mm: vmalloc: WARN_ON if mapping size is not PAGE_SIZE aligned"), to protect against unaligned ranges.

So will do next version.

Quoted 56 lines
Quoted 6 lines
mm/vmalloc.c +1 -1
  int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
  	if (ret)
  		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+

mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 738

Quoted 10 lines
mm/vmalloc.c +9 -0
+	ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+	/*
+	 * The page tables undo themselves on failure. Tear down the
+	 * metadata that was fully set up before the mapping failed.
+	 */
+	if (ret)
+		kmsan_vunmap_range_noflush(addr, end);
+
+	return ret;
  }

[Severity: High]
Can calling __vunmap_range_noflush() and kmsan_vunmap_range_noflush() here trigger a sleeping-in-atomic-context bug when __vmalloc() or kvmalloc() is called with GFP_ATOMIC or GFP_NOWAIT?

__vmalloc_area_node() supports non-blocking allocations by wrapping __vmap_pages_range() in memalloc_apply_gfp_scope(gfp_mask):

mm/vmalloc.c:__vmalloc_area_node() {
	...
	flags = memalloc_apply_gfp_scope(gfp_mask);
	do {
		ret = __vmap_pages_range(addr, addr + size, prot, area->pages,
				page_shift, nested_gfp);
	} while ((gfp_mask & __GFP_NOFAIL) && (ret < 0));
	memalloc_restore_gfp_scope(flags);
}

When a GFP_ATOMIC or GFP_NOWAIT allocation runs in atomic context (such as under a spinlock, preempt_disable(), or rcu_read_lock()) and page table allocation fails in vmap_small_pages_range_noflush() after at least one PMD in [start, end) is populated (or after kmsan_vmap_pages_range_noflush() populated the KMSAN shadow/origin ranges), __vmap_pages_range_noflush() and kmsan_vunmap_range_noflush() synchronously call __vunmap_range_noflush().

__vunmap_range_noflush() walks the populated PMDs via vunmap_pmd_range(), which unconditionally calls cond_resched():

mm/vmalloc.c:vunmap_pmd_range() {
	...
	do {
		next = pmd_addr_end(addr, end);
		if (pmd_none_or_clear_bad(pmd))
			continue;
		vunmap_pte_range(pmd, addr, next, mask);

		cond_resched();
	} while (pmd++, addr = next, addr != end);
}
Quoted 7 lines

Commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct") deferred error-path cleanup in __vmalloc_area_node() to a workqueue specifically because unmapping on failure can sleep. Doesn't calling __vunmap_range_noflush() synchronously inside __vmap_pages_range_noflush() and vmap_pages_range_noflush() reintroduce cond_resched() in atomic context?

Quoted 3 lines

This issue also appears real. One possible approach is to plumb the caller's gfp_mask into vunmap_pmd_range() to conditionally skip cond_resched().
Suren, Ulad, do you have any preferred or cleaner way to handle this?

Alternatively, we can follow the approach from commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct"):

Quoted 8 lines
Quoted 6 lines

defer the cleanup to a workqueue when |gfp_mask| has GFP_ATOMIC or GFP_NOWAIT,and perform cleanup synchronously

if we are already in process context. Anyway, I will look into this further tomorrow.

I will check the AI concern.

Thank you very much for the quick reply and for offering to check the AI concern. Much appreciated!

Here is a summary of what I have found over the last few days.
I went through the history first. The cond_resched() in vunmap_pmd_range() was added by commit e47110e90584 ("mm/vunmap: add cond_resched() in vunmap_pmd_range"), modeled after zap_pte_range(). It fixed a real soft lockup: on a non-preemptible kernel, unmapping a large persistent-memory namespace (hence a large I/O map region) kept a CPU busy for 23 seconds inside the walk:

	watchdog: BUG: soft lockup - CPU#49 stuck for 23s! [ndctl:50777]

Then I looked at how vfree() handles atomic contexts: it defers the whole free to a workqueue from interrupt context (vfree_atomic()), and 9c47753167a6 does the same for a partly initialized vm_struct.

Unfortunately I do not think deferral works for this rollback case. A deferred rollback returns with the PTEs still installed, and if that range is mapped again before the workqueue runs, the mapping hits the leftovers: a warning and -EBUSY on the small-page path, a BUG() on the huge path.

So I kept it simple: plumb the caller's gfp_mask down to the unmap walk and call cond_resched() conditionally, like
commit 660654f90e7f ("mm/vmalloc.c: add a schedule point to vmalloc()") did:

	if (gfp_mask & __GFP_WAIT)
		cond_resched();

Of course, this may not be the perfect solution either. I would gladly hear your thoughts on it.

Thanks
Best Regards
Hao

Quoted 2 lines

--
Uladzislau Rezki

maintainer
#13 lore raw

Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions

Quoted 303 lines · Hao Ge

Hi Uladzislau and Suren

Quoted 269 lines · Uladzislau Rezki
Quoted 268 lines · Hao Ge
Quoted 258 lines · Hao Ge

Hi Suren and Ulad

Sashiko reported several issues, and they all look real. I have some ideas for fixing the first few;
of course, please correct me if I get anything wrong.

I'd like to discuss the last issue with you all.

Quoted 63 lines · On 2026/10/20 13:45, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop in vunmap_pte_range() on unaligned error rollback
- [High] mm/vmalloc: __vmap_pages_range_noflush() calls cond_resched() in atomic context
- [Medium] mm/vmalloc: vmap_page_range() calls flush_cache_vmap() before error rollback --

commit 043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>

mm/vmalloc: undo partial mappings inside the mapping functions

This commit adds rollback logic using __vunmap_range_noflush() and kmsan_vunmap_range_noflush() into vmap_page_range(), __vmap_pages_range_noflush(), and vmap_pages_range_noflush() when page table installation or KMSAN metadata setup fails. It also adds range and page_shift validation in __vmap_pages_range_noflush().

mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370

Quoted 13 lines
mm/vmalloc.c +3 -0
index 44e0eb32708f1..75aa5acf55660 100644
@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,
  	if (!err)
  		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
  					       ioremap_max_page_shift);
+	if (err)
+		__vunmap_range_noflush(addr, end);
+
  	return err;
  }

[Severity: High]
What happens here when vmap_range_noflush() returns -EINVAL for a range where end - addr is not PAGE_SIZE-aligned?

In vmap_pte_range(), commit a739e6b557af09 added a check to bail out on unaligned ranges specifically to avoid an infinite loop:

mm/vmalloc.c:vmap_pte_range() {
	...
	if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr)))
		return -EINVAL;
}

This can be reached on 64KB page kernels via pci_remap_iospace() -> vmap_page_range() with 16KB/32KB/48KB PCI I/O ranges (for example via loongson_pci_probe() or acpi_pci_root_remap_iospace()).

When vmap_pte_range() returns -EINVAL, vmap_page_range() now calls __vunmap_range_noflush(addr, end) with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as the PCI_IOBASE PMD populated during early boot by reserve_pio_range(), or an earlier PMD mapped in the same call), __vunmap_range_noflush() descends into vunmap_pte_range(), which has no alignment check:

mm/vmalloc.c:vunmap_pte_range() {
	...
	do {
		...
			ptent = ptep_get_and_clear(&init_mm, addr, pte);
		WARN_ON(!pte_none(ptent) && !pte_present(ptent));
	} while (pte += (size >> PAGE_SHIFT), addr += size, addr != end);
}
Quoted 4 lines

Because end - addr is not a multiple of PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?

That sounds reasonable. I will add a separate patch in the next version to fix this problem.

Quoted 41 lines

[Severity: Medium]
Also in vmap_page_range(), flush_cache_vmap(addr, end) is called unconditionally before checking err and before rolling back with __vunmap_range_noflush(addr, end):

mm/vmalloc.c:vmap_page_range() {
	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
				 ioremap_max_page_shift);
	flush_cache_vmap(addr, end);
	if (!err)
		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
					       ioremap_max_page_shift);
	if (err)
		__vunmap_range_noflush(addr, end);

	return err;
}

On UML, flush_cache_vmap is defined as flush_tlb_kernel_range:

arch/um/include/asm/cacheflush.h:
#define flush_cache_vmap flush_tlb_kernel_range

When vmap_range_noflush() fails partway through or kmsan_ioremap_page_range() fails, flush_cache_vmap() runs while the partially installed PTEs are still pte_present(), causing update_pte_range() in arch/um/kernel/tlb.c to map them into the host OS via ops->mmap() and clear _PAGE_NEEDSYNC:

arch/um/kernel/tlb.c:update_pte_range() {
	...
		if (pte_present(*pte)) {
			...
			ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE,
					prot, fd, offset);
		} else
			ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE);

		*pte = pte_mkuptodate(*pte);
}
Quoted 6 lines

When __vunmap_range_noflush(addr, end) then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Should flush_cache_vmap(addr, end) only be called after both vmap_range_noflush() and kmsan_ioremap_page_range() succeed?

That sounds reasonable. I will add this logic in the next version.

Quoted 52 lines
Quoted 31 lines
mm/vmalloc.c +12 -10
@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
  		pgprot_t prot, struct page **pages, unsigned int page_shift)
  {
  	unsigned int i, nr = (end - addr) >> PAGE_SHIFT;
+	unsigned long start = addr;
+	int err = 0;
+
+	if (WARN_ON_ONCE(addr >= end))
+		return -EINVAL;
  	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
  		return -EINVAL;
  	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
-			page_shift == PAGE_SHIFT)
-		return vmap_small_pages_range_noflush(addr, end, prot, pages);
-
-	for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
-		int err;
-
-		err = vmap_range_noflush(addr, addr + (1UL << page_shift),
+			page_shift == PAGE_SHIFT) {
+		err = vmap_small_pages_range_noflush(addr, end, prot, pages);
+	} else {
+		for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
+			err = vmap_range_noflush(addr, addr + (1UL << page_shift),
  					page_to_phys(pages[i]), prot,
  					page_shift);
-		if (err)
-			return err;
+			if (err)
+				break;
-		addr += 1UL << page_shift;

mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 714

Quoted 10 lines
mm/vmalloc.c +7 -1
+			addr += 1UL << page_shift;
+		}
  	}
-	return 0;
+	/* Undo the PTEs installed before the failure. */
+	if (err)
+		__vunmap_range_noflush(start, end);
+
+	return err;
  }

[Severity: High]
Similarly here, __vmap_pages_range_noflush() checks addr >= end, but does not check PAGE_ALIGNED(addr) or PAGE_ALIGNED(end).

If a caller passes a sub-page or unaligned range (for example, 0 < end - addr < PAGE_SIZE), nr = (end - addr) >> PAGE_SHIFT evaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely in vmap_pages_pte_range(). And if the huge-page path fails on an unaligned range, __vunmap_range_noflush(start, end) will hit the same infinite loop in vunmap_pte_range().

Quoted 4 lines

Should both vmap_page_range() and __vmap_pages_range_noflush() validate PAGE_ALIGNED(addr) && PAGE_ALIGNED(end) upfront before attempting to map or unmap?

I think We need to check both PAGE_ALIGNED(addr) and PAGE_ALIGNED(end). As an example: if addr is PAGE-aligned, end is not, and end - addr < PAGE_SIZE, then nr becomes zero. On the huge page path, we simply return 0 with no work performed.

At the same time, vmap_pages_pte_range() needs the same guard as commit a739e6b557af ("mm: vmalloc: WARN_ON if mapping size is not PAGE_SIZE aligned"), to protect against unaligned ranges.

So will do next version.

Quoted 56 lines
Quoted 6 lines
mm/vmalloc.c +1 -1
  int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
  	if (ret)
  		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+

mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 738

Quoted 10 lines
mm/vmalloc.c +9 -0
+	ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+	/*
+	 * The page tables undo themselves on failure. Tear down the
+	 * metadata that was fully set up before the mapping failed.
+	 */
+	if (ret)
+		kmsan_vunmap_range_noflush(addr, end);
+
+	return ret;
  }

[Severity: High]
Can calling __vunmap_range_noflush() and kmsan_vunmap_range_noflush() here trigger a sleeping-in-atomic-context bug when __vmalloc() or kvmalloc() is called with GFP_ATOMIC or GFP_NOWAIT?

__vmalloc_area_node() supports non-blocking allocations by wrapping __vmap_pages_range() in memalloc_apply_gfp_scope(gfp_mask):

mm/vmalloc.c:__vmalloc_area_node() {
	...
	flags = memalloc_apply_gfp_scope(gfp_mask);
	do {
		ret = __vmap_pages_range(addr, addr + size, prot, area->pages,
				page_shift, nested_gfp);
	} while ((gfp_mask & __GFP_NOFAIL) && (ret < 0));
	memalloc_restore_gfp_scope(flags);
}

When a GFP_ATOMIC or GFP_NOWAIT allocation runs in atomic context (such as under a spinlock, preempt_disable(), or rcu_read_lock()) and page table allocation fails in vmap_small_pages_range_noflush() after at least one PMD in [start, end) is populated (or after kmsan_vmap_pages_range_noflush() populated the KMSAN shadow/origin ranges), __vmap_pages_range_noflush() and kmsan_vunmap_range_noflush() synchronously call __vunmap_range_noflush().

__vunmap_range_noflush() walks the populated PMDs via vunmap_pmd_range(), which unconditionally calls cond_resched():

mm/vmalloc.c:vunmap_pmd_range() {
	...
	do {
		next = pmd_addr_end(addr, end);
		if (pmd_none_or_clear_bad(pmd))
			continue;
		vunmap_pte_range(pmd, addr, next, mask);

		cond_resched();
	} while (pmd++, addr = next, addr != end);
}
Quoted 7 lines

Commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct") deferred error-path cleanup in __vmalloc_area_node() to a workqueue specifically because unmapping on failure can sleep. Doesn't calling __vunmap_range_noflush() synchronously inside __vmap_pages_range_noflush() and vmap_pages_range_noflush() reintroduce cond_resched() in atomic context?

Quoted 3 lines

This issue also appears real. One possible approach is to plumb the caller's gfp_mask into vunmap_pmd_range() to conditionally skip cond_resched().
Suren, Ulad, do you have any preferred or cleaner way to handle this?

Alternatively, we can follow the approach from commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct"):

Quoted 8 lines
Quoted 6 lines

defer the cleanup to a workqueue when |gfp_mask| has GFP_ATOMIC or GFP_NOWAIT,and perform cleanup synchronously

if we are already in process context. Anyway, I will look into this further tomorrow.

I will check the AI concern.

Thank you very much for the quick reply and for offering to check the AI concern. Much appreciated!

Here is a summary of what I have found over the last few days.
I went through the history first. The cond_resched() in vunmap_pmd_range() was added by commit e47110e90584 ("mm/vunmap: add cond_resched() in vunmap_pmd_range"), modeled after zap_pte_range(). It fixed a real soft lockup: on a non-preemptible kernel, unmapping a large persistent-memory namespace (hence a large I/O map region) kept a CPU busy for 23 seconds inside the walk:

	watchdog: BUG: soft lockup - CPU#49 stuck for 23s! [ndctl:50777]

Then I looked at how vfree() handles atomic contexts: it defers the whole free to a workqueue from interrupt context (vfree_atomic()), and 9c47753167a6 does the same for a partly initialized vm_struct.

Unfortunately I do not think deferral works for this rollback case. A deferred rollback returns with the PTEs still installed, and if that range is mapped again before the workqueue runs, the mapping hits the leftovers: a warning and -EBUSY on the small-page path, a BUG() on the huge path.

Quoted 7 lines

So I kept it simple: plumb the caller's gfp_mask down to the unmap walk and call cond_resched() conditionally, like
commit 660654f90e7f ("mm/vmalloc.c: add a schedule point to vmalloc()") did:

	if (gfp_mask & __GFP_WAIT)
		cond_resched();

Sounds good to me. But we have the helper: gfpflags_allow_blocking(gfp_mask)) Another option is probably we can remove cond_resched(); from unmap path?

--
Uladzsislau Rezki

author
#14 lore raw

Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions

Quoted 304 lines · Uladzislau Rezki
Quoted 303 lines · Hao Ge

Hi Uladzislau and Suren

Quoted 269 lines · Uladzislau Rezki
Quoted 268 lines · Hao Ge
Quoted 258 lines · Hao Ge

Hi Suren and Ulad

Sashiko reported several issues, and they all look real. I have some ideas for fixing the first few;
of course, please correct me if I get anything wrong.

I'd like to discuss the last issue with you all.

Quoted 63 lines · On 2026/10/20 13:45, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop in vunmap_pte_range() on unaligned error rollback
- [High] mm/vmalloc: __vmap_pages_range_noflush() calls cond_resched() in atomic context
- [Medium] mm/vmalloc: vmap_page_range() calls flush_cache_vmap() before error rollback --

commit 043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>

mm/vmalloc: undo partial mappings inside the mapping functions

This commit adds rollback logic using __vunmap_range_noflush() and kmsan_vunmap_range_noflush() into vmap_page_range(), __vmap_pages_range_noflush(), and vmap_pages_range_noflush() when page table installation or KMSAN metadata setup fails. It also adds range and page_shift validation in __vmap_pages_range_noflush().

mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370

Quoted 13 lines
mm/vmalloc.c +3 -0
index 44e0eb32708f1..75aa5acf55660 100644
@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,
  	if (!err)
  		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
  					       ioremap_max_page_shift);
+	if (err)
+		__vunmap_range_noflush(addr, end);
+
  	return err;
  }

[Severity: High]
What happens here when vmap_range_noflush() returns -EINVAL for a range where end - addr is not PAGE_SIZE-aligned?

In vmap_pte_range(), commit a739e6b557af09 added a check to bail out on unaligned ranges specifically to avoid an infinite loop:

mm/vmalloc.c:vmap_pte_range() {
	...
	if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr)))
		return -EINVAL;
}

This can be reached on 64KB page kernels via pci_remap_iospace() -> vmap_page_range() with 16KB/32KB/48KB PCI I/O ranges (for example via loongson_pci_probe() or acpi_pci_root_remap_iospace()).

When vmap_pte_range() returns -EINVAL, vmap_page_range() now calls __vunmap_range_noflush(addr, end) with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as the PCI_IOBASE PMD populated during early boot by reserve_pio_range(), or an earlier PMD mapped in the same call), __vunmap_range_noflush() descends into vunmap_pte_range(), which has no alignment check:

mm/vmalloc.c:vunmap_pte_range() {
	...
	do {
		...
			ptent = ptep_get_and_clear(&init_mm, addr, pte);
		WARN_ON(!pte_none(ptent) && !pte_present(ptent));
	} while (pte += (size >> PAGE_SHIFT), addr += size, addr != end);
}
Quoted 4 lines

Because end - addr is not a multiple of PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?

That sounds reasonable. I will add a separate patch in the next version to fix this problem.

Quoted 41 lines

[Severity: Medium]
Also in vmap_page_range(), flush_cache_vmap(addr, end) is called unconditionally before checking err and before rolling back with __vunmap_range_noflush(addr, end):

mm/vmalloc.c:vmap_page_range() {
	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
				 ioremap_max_page_shift);
	flush_cache_vmap(addr, end);
	if (!err)
		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
					       ioremap_max_page_shift);
	if (err)
		__vunmap_range_noflush(addr, end);

	return err;
}

On UML, flush_cache_vmap is defined as flush_tlb_kernel_range:

arch/um/include/asm/cacheflush.h:
#define flush_cache_vmap flush_tlb_kernel_range

When vmap_range_noflush() fails partway through or kmsan_ioremap_page_range() fails, flush_cache_vmap() runs while the partially installed PTEs are still pte_present(), causing update_pte_range() in arch/um/kernel/tlb.c to map them into the host OS via ops->mmap() and clear _PAGE_NEEDSYNC:

arch/um/kernel/tlb.c:update_pte_range() {
	...
		if (pte_present(*pte)) {
			...
			ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE,
					prot, fd, offset);
		} else
			ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE);

		*pte = pte_mkuptodate(*pte);
}
Quoted 6 lines

When __vunmap_range_noflush(addr, end) then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Should flush_cache_vmap(addr, end) only be called after both vmap_range_noflush() and kmsan_ioremap_page_range() succeed?

That sounds reasonable. I will add this logic in the next version.

Quoted 52 lines
Quoted 31 lines
mm/vmalloc.c +12 -10
@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
  		pgprot_t prot, struct page **pages, unsigned int page_shift)
  {
  	unsigned int i, nr = (end - addr) >> PAGE_SHIFT;
+	unsigned long start = addr;
+	int err = 0;
+
+	if (WARN_ON_ONCE(addr >= end))
+		return -EINVAL;
  	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
  		return -EINVAL;
  	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
-			page_shift == PAGE_SHIFT)
-		return vmap_small_pages_range_noflush(addr, end, prot, pages);
-
-	for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
-		int err;
-
-		err = vmap_range_noflush(addr, addr + (1UL << page_shift),
+			page_shift == PAGE_SHIFT) {
+		err = vmap_small_pages_range_noflush(addr, end, prot, pages);
+	} else {
+		for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
+			err = vmap_range_noflush(addr, addr + (1UL << page_shift),
  					page_to_phys(pages[i]), prot,
  					page_shift);
-		if (err)
-			return err;
+			if (err)
+				break;
-		addr += 1UL << page_shift;

mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 714

Quoted 10 lines
mm/vmalloc.c +7 -1
+			addr += 1UL << page_shift;
+		}
  	}
-	return 0;
+	/* Undo the PTEs installed before the failure. */
+	if (err)
+		__vunmap_range_noflush(start, end);
+
+	return err;
  }

[Severity: High]
Similarly here, __vmap_pages_range_noflush() checks addr >= end, but does not check PAGE_ALIGNED(addr) or PAGE_ALIGNED(end).

If a caller passes a sub-page or unaligned range (for example, 0 < end - addr < PAGE_SIZE), nr = (end - addr) >> PAGE_SHIFT evaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely in vmap_pages_pte_range(). And if the huge-page path fails on an unaligned range, __vunmap_range_noflush(start, end) will hit the same infinite loop in vunmap_pte_range().

Quoted 4 lines

Should both vmap_page_range() and __vmap_pages_range_noflush() validate PAGE_ALIGNED(addr) && PAGE_ALIGNED(end) upfront before attempting to map or unmap?

I think We need to check both PAGE_ALIGNED(addr) and PAGE_ALIGNED(end). As an example: if addr is PAGE-aligned, end is not, and end - addr < PAGE_SIZE, then nr becomes zero. On the huge page path, we simply return 0 with no work performed.

At the same time, vmap_pages_pte_range() needs the same guard as commit a739e6b557af ("mm: vmalloc: WARN_ON if mapping size is not PAGE_SIZE aligned"), to protect against unaligned ranges.

So will do next version.

Quoted 56 lines
Quoted 6 lines
mm/vmalloc.c +1 -1
  int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
  	if (ret)
  		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+

mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 738

Quoted 10 lines
mm/vmalloc.c +9 -0
+	ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+	/*
+	 * The page tables undo themselves on failure. Tear down the
+	 * metadata that was fully set up before the mapping failed.
+	 */
+	if (ret)
+		kmsan_vunmap_range_noflush(addr, end);
+
+	return ret;
  }

[Severity: High]
Can calling __vunmap_range_noflush() and kmsan_vunmap_range_noflush() here trigger a sleeping-in-atomic-context bug when __vmalloc() or kvmalloc() is called with GFP_ATOMIC or GFP_NOWAIT?

__vmalloc_area_node() supports non-blocking allocations by wrapping __vmap_pages_range() in memalloc_apply_gfp_scope(gfp_mask):

mm/vmalloc.c:__vmalloc_area_node() {
	...
	flags = memalloc_apply_gfp_scope(gfp_mask);
	do {
		ret = __vmap_pages_range(addr, addr + size, prot, area->pages,
				page_shift, nested_gfp);
	} while ((gfp_mask & __GFP_NOFAIL) && (ret < 0));
	memalloc_restore_gfp_scope(flags);
}

When a GFP_ATOMIC or GFP_NOWAIT allocation runs in atomic context (such as under a spinlock, preempt_disable(), or rcu_read_lock()) and page table allocation fails in vmap_small_pages_range_noflush() after at least one PMD in [start, end) is populated (or after kmsan_vmap_pages_range_noflush() populated the KMSAN shadow/origin ranges), __vmap_pages_range_noflush() and kmsan_vunmap_range_noflush() synchronously call __vunmap_range_noflush().

__vunmap_range_noflush() walks the populated PMDs via vunmap_pmd_range(), which unconditionally calls cond_resched():

mm/vmalloc.c:vunmap_pmd_range() {
	...
	do {
		next = pmd_addr_end(addr, end);
		if (pmd_none_or_clear_bad(pmd))
			continue;
		vunmap_pte_range(pmd, addr, next, mask);

		cond_resched();
	} while (pmd++, addr = next, addr != end);
}
Quoted 7 lines

Commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct") deferred error-path cleanup in __vmalloc_area_node() to a workqueue specifically because unmapping on failure can sleep. Doesn't calling __vunmap_range_noflush() synchronously inside __vmap_pages_range_noflush() and vmap_pages_range_noflush() reintroduce cond_resched() in atomic context?

Quoted 3 lines

This issue also appears real. One possible approach is to plumb the caller's gfp_mask into vunmap_pmd_range() to conditionally skip cond_resched().
Suren, Ulad, do you have any preferred or cleaner way to handle this?

Alternatively, we can follow the approach from commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct"):

Quoted 8 lines
Quoted 6 lines

defer the cleanup to a workqueue when |gfp_mask| has GFP_ATOMIC or GFP_NOWAIT,and perform cleanup synchronously

if we are already in process context. Anyway, I will look into this further tomorrow.

I will check the AI concern.

Thank you very much for the quick reply and for offering to check the AI concern. Much appreciated!

Here is a summary of what I have found over the last few days.
I went through the history first. The cond_resched() in vunmap_pmd_range() was added by commit e47110e90584 ("mm/vunmap: add cond_resched() in vunmap_pmd_range"), modeled after zap_pte_range(). It fixed a real soft lockup: on a non-preemptible kernel, unmapping a large persistent-memory namespace (hence a large I/O map region) kept a CPU busy for 23 seconds inside the walk:

	watchdog: BUG: soft lockup - CPU#49 stuck for 23s! [ndctl:50777]

Then I looked at how vfree() handles atomic contexts: it defers the whole free to a workqueue from interrupt context (vfree_atomic()), and 9c47753167a6 does the same for a partly initialized vm_struct.

Unfortunately I do not think deferral works for this rollback case. A deferred rollback returns with the PTEs still installed, and if that range is mapped again before the workqueue runs, the mapping hits the leftovers: a warning and -EBUSY on the small-page path, a BUG() on the huge path.

Quoted 8 lines
Quoted 7 lines

So I kept it simple: plumb the caller's gfp_mask down to the unmap walk and call cond_resched() conditionally, like
commit 660654f90e7f ("mm/vmalloc.c: add a schedule point to vmalloc()") did:

	if (gfp_mask & __GFP_WAIT)
		cond_resched();

Sounds good to me. But we have the helper: gfpflags_allow_blocking(gfp_mask))

Right.

Quoted 1 line

Another option is probably we can remove cond_resched(); from unmap path?

I'm wondering if removing cond_resched() from the unmap path would reintroduce the softlockup issue fixed by below commit:

commit e47110e90584a22e9980510b00d0dfad3a83354e

Author: Aneesh Kumar K.V <[email protected]>
Date:   Thu Aug 20 17:42:05 2020 -0700
    mm/vunmap: add cond_resched() in vunmap_pmd_range
    
    Like zap_pte_range add cond_resched so that we can avoid softlockups as
    reported below.  On non-preemptible kernel with large I/O map region (like
    the one we get when using persistent memory with sector mode), an unmap of
    the namespace can report below softlockups.
    
    22724.027334] watchdog: BUG: soft lockup - CPU#49 stuck for 23s! [ndctl:50777]
     NIP [c0000000000dc224] plpar_hcall+0x38/0x58
     LR [c0000000000d8898] pSeries_lpar_hpte_invalidate+0x68/0xb0
     Call Trace:
        flush_hash_page+0x114/0x200
        hpte_need_flush+0x2dc/0x540
        vunmap_page_range+0x538/0x6f0
        free_unmap_vmap_area+0x30/0x70
        remove_vm_area+0xfc/0x140
        __vunmap+0x68/0x270
        __iounmap.part.0+0x34/0x60
        memunmap+0x54/0x70
        release_nodes+0x28c/0x300
        device_release_driver_internal+0x16c/0x280
        unbind_store+0x124/0x170
        drv_attr_store+0x44/0x60
        sysfs_kf_write+0x64/0x90
        kernfs_fop_write+0x1b0/0x290
        __vfs_write+0x3c/0x70
        vfs_write+0xd8/0x260
        ksys_write+0xdc/0x130
        system_call+0x5c/0x70

Even though we gained lazy MMU mode after that issue was fixed, I suspect I may be missing some technical details. Could you walk me through the reasoning for removing this cond_resched()?

Thanks very much!

Thanks
Best Regards
Hao

Quoted 3 lines

--
Uladzsislau Rezki

maintainer
#15 lore raw

Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions

Quoted 357 lines · Hao Ge
Quoted 304 lines · Uladzislau Rezki
Quoted 303 lines · Hao Ge

Hi Uladzislau and Suren

Quoted 269 lines · Uladzislau Rezki
Quoted 268 lines · Hao Ge
Quoted 258 lines · Hao Ge

Hi Suren and Ulad

Sashiko reported several issues, and they all look real. I have some ideas for fixing the first few;
of course, please correct me if I get anything wrong.

I'd like to discuss the last issue with you all.

Quoted 63 lines · On 2026/10/20 13:45, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop in vunmap_pte_range() on unaligned error rollback
- [High] mm/vmalloc: __vmap_pages_range_noflush() calls cond_resched() in atomic context
- [Medium] mm/vmalloc: vmap_page_range() calls flush_cache_vmap() before error rollback --

commit 043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>

mm/vmalloc: undo partial mappings inside the mapping functions

This commit adds rollback logic using __vunmap_range_noflush() and kmsan_vunmap_range_noflush() into vmap_page_range(), __vmap_pages_range_noflush(), and vmap_pages_range_noflush() when page table installation or KMSAN metadata setup fails. It also adds range and page_shift validation in __vmap_pages_range_noflush().

mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370

Quoted 13 lines
mm/vmalloc.c +3 -0
index 44e0eb32708f1..75aa5acf55660 100644
@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,
  	if (!err)
  		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
  					       ioremap_max_page_shift);
+	if (err)
+		__vunmap_range_noflush(addr, end);
+
  	return err;
  }

[Severity: High]
What happens here when vmap_range_noflush() returns -EINVAL for a range where end - addr is not PAGE_SIZE-aligned?

In vmap_pte_range(), commit a739e6b557af09 added a check to bail out on unaligned ranges specifically to avoid an infinite loop:

mm/vmalloc.c:vmap_pte_range() {
	...
	if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr)))
		return -EINVAL;
}

This can be reached on 64KB page kernels via pci_remap_iospace() -> vmap_page_range() with 16KB/32KB/48KB PCI I/O ranges (for example via loongson_pci_probe() or acpi_pci_root_remap_iospace()).

When vmap_pte_range() returns -EINVAL, vmap_page_range() now calls __vunmap_range_noflush(addr, end) with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as the PCI_IOBASE PMD populated during early boot by reserve_pio_range(), or an earlier PMD mapped in the same call), __vunmap_range_noflush() descends into vunmap_pte_range(), which has no alignment check:

mm/vmalloc.c:vunmap_pte_range() {
	...
	do {
		...
			ptent = ptep_get_and_clear(&init_mm, addr, pte);
		WARN_ON(!pte_none(ptent) && !pte_present(ptent));
	} while (pte += (size >> PAGE_SHIFT), addr += size, addr != end);
}
Quoted 4 lines

Because end - addr is not a multiple of PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?

That sounds reasonable. I will add a separate patch in the next version to fix this problem.

Quoted 41 lines

[Severity: Medium]
Also in vmap_page_range(), flush_cache_vmap(addr, end) is called unconditionally before checking err and before rolling back with __vunmap_range_noflush(addr, end):

mm/vmalloc.c:vmap_page_range() {
	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
				 ioremap_max_page_shift);
	flush_cache_vmap(addr, end);
	if (!err)
		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
					       ioremap_max_page_shift);
	if (err)
		__vunmap_range_noflush(addr, end);

	return err;
}

On UML, flush_cache_vmap is defined as flush_tlb_kernel_range:

arch/um/include/asm/cacheflush.h:
#define flush_cache_vmap flush_tlb_kernel_range

When vmap_range_noflush() fails partway through or kmsan_ioremap_page_range() fails, flush_cache_vmap() runs while the partially installed PTEs are still pte_present(), causing update_pte_range() in arch/um/kernel/tlb.c to map them into the host OS via ops->mmap() and clear _PAGE_NEEDSYNC:

arch/um/kernel/tlb.c:update_pte_range() {
	...
		if (pte_present(*pte)) {
			...
			ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE,
					prot, fd, offset);
		} else
			ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE);

		*pte = pte_mkuptodate(*pte);
}
Quoted 6 lines

When __vunmap_range_noflush(addr, end) then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Should flush_cache_vmap(addr, end) only be called after both vmap_range_noflush() and kmsan_ioremap_page_range() succeed?

That sounds reasonable. I will add this logic in the next version.

Quoted 52 lines
Quoted 31 lines
mm/vmalloc.c +12 -10
@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
  		pgprot_t prot, struct page **pages, unsigned int page_shift)
  {
  	unsigned int i, nr = (end - addr) >> PAGE_SHIFT;
+	unsigned long start = addr;
+	int err = 0;
+
+	if (WARN_ON_ONCE(addr >= end))
+		return -EINVAL;
  	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
  		return -EINVAL;
  	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
-			page_shift == PAGE_SHIFT)
-		return vmap_small_pages_range_noflush(addr, end, prot, pages);
-
-	for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
-		int err;
-
-		err = vmap_range_noflush(addr, addr + (1UL << page_shift),
+			page_shift == PAGE_SHIFT) {
+		err = vmap_small_pages_range_noflush(addr, end, prot, pages);
+	} else {
+		for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
+			err = vmap_range_noflush(addr, addr + (1UL << page_shift),
  					page_to_phys(pages[i]), prot,
  					page_shift);
-		if (err)
-			return err;
+			if (err)
+				break;
-		addr += 1UL << page_shift;

mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 714

Quoted 10 lines
mm/vmalloc.c +7 -1
+			addr += 1UL << page_shift;
+		}
  	}
-	return 0;
+	/* Undo the PTEs installed before the failure. */
+	if (err)
+		__vunmap_range_noflush(start, end);
+
+	return err;
  }

[Severity: High]
Similarly here, __vmap_pages_range_noflush() checks addr >= end, but does not check PAGE_ALIGNED(addr) or PAGE_ALIGNED(end).

If a caller passes a sub-page or unaligned range (for example, 0 < end - addr < PAGE_SIZE), nr = (end - addr) >> PAGE_SHIFT evaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely in vmap_pages_pte_range(). And if the huge-page path fails on an unaligned range, __vunmap_range_noflush(start, end) will hit the same infinite loop in vunmap_pte_range().

Quoted 4 lines

Should both vmap_page_range() and __vmap_pages_range_noflush() validate PAGE_ALIGNED(addr) && PAGE_ALIGNED(end) upfront before attempting to map or unmap?

I think We need to check both PAGE_ALIGNED(addr) and PAGE_ALIGNED(end). As an example: if addr is PAGE-aligned, end is not, and end - addr < PAGE_SIZE, then nr becomes zero. On the huge page path, we simply return 0 with no work performed.

At the same time, vmap_pages_pte_range() needs the same guard as commit a739e6b557af ("mm: vmalloc: WARN_ON if mapping size is not PAGE_SIZE aligned"), to protect against unaligned ranges.

So will do next version.

Quoted 56 lines
Quoted 6 lines
mm/vmalloc.c +1 -1
  int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
  	if (ret)
  		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+

mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 738

Quoted 10 lines
mm/vmalloc.c +9 -0
+	ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+	/*
+	 * The page tables undo themselves on failure. Tear down the
+	 * metadata that was fully set up before the mapping failed.
+	 */
+	if (ret)
+		kmsan_vunmap_range_noflush(addr, end);
+
+	return ret;
  }

[Severity: High]
Can calling __vunmap_range_noflush() and kmsan_vunmap_range_noflush() here trigger a sleeping-in-atomic-context bug when __vmalloc() or kvmalloc() is called with GFP_ATOMIC or GFP_NOWAIT?

__vmalloc_area_node() supports non-blocking allocations by wrapping __vmap_pages_range() in memalloc_apply_gfp_scope(gfp_mask):

mm/vmalloc.c:__vmalloc_area_node() {
	...
	flags = memalloc_apply_gfp_scope(gfp_mask);
	do {
		ret = __vmap_pages_range(addr, addr + size, prot, area->pages,
				page_shift, nested_gfp);
	} while ((gfp_mask & __GFP_NOFAIL) && (ret < 0));
	memalloc_restore_gfp_scope(flags);
}

When a GFP_ATOMIC or GFP_NOWAIT allocation runs in atomic context (such as under a spinlock, preempt_disable(), or rcu_read_lock()) and page table allocation fails in vmap_small_pages_range_noflush() after at least one PMD in [start, end) is populated (or after kmsan_vmap_pages_range_noflush() populated the KMSAN shadow/origin ranges), __vmap_pages_range_noflush() and kmsan_vunmap_range_noflush() synchronously call __vunmap_range_noflush().

__vunmap_range_noflush() walks the populated PMDs via vunmap_pmd_range(), which unconditionally calls cond_resched():

mm/vmalloc.c:vunmap_pmd_range() {
	...
	do {
		next = pmd_addr_end(addr, end);
		if (pmd_none_or_clear_bad(pmd))
			continue;
		vunmap_pte_range(pmd, addr, next, mask);

		cond_resched();
	} while (pmd++, addr = next, addr != end);
}
Quoted 7 lines

Commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct") deferred error-path cleanup in __vmalloc_area_node() to a workqueue specifically because unmapping on failure can sleep. Doesn't calling __vunmap_range_noflush() synchronously inside __vmap_pages_range_noflush() and vmap_pages_range_noflush() reintroduce cond_resched() in atomic context?

Quoted 3 lines

This issue also appears real. One possible approach is to plumb the caller's gfp_mask into vunmap_pmd_range() to conditionally skip cond_resched().
Suren, Ulad, do you have any preferred or cleaner way to handle this?

Alternatively, we can follow the approach from commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct"):

Quoted 8 lines
Quoted 6 lines

defer the cleanup to a workqueue when |gfp_mask| has GFP_ATOMIC or GFP_NOWAIT,and perform cleanup synchronously

if we are already in process context. Anyway, I will look into this further tomorrow.

I will check the AI concern.

Thank you very much for the quick reply and for offering to check the AI concern. Much appreciated!

Here is a summary of what I have found over the last few days.
I went through the history first. The cond_resched() in vunmap_pmd_range() was added by commit e47110e90584 ("mm/vunmap: add cond_resched() in vunmap_pmd_range"), modeled after zap_pte_range(). It fixed a real soft lockup: on a non-preemptible kernel, unmapping a large persistent-memory namespace (hence a large I/O map region) kept a CPU busy for 23 seconds inside the walk:

	watchdog: BUG: soft lockup - CPU#49 stuck for 23s! [ndctl:50777]

Then I looked at how vfree() handles atomic contexts: it defers the whole free to a workqueue from interrupt context (vfree_atomic()), and 9c47753167a6 does the same for a partly initialized vm_struct.

Unfortunately I do not think deferral works for this rollback case. A deferred rollback returns with the PTEs still installed, and if that range is mapped again before the workqueue runs, the mapping hits the leftovers: a warning and -EBUSY on the small-page path, a BUG() on the huge path.

Quoted 8 lines
Quoted 7 lines

So I kept it simple: plumb the caller's gfp_mask down to the unmap walk and call cond_resched() conditionally, like
commit 660654f90e7f ("mm/vmalloc.c: add a schedule point to vmalloc()") did:

	if (gfp_mask & __GFP_WAIT)
		cond_resched();

Sounds good to me. But we have the helper: gfpflags_allow_blocking(gfp_mask))

Right.

Quoted 1 line

Another option is probably we can remove cond_resched(); from unmap path?

I'm wondering if removing cond_resched() from the unmap path would reintroduce the softlockup issue fixed by below commit:

commit e47110e90584a22e9980510b00d0dfad3a83354e

Author: Aneesh Kumar K.V <[email protected]>
Date:   Thu Aug 20 17:42:05 2020 -0700
    mm/vunmap: add cond_resched() in vunmap_pmd_range
    
    Like zap_pte_range add cond_resched so that we can avoid softlockups as
    reported below.  On non-preemptible kernel with large I/O map region (like
    the one we get when using persistent memory with sector mode), an unmap of
    the namespace can report below softlockups.
    
    22724.027334] watchdog: BUG: soft lockup - CPU#49 stuck for 23s! [ndctl:50777]
     NIP [c0000000000dc224] plpar_hcall+0x38/0x58
     LR [c0000000000d8898] pSeries_lpar_hpte_invalidate+0x68/0xb0
     Call Trace:
        flush_hash_page+0x114/0x200
        hpte_need_flush+0x2dc/0x540
        vunmap_page_range+0x538/0x6f0
        free_unmap_vmap_area+0x30/0x70
        remove_vm_area+0xfc/0x140
        __vunmap+0x68/0x270
        __iounmap.part.0+0x34/0x60
        memunmap+0x54/0x70
        release_nodes+0x28c/0x300
        device_release_driver_internal+0x16c/0x280
        unbind_store+0x124/0x170
        drv_attr_store+0x44/0x60
        sysfs_kf_write+0x64/0x90
        kernfs_fop_write+0x1b0/0x290
        __vfs_write+0x3c/0x70
        vfs_write+0xd8/0x260
        ksys_write+0xdc/0x130
        system_call+0x5c/0x70
Quoted 4 lines

Even though we gained lazy MMU mode after that issue was fixed, I suspect I may be missing some technical details. Could you walk me through the reasoning for removing this cond_resched()?

This is a good point. And you showed the good example why it is a bad idea to remove it. I was thinking about this as preventing CPU hogging by unmap path and you proved it.

Checking gfp sounds like is a correct away to address AI concern which is correct.

--
Uladzislau Rezki