mm/vmalloc: make the mapping functions undo their partial mappings
Subsystems (6)
v2 cover mm/vmalloc: make the mapping functions undo their partial mappings
__vmap_pages_range_noflush() and friends can install some PTEs before failing and leave them mapped, and the kernel callers do not agree on who cleans them up. pcpu_map_pages() and kmsan_ioremap_page_range() roll back what they mapped before the failure, while vm_module_tags_populate() and the __GFP_NOFAIL retry loop in __vmalloc_area_node() rely on the mapping functions cleaning up and do not call anything like vunmap_range() themselves. When the same range is mapped again, the attempt hits the leftovers and fails, with a BUG() in vmap_pte_range() for huge mappings and a warning on the small-page path.
After discussing with Suren and Ulad [1], we move the rollback into the mapping functions. A failed vmap leaves nothing mapped, and the callers no longer clean up after it.
[1] https://lore.kernel.org/all/[email protected]/
Patch 1 is the KMSAN part. The shadow and the origin of a range are mapped by two separate calls, so the rollback there has to cover both. Patch 2 is its disabled-KMSAN counterpart. Patch 3 adds the rollback to __vmap_pages_range_noflush() and vmap_page_range(). Patches 4 and 5 drop the powerpc and percpu cleanups that are not needed anymore.
Changes since v1:
- Add patch 2. Sashiko reported that an empty range that hits patch 3's WARN_ON_ONCE() trips __vunmap_range_noflush()'s BUG_ON() through kmsan_vunmap_range_noflush() when kmsan=0. Patch 2 adds the kmsan_enabled check the unmap side was missing.
- Pick up Reviewed-by tags from Suren and Ulad on patches 1, 4 and 5.
- Add Cc: stable to patch 3.
v1 link: https://lore.kernel.org/all/[email protected]/
Hao Ge (5):
mm/kmsan: undo the shadow mapping when the origin mapping fails mm/kmsan: skip vunmap_range_noflush when KMSAN is disabled mm/vmalloc: undo partial mappings inside the mapping functions
powerpc: drop redundant unmaps of failed vmap mappings mm/percpu: stop unmapping the CPU that failed to map
arch/powerpc/kernel/isa-bridge.c | 5 ++-- arch/powerpc/kernel/pci_64.c | 4 +-- mm/kmsan/hooks.c | 3 +++ mm/kmsan/shadow.c | 4 +++ mm/percpu-vm.c | 5 ++-- mm/vmalloc.c | 44 +++++++++++++++++++++++--------- 6 files changed, 45 insertions(+), 20 deletions(-)
Signature · 1 line
2.25.1
PATCH v2 1/5 mm/kmsan: undo the shadow mapping when the origin mapping fails
kmsan_vmap_pages_range_noflush() first maps the shadow pages and then the origin pages. If the second mapping fails, the first one is left mapped, and the callers do not roll it back. The next vmap of the same metadata range hits the stale PTEs again and BUG()s on the huge mapping path, or gets -EBUSY with a WARN_ON() on the small page one.
Undo the shadow mapping on that error path, the same way kmsan_ioremap_page_range() cleans up after a partial failure. __vunmap_range_noflush() only clears the PTEs; nothing has touched the shadow mapping, so no TLB flush is needed.
--- mm/kmsan/shadow.c | 4 ++++ 1 file changed, 4 insertions(+)
mm/kmsan/shadow.c +4 -0
@@ -258,6 +258,10 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end, o_pages, page_shift);kmsan_leave_runtime ();if (mapped) { +/* Undo the shadow mapping set up above. */ +kmsan_enter_runtime (); +__vunmap_range_noflush (shadow_start, shadow_end); +kmsan_leave_runtime (); err = mapped; goto ret; }
Signature · 1 line
2.25.1
PATCH v2 2/5 mm/kmsan: skip vunmap_range_noflush when KMSAN is disabled
In reply to Hao Gekmsan_vmap_pages_range_noflush() returns 0 when kmsan_enabled is false. The unmap side does not, so it still walks the shadow and origin page tables even though KMSAN never installed any metadata there.
Add the matching early return.
--- mm/kmsan/hooks.c | 3 +++ 1 file changed, 3 insertions(+)
mm/kmsan/hooks.c +3 -0
@@ -138,6 +138,9 @@ static unsigned long vmalloc_origin(unsigned long addr)void kmsan_vunmap_range_noflush (unsigned long start,unsigned long end) { +if (!kmsan_enabled) +return ; +__vunmap_range_noflush (vmalloc_shadow (start),vmalloc_shadow (end));__vunmap_range_noflush (vmalloc_origin (start),vmalloc_origin (end));flush_cache_vmap (vmalloc_shadow (start),vmalloc_shadow (end));
Signature · 1 line
2.25.1
PATCH v2 3/5 mm/vmalloc: undo partial mappings inside the mapping functions
In reply to Hao Ge__vmap_pages_range_noflush() and friends can install some PTEs before failing and leave them mapped, and the kernel callers do not agree on who cleans them up. pcpu_map_pages() and kmsan_ioremap_page_range() roll back what they mapped before the failure, while vm_module_tags_populate() and the __GFP_NOFAIL retry loop in __vmalloc_area_node() rely on the mapping functions cleaning up and do not call anything like vunmap_range() themselves. When the same range is mapped again, the attempt hits the leftovers and fails, with a BUG() in vmap_pte_range() for huge mappings and a warning on the small-page path.
After discussing with Suren and Ulad, we decided to put the rollback into the entry points of the vmap API rather than into every low-level helper. __vmap_pages_range_noflush() undoes the whole range it was asked to map when it fails, and vmap_page_range() undoes its range for the ioremap-style mappings.
The rollback is __vunmap_range_noflush(), it only clears the PTEs, no TLB flush, nothing has touched these mappings. vmap_pages_range_noflush() drops the KMSAN metadata when the data mapping fails, and an empty range bails out with -EINVAL now, which used to BUG_ON() on the small-page path and map nothing at all on the huge-page one.
--- mm/vmalloc.c | 44 ++++++++++++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 12 deletions(-)
mm/vmalloc.c +32 -12
@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,if (!err) err =kmsan_ioremap_page_range (addr, end, phys_addr, prot, ioremap_max_page_shift); +if (err) +__vunmap_range_noflush (addr, end); +return err; }
@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; +unsigned long start = addr; +int err =0 ; + +if (WARN_ON_ONCE (addr >= end)) +return -EINVAL ;if (WARN_ON_ONCE (page_shift <PAGE_SHIFT ))return -EINVAL ;if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) -return vmap_small_pages_range_noflush (addr, end, prot, pages); - -for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { -int err; - - err =vmap_range_noflush (addr, addr + (1UL << page_shift), + page_shift ==PAGE_SHIFT ) { + err =vmap_small_pages_range_noflush (addr, end, prot, pages); + }else { +for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { + err =vmap_range_noflush (addr, addr + (1UL << page_shift),page_to_phys (pages[i]), prot, page_shift); -if (err) -return err; +if (err) +break ; - addr +=1UL << page_shift; + addr +=1UL << page_shift; + } } -return 0 ; +/* Undo the PTEs installed before the failure. */ +if (err) +__vunmap_range_noflush (start, end); + +return err; } int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,if (ret)return ret; -return __vmap_pages_range_noflush (addr, end, prot, pages, page_shift); + + ret =__vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +/* +* The page tables undo themselves on failure. Tear down the +* metadata that was fully set up before the mapping failed. +*/ +if (ret) +kmsan_vunmap_range_noflush (addr, end); + +return ret; } static int __vmap_pages_range(unsigned long addr, unsigned long end,
Signature · 1 line
2.25.1
PATCH v2 4/5 powerpc: drop redundant unmaps of failed vmap mappings
In reply to Hao Gevmap_page_range() and ioremap_page_range() clear the PTEs they installed when they fail, so the vunmap_range() in remap_isa_base() and ioremap_phb() has nothing left to remove.
--- arch/powerpc/kernel/isa-bridge.c | 5 ++--- arch/powerpc/kernel/pci_64.c | 4 +--- 2 files changed, 3 insertions(+), 6 deletions(-)
arch/powerpc/kernel/isa-bridge.c +2 -3
@@ -46,9 +46,8 @@ static void remap_isa_base(phys_addr_t pa, unsigned long size)WARN_ON_ONCE (size & ~PAGE_MASK );if (slab_is_available ()) { -if (vmap_page_range (ISA_IO_BASE ,ISA_IO_BASE + size, pa, -pgprot_noncached (PAGE_KERNEL ))) -vunmap_range (ISA_IO_BASE ,ISA_IO_BASE + size); +vmap_page_range (ISA_IO_BASE ,ISA_IO_BASE + size, pa, +pgprot_noncached (PAGE_KERNEL )); }else {early_ioremap_range (ISA_IO_BASE , pa, size,pgprot_noncached (PAGE_KERNEL ));
arch/powerpc/kernel/pci_64.c +1 -3
@@ -139,10 +139,8 @@ void __iomem *ioremap_phb(phys_addr_t paddr, unsigned long size) addr = (unsigned long )area->addr;if (ioremap_page_range (addr, addr + size, paddr, -pgprot_noncached (PAGE_KERNEL ))) { -vunmap_range (addr, addr + size); +pgprot_noncached (PAGE_KERNEL )))return NULL ; - }return (void __iomem *)addr; }
Signature · 1 line
2.25.1
PATCH v2 5/5 mm/percpu: stop unmapping the CPU that failed to map
In reply to Hao Ge__pcpu_map_pages() clears the PTEs it installed when it fails, so pcpu_map_pages() only has to tear down the CPUs it mapped before the failing one.
--- mm/percpu-vm.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/percpu-vm.c +3 -2
@@ -252,10 +252,11 @@ static int pcpu_map_pages(struct pcpu_chunk *chunk, struct page **pages,return 0 ; err:for_each_possible_cpu (tcpu) { -__pcpu_unmap_pages (pcpu_chunk_addr (chunk, tcpu, page_start), - page_end - page_start); +/* The failing CPU's partial mapping undoes itself */ if (tcpu == cpu)break ; +__pcpu_unmap_pages (pcpu_chunk_addr (chunk, tcpu, page_start), + page_end - page_start); }pcpu_post_unmap_tlb_flush (chunk, page_start, page_end);return err;
Signature · 1 line
2.25.1
Re: [PATCH v2 1/5] mm/kmsan: undo the shadow mapping when the origin mapping fails
In reply to Hao GeQuoted 11 lines · Hao Ge
kmsan_vmap_pages_range_noflush()first maps the shadow pages and then the origin pages. If the second mapping fails, the first one is left mapped, and the callers do not roll it back. The next vmap of the same metadata range hits the stale PTEs again and BUG()s on the huge mapping path, or gets-EBUSYwith aWARN_ON()on the small page one.Undo the shadow mapping on that error path, the same way
kmsan_ioremap_page_range()cleans up after a partial failure.__vunmap_range_noflush()only clears the PTEs; nothing has touched the shadow mapping, so no TLB flush is needed.
Quoted 5 lines
Fixes 47ebd0310e89 ("mm: kmsan: handle alloc failures in kmsan_vmap_pages_range_noflush()")Reviewed-by Suren Baghdasaryan <[email protected]>Signed-off-by Hao Ge <[email protected]>---
Quoted 18 lines
mm/kmsan/shadow.c | 4 ++++ 1 file changed, 4 insertions(+)mm/kmsan/shadow.c +4
-0@@ -258,6 +258,10 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end, o_pages, page_shift);kmsan_leave_runtime ();if (mapped) { +/* Undo the shadow mapping set up above. */ +kmsan_enter_runtime (); +__vunmap_range_noflush (shadow_start, shadow_end); +kmsan_leave_runtime (); err = mapped; goto ret; }
Re: [PATCH v2 2/5] mm/kmsan: skip vunmap_range_noflush when KMSAN is disabled
In reply to Hao GeQuoted 9 lines · Hao Ge
kmsan_vmap_pages_range_noflush()returns 0 whenkmsan_enabledis false. The unmap side does not, so it still walks the shadow and origin page tables even though KMSAN never installed any metadata there.Add the matching early return.
Reported-by Sashiko <[email protected]>
Cc stable without fixes?
I guess without the fix nothing is actually wrong, except doing wasteful work?
Quoted 19 lines
Signed-off-by Hao Ge <[email protected]>--- mm/kmsan/hooks.c | 3 +++ 1 file changed, 3 insertions(+)mm/kmsan/hooks.c +3
-0@@ -138,6 +138,9 @@ static unsigned long vmalloc_origin(unsigned long addr)void kmsan_vunmap_range_noflush (unsigned long start,unsigned long end) { +if (!kmsan_enabled) +return ; +__vunmap_range_noflush (vmalloc_shadow (start),vmalloc_shadow (end));__vunmap_range_noflush (vmalloc_origin (start),vmalloc_origin (end));flush_cache_vmap (vmalloc_shadow (start),vmalloc_shadow (end));
Re: [PATCH v2 2/5] mm/kmsan: skip vunmap_range_noflush when KMSAN is disabled
Hi Dev
Quoted 8 lines · Dev Jain
Quoted 5 lines · Hao Ge
kmsan_vmap_pages_range_noflush()returns 0 whenkmsan_enabledis false. The unmap side does not, so it still walks the shadow and origin page tables even though KMSAN never installed any metadata there.
Quoted 6 lines
Quoted 4 lines
Add the matching early return.
Reported-by Sashiko <[email protected]>Cc stable without fixes?
Oops, my bad. The Cc stable was added mistakenly.
Quoted 5 lines
I guess without the fix nothing is actually wrong, except doing wasteful work?
That was true before, but not anymore after Patch 3 (mm/vmalloc: undo partial mappings inside the mapping functions). vmap_pages_range_noflush() now does:
int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
pgprot_t prot, struct page **pages, unsigned int page_shift, gfp_t gfp_mask)
{
int ret = kmsan_vmap_pages_range_noflush(addr, end, prot, pages,
page_shift, gfp_mask);
if (ret)
return ret;
ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
/*
* The page tables undo themselves on failure. Tear down the
* metadata that was fully set up before the mapping failed.
*/
if (ret)
kmsan_vunmap_range_noflush(addr, end);
return ret;
}Sashiko report: with !kmsan_enabled and a caller passing an empty range (addr >= end) the flow is:
1. kmsan_vmap_pages_range_noflush() returns 0 at its first check, it never looks at the range.
2. __vmap_pages_range_noflush() hits the new WARN_ON_ONCE(addr >= end) and returns -EINVAL.
3. ret is non-zero now, so the rollback calls kmsan_vunmap_range_noflush(addr, end), which unconditionally does
__vunmap_range_noflush(vmalloc_shadow(start),vmalloc_shadow(end));
vmalloc_shadow() is a fixed offset, so the shadow range is empty too, and __vunmap_range_noflush() starts with BUG_ON(addr >= end). Panic.
So the guard is not about the empty range itself: a caller passing an empty range is buggy, and on today's kernels it already BUG_ON() in vmap_small_pages_range_noflush() on the small-page path. Patch 3 deliberately turns that into WARN_ON_ONCE() + -EINVAL, and this patch makes sure the new metadata rollback does not turn that recoverable error back into a panic when KMSAN is disabled.
But it's only theoretical. We haven't hit this case in real usage. We add this guard merely to reduce unnecessary work, and incidentally harden Patch 3.
Thanks for pointing this out.
Best Regards
Hao
Quoted 20 lines
Quoted 19 lines
Signed-off-by Hao Ge <[email protected]>--- mm/kmsan/hooks.c | 3 +++ 1 file changed, 3 insertions(+)mm/kmsan/hooks.c +3
-0@@ -138,6 +138,9 @@ static unsigned long vmalloc_origin(unsigned long addr)void kmsan_vunmap_range_noflush (unsigned long start,unsigned long end) { +if (!kmsan_enabled) +return ; +__vunmap_range_noflush (vmalloc_shadow (start),vmalloc_shadow (end));__vunmap_range_noflush (vmalloc_origin (start),vmalloc_origin (end));flush_cache_vmap (vmalloc_shadow (start),vmalloc_shadow (end));
Re: [PATCH v2 2/5] mm/kmsan: skip vunmap_range_noflush when KMSAN is disabled
In reply to Hao GeQuoted 11 lines · Hao Ge
kmsan_vmap_pages_range_noflush()returns 0 whenkmsan_enabledis false. The unmap side does not, so it still walks the shadow and origin page tables even though KMSAN never installed any metadata there.Add the matching early return.
---
Quoted 17 lines
mm/kmsan/hooks.c | 3 +++ 1 file changed, 3 insertions(+)mm/kmsan/hooks.c +3
-0@@ -138,6 +138,9 @@ static unsigned long vmalloc_origin(unsigned long addr)void kmsan_vunmap_range_noflush (unsigned long start,unsigned long end) { +if (!kmsan_enabled) +return ; +__vunmap_range_noflush (vmalloc_shadow (start),vmalloc_shadow (end));__vunmap_range_noflush (vmalloc_origin (start),vmalloc_origin (end));flush_cache_vmap (vmalloc_shadow (start),vmalloc_shadow (end));
[email protected]
AI reviewRe: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions
In reply to Hao GeThank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] mm/vmalloc: infinite loop in vunmap_pte_range() on unaligned error rollback - [High] mm/vmalloc…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop in vunmap_pte_range() on unaligned error rollback
- [High] mm/vmalloc: __vmap_pages_range_noflush() calls cond_resched() in atomic context
- [Medium] mm/vmalloc: vmap_page_range() calls flush_cache_vmap() before error rollback --
commit 043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>
mm/vmalloc: undo partial mappings inside the mapping functions
This commit adds rollback logic using __vunmap_range_noflush() and kmsan_vunmap_range_noflush() into vmap_page_range(), __vmap_pages_range_noflush(), and vmap_pages_range_noflush() when page table installation or KMSAN metadata setup fails. It also adds range and page_shift validation in __vmap_pages_range_noflush().
mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370
Quoted 13 lines
mm/vmalloc.c +3
-0@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,if (!err) err =kmsan_ioremap_page_range (addr, end, phys_addr, prot, ioremap_max_page_shift); +if (err) +__vunmap_range_noflush (addr, end); +return err; }
[Severity: High]
What happens here when vmap_range_noflush() returns -EINVAL for a range where end - addr is not PAGE_SIZE-aligned?
In vmap_pte_range(), commit a739e6b557af09 added a check to bail out on unaligned ranges specifically to avoid an infinite loop:
mm/vmalloc.c:vmap_pte_range() {
...
if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr)))
return -EINVAL;
}This can be reached on 64KB page kernels via pci_remap_iospace() -> vmap_page_range() with 16KB/32KB/48KB PCI I/O ranges (for example via loongson_pci_probe() or acpi_pci_root_remap_iospace()).
When vmap_pte_range() returns -EINVAL, vmap_page_range() now calls __vunmap_range_noflush(addr, end) with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as the PCI_IOBASE PMD populated during early boot by reserve_pio_range(), or an earlier PMD mapped in the same call), __vunmap_range_noflush() descends into vunmap_pte_range(), which has no alignment check:
mm/vmalloc.c:vunmap_pte_range() {
...
do {
...
ptent = ptep_get_and_clear(&init_mm, addr, pte);
WARN_ON(!pte_none(ptent) && !pte_present(ptent));
} while (pte += (size >> PAGE_SHIFT), addr += size, addr != end);
}Because end - addr is not a multiple of PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?
[Severity: Medium]
Also in vmap_page_range(), flush_cache_vmap(addr, end) is called unconditionally before checking err and before rolling back with __vunmap_range_noflush(addr, end):
mm/vmalloc.c:vmap_page_range() {
err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
ioremap_max_page_shift);
flush_cache_vmap(addr, end);
if (!err)
err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
ioremap_max_page_shift);
if (err)
__vunmap_range_noflush(addr, end);
return err;
}On UML, flush_cache_vmap is defined as flush_tlb_kernel_range:
arch/um/include/asm/cacheflush.h:
#define flush_cache_vmap flush_tlb_kernel_range
When vmap_range_noflush() fails partway through or kmsan_ioremap_page_range() fails, flush_cache_vmap() runs while the partially installed PTEs are still pte_present(), causing update_pte_range() in arch/um/kernel/tlb.c to map them into the host OS via ops->mmap() and clear _PAGE_NEEDSYNC:
arch/um/kernel/tlb.c:update_pte_range() {
...
if (pte_present(*pte)) {
...
ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE,
prot, fd, offset);
} else
ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE);
*pte = pte_mkuptodate(*pte);
}When __vunmap_range_noflush(addr, end) then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Should flush_cache_vmap(addr, end) only be called after both vmap_range_noflush() and kmsan_ioremap_page_range() succeed?
Quoted 35 lines
mm/vmalloc.c +13
-10@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; +unsigned long start = addr; +int err =0 ; + +if (WARN_ON_ONCE (addr >= end)) +return -EINVAL ;if (WARN_ON_ONCE (page_shift <PAGE_SHIFT ))return -EINVAL ;if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) -return vmap_small_pages_range_noflush (addr, end, prot, pages); - -for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { -int err; - - err =vmap_range_noflush (addr, addr + (1UL << page_shift), + page_shift ==PAGE_SHIFT ) { + err =vmap_small_pages_range_noflush (addr, end, prot, pages); + }else { +for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { + err =vmap_range_noflush (addr, addr + (1UL << page_shift),page_to_phys (pages[i]), prot, page_shift); -if (err) -return err; +if (err) +break ; - addr +=1UL << page_shift; + addr +=1UL << page_shift;
mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 718
Quoted 10 lines
mm/vmalloc.c +6
-1+ } } -return 0 ; +/* Undo the PTEs installed before the failure. */ +if (err) +__vunmap_range_noflush (start, end); + +return err; }
[Severity: High]
Similarly here, __vmap_pages_range_noflush() checks addr >= end, but does not check PAGE_ALIGNED(addr) or PAGE_ALIGNED(end).
If a caller passes a sub-page or unaligned range (for example, 0 < end - addr < PAGE_SIZE), nr = (end - addr) >> PAGE_SHIFT evaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely in vmap_pages_pte_range(). And if the huge-page path fails on an unaligned range, __vunmap_range_noflush(start, end) will hit the same infinite loop in vunmap_pte_range().
Should both vmap_page_range() and __vmap_pages_range_noflush() validate PAGE_ALIGNED(addr) && PAGE_ALIGNED(end) upfront before attempting to map or unmap?
Quoted 8 lines
mm/vmalloc.c +1
-1int vmap_pages_range_noflush(unsigned long addr, unsigned long end,@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,if (ret)return ret; -return __vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +
mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 739
Quoted 10 lines
mm/vmalloc.c +9
-0+ ret =__vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +/* +* The page tables undo themselves on failure. Tear down the +* metadata that was fully set up before the mapping failed. +*/ +if (ret) +kmsan_vunmap_range_noflush (addr, end); + +return ret; }
[Severity: High]
Can calling __vunmap_range_noflush() and kmsan_vunmap_range_noflush() here trigger a sleeping-in-atomic-context bug when __vmalloc() or kvmalloc() is called with GFP_ATOMIC or GFP_NOWAIT?
__vmalloc_area_node() supports non-blocking allocations by wrapping __vmap_pages_range() in memalloc_apply_gfp_scope(gfp_mask):
mm/vmalloc.c:__vmalloc_area_node() {
...
flags = memalloc_apply_gfp_scope(gfp_mask);
do {
ret = __vmap_pages_range(addr, addr + size, prot, area->pages,
page_shift, nested_gfp);
} while ((gfp_mask & __GFP_NOFAIL) && (ret < 0));
memalloc_restore_gfp_scope(flags);
}When a GFP_ATOMIC or GFP_NOWAIT allocation runs in atomic context (such as under a spinlock, preempt_disable(), or rcu_read_lock()) and page table allocation fails in vmap_small_pages_range_noflush() after at least one PMD in [start, end) is populated (or after kmsan_vmap_pages_range_noflush() populated the KMSAN shadow/origin ranges), __vmap_pages_range_noflush() and kmsan_vunmap_range_noflush() synchronously call __vunmap_range_noflush().
__vunmap_range_noflush() walks the populated PMDs via vunmap_pmd_range(), which unconditionally calls cond_resched():
mm/vmalloc.c:vunmap_pmd_range() {
...
do {
next = pmd_addr_end(addr, end);
if (pmd_none_or_clear_bad(pmd))
continue;
vunmap_pte_range(pmd, addr, next, mask);
cond_resched();
} while (pmd++, addr = next, addr != end);
}Commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct") deferred error-path cleanup in __vmalloc_area_node() to a workqueue specifically because unmapping on failure can sleep. Doesn't calling __vunmap_range_noflush() synchronously inside __vmap_pages_range_noflush() and vmap_pages_range_noflush() reintroduce cond_resched() in atomic context?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
[email protected]
AI reviewRe: [PATCH v2 4/5] powerpc: drop redundant unmaps of failed vmap mappings
In reply to Hao GeThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] powerpc/pci: leak of vm_struct in ioremap_phb() when ioremap_page_range() fails --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] powerpc/pci: leak of vm_struct in ioremap_phb() when ioremap_page_range() fails --
commit 293efb97e53f2ab49d30288da6e26fcbf75e032f
Author: Hao Ge <[email protected]>
powerpc: drop redundant unmaps of failed vmap mappings
This commit removes vunmap_range() calls from the error paths of remap_isa_base() and ioremap_phb(), since vmap_page_range() and ioremap_page_range() already clean up their installed PTEs upon failure.
Quoted 6 lines
arch/powerpc/kernel/pci_64.c +0
-0@@ -139,10 +139,8 @@ void __iomem *ioremap_phb(phys_addr_t paddr, unsigned long size)
arch/powerpc/kernel/pci_64.c void __iomem *ioremap_phb(phys_addr_t paddr, unsigned long size) · line 146
Quoted 10 lines
arch/powerpc/kernel/pci_64.c +1
-3addr = (unsigned long )area->addr;if (ioremap_page_range (addr, addr + size, paddr, -pgprot_noncached (PAGE_KERNEL ))) { -vunmap_range (addr, addr + size); +pgprot_noncached (PAGE_KERNEL )))return NULL ; - }return (void __iomem *)addr; }
[Severity: Medium]
Does ioremap_phb() still leak area and its reserved virtual address range in [PHB_IO_BASE, PHB_IO_END) when ioremap_page_range() fails?
Prior to calling ioremap_page_range(), ioremap_phb() allocates a virtual memory area:
arch/powerpc/kernel/pci_64.c:ioremap_phb() {
...
area = __get_vm_area_caller(size, VM_IOREMAP, PHB_IO_BASE, PHB_IO_END,
__builtin_return_address(0));
if (!area)
return NULL;
...
}When ioremap_page_range() fails (for example, -ENOMEM when allocating page tables in vmap_range_noflush()), vmap_page_range() rolls back the PTEs via __vunmap_range_noflush(), but it does not free or unlink the struct vm_struct and struct vmap_area allocated by __get_vm_area_caller().
Should this error path call free_vm_area(area) before returning NULL, matching generic_ioremap_prot() in mm/ioremap.c?
mm/ioremap.c:generic_ioremap_prot() {
...
if (ioremap_page_range(vaddr, vaddr + size, phys_addr, prot)) {
free_vm_area(area);
return NULL;
}
...
}Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions
In reply to Hao GeQuoted 47 lines · Hao Ge
__vmap_pages_range_noflush()and friends can install some PTEs before failing and leave them mapped, and the kernel callers do not agree on who cleans them up.pcpu_map_pages()andkmsan_ioremap_page_range()roll back what they mapped before the failure, whilevm_module_tags_populate()and the__GFP_NOFAILretry loop in__vmalloc_area_node()rely on the mapping functions cleaning up and do not call anything likevunmap_range()themselves. When the same range is mapped again, the attempt hits the leftovers and fails, with aBUG()invmap_pte_range()for huge mappings and a warning on the small-page path.After discussing with Suren and Ulad, we decided to put the rollback into the entry points of the vmap API rather than into every low-level helper.
__vmap_pages_range_noflush()undoes the whole range it was asked to map when it fails, andvmap_page_range()undoes its range for the ioremap-style mappings.The rollback is
__vunmap_range_noflush(), it only clears the PTEs, no TLB flush, nothing has touched these mappings.vmap_pages_range_noflush()drops the KMSAN metadata when the data mapping fails, and an empty range bails out with-EINVALnow, which used toBUG_ON()on the small-page path and map nothing at all on the huge-page one.Reported-by Sashiko <[email protected]>Suggested-by Uladyslau Rezki <[email protected]>Signed-off-by Hao Ge <[email protected]>--- mm/vmalloc.c | 44 ++++++++++++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 12 deletions(-)mm/vmalloc.c +3
-0@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,if (!err) err =kmsan_ioremap_page_range (addr, end, phys_addr, prot, ioremap_max_page_shift); +if (err) +__vunmap_range_noflush (addr, end); +return err; }
mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 693
Quoted 8 lines
mm/vmalloc.c +5
-0@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; +unsigned long start = addr; +int err =0 ; + +if (WARN_ON_ONCE (addr >= end)) +return -EINVAL ;
IIUC this is a general change independent of this patch? Caller could pass addr >= end and if we take the huge mapping path, we lose the sanity check.
So this could become a separate patch.
Quoted 56 lines
mm/vmalloc.c +24
-12if (WARN_ON_ONCE (page_shift <PAGE_SHIFT ))return -EINVAL ;if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) -return vmap_small_pages_range_noflush (addr, end, prot, pages); - -for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { -int err; - - err =vmap_range_noflush (addr, addr + (1UL << page_shift), + page_shift ==PAGE_SHIFT ) { + err =vmap_small_pages_range_noflush (addr, end, prot, pages); + }else { +for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { + err =vmap_range_noflush (addr, addr + (1UL << page_shift),page_to_phys (pages[i]), prot, page_shift); -if (err) -return err; +if (err) +break ; - addr +=1UL << page_shift; + addr +=1UL << page_shift; + } } -return 0 ; +/* Undo the PTEs installed before the failure. */ +if (err) +__vunmap_range_noflush (start, end); + +return err; } int vmap_pages_range_noflush(unsigned long addr, unsigned long end,@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,if (ret)return ret; -return __vmap_pages_range_noflush (addr, end, prot, pages, page_shift); + + ret =__vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +/* +* The page tables undo themselves on failure. Tear down the +* metadata that was fully set up before the mapping failed. +*/ +if (ret) +kmsan_vunmap_range_noflush (addr, end); + +return ret; } static int __vmap_pages_range(unsigned long addr, unsigned long end,
Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions
In reply to [email protected]Hi Suren and Ulad
Sashiko reported several issues, and they all look real. I have some ideas for fixing the first few;
of course, please correct me if I get anything wrong.
I'd like to discuss the last issue with you all.
Quoted 64 lines · On 2026/10/20 13:45, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop invunmap_pte_range()on unaligned error rollback
- [High] mm/vmalloc:__vmap_pages_range_noflush()callscond_resched()in atomic context
- [Medium] mm/vmalloc:vmap_page_range()callsflush_cache_vmap()before error rollback --commit
043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>mm/vmalloc: undo partial mappings inside the mapping functions
This commit adds rollback logic using
__vunmap_range_noflush()andkmsan_vunmap_range_noflush()intovmap_page_range(),__vmap_pages_range_noflush(), andvmap_pages_range_noflush()when page table installation or KMSAN metadata setup fails. It also adds range andpage_shiftvalidation in__vmap_pages_range_noflush().mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370
Quoted 13 lines
mm/vmalloc.c +3
-0@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,if (!err) err =kmsan_ioremap_page_range (addr, end, phys_addr, prot, ioremap_max_page_shift); +if (err) +__vunmap_range_noflush (addr, end); +return err; }[Severity: High]
What happens here whenvmap_range_noflush()returns-EINVALfor a range where end - addr is not PAGE_SIZE-aligned?In
vmap_pte_range(), commita739e6b557af09added a check to bail out on unaligned ranges specifically to avoid an infinite loop:mm/vmalloc.c:vmap_pte_range() { ... if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr))) return -EINVAL; }This can be reached on 64KB page kernels via
pci_remap_iospace()->vmap_page_range()with 16KB/32KB/48KB PCI I/O ranges (for example vialoongson_pci_probe()oracpi_pci_root_remap_iospace()).When
vmap_pte_range()returns-EINVAL,vmap_page_range()now calls__vunmap_range_noflush(addr, end)with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as thePCI_IOBASEPMD populated during early boot byreserve_pio_range(), or an earlier PMD mapped in the same call),__vunmap_range_noflush()descends intovunmap_pte_range(), which has no alignment check:mm/vmalloc.c:vunmap_pte_range() { ... do { ... ptent = ptep_get_and_clear(&init_mm, addr, pte); WARN_ON(!pte_none(ptent) && !pte_present(ptent)); } while (pte += (size >> PAGE_SHIFT), addr += size, addr != end); }
Quoted 4 lines
Because end - addr is not a multiple of
PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?
That sounds reasonable. I will add a separate patch in the next version to fix this problem.
Quoted 41 lines
[Severity: Medium]
Also invmap_page_range(),flush_cache_vmap(addr, end)is called unconditionally before checking err and before rolling back with__vunmap_range_noflush(addr, end):mm/vmalloc.c:vmap_page_range() { err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot), ioremap_max_page_shift); flush_cache_vmap(addr, end); if (!err) err = kmsan_ioremap_page_range(addr, end, phys_addr, prot, ioremap_max_page_shift); if (err) __vunmap_range_noflush(addr, end); return err; }On UML,
flush_cache_vmapis defined asflush_tlb_kernel_range:
arch/um/include/asm/cacheflush.h:
#defineflush_cache_vmapflush_tlb_kernel_rangeWhen
vmap_range_noflush()fails partway through orkmsan_ioremap_page_range()fails,flush_cache_vmap()runs while the partially installed PTEs are stillpte_present(), causingupdate_pte_range()inarch/um/kernel/tlb.cto map them into the host OS viaops->mmap()and clear_PAGE_NEEDSYNC:arch/um/kernel/tlb.c:update_pte_range() { ... if (pte_present(*pte)) { ... ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE, prot, fd, offset); } else ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE); *pte = pte_mkuptodate(*pte); }
Quoted 6 lines
When
__vunmap_range_noflush(addr, end)then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Shouldflush_cache_vmap(addr, end)only be called after bothvmap_range_noflush()andkmsan_ioremap_page_range()succeed?
That sounds reasonable. I will add this logic in the next version.
Quoted 57 lines
Quoted 35 lines
mm/vmalloc.c +13
-10@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; +unsigned long start = addr; +int err =0 ; + +if (WARN_ON_ONCE (addr >= end)) +return -EINVAL ;if (WARN_ON_ONCE (page_shift <PAGE_SHIFT ))return -EINVAL ;if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) -return vmap_small_pages_range_noflush (addr, end, prot, pages); - -for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { -int err; - - err =vmap_range_noflush (addr, addr + (1UL << page_shift), + page_shift ==PAGE_SHIFT ) { + err =vmap_small_pages_range_noflush (addr, end, prot, pages); + }else { +for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { + err =vmap_range_noflush (addr, addr + (1UL << page_shift),page_to_phys (pages[i]), prot, page_shift); -if (err) -return err; +if (err) +break ; - addr +=1UL << page_shift; + addr +=1UL << page_shift;mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 718
Quoted 10 lines
mm/vmalloc.c +6
-1+ } } -return 0 ; +/* Undo the PTEs installed before the failure. */ +if (err) +__vunmap_range_noflush (start, end); + +return err; }[Severity: High]
Similarly here,__vmap_pages_range_noflush()checks addr >= end, but does not checkPAGE_ALIGNED(addr)orPAGE_ALIGNED(end).If a caller passes a sub-page or unaligned range (for example, 0 < end - addr <
PAGE_SIZE), nr = (end - addr) >>PAGE_SHIFTevaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely invmap_pages_pte_range(). And if the huge-page path fails on an unaligned range,__vunmap_range_noflush(start, end)will hit the same infinite loop invunmap_pte_range().
Quoted 4 lines
Should both
vmap_page_range()and__vmap_pages_range_noflush()validatePAGE_ALIGNED(addr)&&PAGE_ALIGNED(end)upfront before attempting to map or unmap?
I think We need to check both PAGE_ALIGNED(addr) and PAGE_ALIGNED(end). As an example: if addr is PAGE-aligned, end is not, and end - addr < PAGE_SIZE, then nr becomes zero. On the huge page path, we simply return 0 with no work performed.
At the same time, vmap_pages_pte_range() needs the same guard as commit a739e6b557af ("mm: vmalloc: WARN_ON if mapping size is not PAGE_SIZE aligned"), to protect against unaligned ranges.
So will do next version.
Quoted 59 lines
Quoted 8 lines
mm/vmalloc.c +1
-1int vmap_pages_range_noflush(unsigned long addr, unsigned long end,@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,if (ret)return ret; -return __vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 739
Quoted 10 lines
mm/vmalloc.c +9
-0+ ret =__vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +/* +* The page tables undo themselves on failure. Tear down the +* metadata that was fully set up before the mapping failed. +*/ +if (ret) +kmsan_vunmap_range_noflush (addr, end); + +return ret; }[Severity: High]
Can calling__vunmap_range_noflush()andkmsan_vunmap_range_noflush()here trigger a sleeping-in-atomic-context bug when__vmalloc()orkvmalloc()is called withGFP_ATOMICorGFP_NOWAIT?
__vmalloc_area_node()supports non-blocking allocations by wrapping__vmap_pages_range()inmemalloc_apply_gfp_scope(gfp_mask):mm/vmalloc.c:__vmalloc_area_node() { ... flags = memalloc_apply_gfp_scope(gfp_mask); do { ret = __vmap_pages_range(addr, addr + size, prot, area->pages, page_shift, nested_gfp); } while ((gfp_mask & __GFP_NOFAIL) && (ret < 0)); memalloc_restore_gfp_scope(flags); }When a
GFP_ATOMICorGFP_NOWAITallocation runs in atomic context (such as under a spinlock,preempt_disable(), orrcu_read_lock()) and page table allocation fails invmap_small_pages_range_noflush()after at least one PMD in [start, end) is populated (or afterkmsan_vmap_pages_range_noflush()populated the KMSAN shadow/origin ranges),__vmap_pages_range_noflush()andkmsan_vunmap_range_noflush()synchronously call__vunmap_range_noflush().
__vunmap_range_noflush()walks the populated PMDs viavunmap_pmd_range(), which unconditionally callscond_resched():mm/vmalloc.c:vunmap_pmd_range() { ... do { next = pmd_addr_end(addr, end); if (pmd_none_or_clear_bad(pmd)) continue; vunmap_pte_range(pmd, addr, next, mask); cond_resched(); } while (pmd++, addr = next, addr != end); }
Quoted 7 lines
Commit
9c47753167a6("mm/vmalloc: defer freeing partly initializedvm_struct") deferred error-path cleanup in__vmalloc_area_node()to a workqueue specifically because unmapping on failure can sleep. Doesn't calling__vunmap_range_noflush()synchronously inside__vmap_pages_range_noflush()andvmap_pages_range_noflush()reintroducecond_resched()in atomic context?
This issue also appears real. One possible approach is to plumb the caller's gfp_mask into vunmap_pmd_range() to conditionally skip cond_resched().
Suren, Ulad, do you have any preferred or cleaner way to handle this?
Thanks
Best Regards
Hao
Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions
In reply to Dev JainHi Dev
Quoted 60 lines · Dev Jain
Quoted 47 lines · Hao Ge
__vmap_pages_range_noflush()and friends can install some PTEs before failing and leave them mapped, and the kernel callers do not agree on who cleans them up.pcpu_map_pages()andkmsan_ioremap_page_range()roll back what they mapped before the failure, whilevm_module_tags_populate()and the__GFP_NOFAILretry loop in__vmalloc_area_node()rely on the mapping functions cleaning up and do not call anything likevunmap_range()themselves. When the same range is mapped again, the attempt hits the leftovers and fails, with aBUG()invmap_pte_range()for huge mappings and a warning on the small-page path.After discussing with Suren and Ulad, we decided to put the rollback into the entry points of the vmap API rather than into every low-level helper.
__vmap_pages_range_noflush()undoes the whole range it was asked to map when it fails, andvmap_page_range()undoes its range for the ioremap-style mappings.The rollback is
__vunmap_range_noflush(), it only clears the PTEs, no TLB flush, nothing has touched these mappings.vmap_pages_range_noflush()drops the KMSAN metadata when the data mapping fails, and an empty range bails out with-EINVALnow, which used toBUG_ON()on the small-page path and map nothing at all on the huge-page one.Reported-by Sashiko <[email protected]>Suggested-by Uladyslau Rezki <[email protected]>Signed-off-by Hao Ge <[email protected]>--- mm/vmalloc.c | 44 ++++++++++++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 12 deletions(-)mm/vmalloc.c +3
-0@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,if (!err) err =kmsan_ioremap_page_range (addr, end, phys_addr, prot, ioremap_max_page_shift); +if (err) +__vunmap_range_noflush (addr, end); +return err; }mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 693
Quoted 8 lines
mm/vmalloc.c +5
-0@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; +unsigned long start = addr; +int err =0 ; + +if (WARN_ON_ONCE (addr >= end)) +return -EINVAL ;IIUC this is a general change independent of this patch? Caller could pass addr >= end and if we take the huge mapping path, we lose the sanity check.
Quoted 1 line
So this could become a separate patch.
OK, and Sashiko has flagged another case:
when 0 < end - addr < PAGE_SIZE, nr = (end - addr) >> PAGE_SHIFT evaluates to 0,
so the huge-page path still maps nothing and returns 0.
So this separate patch would cover this case as well.
Thanks
Best Regards
Hao
Quoted 58 lines
Quoted 58 lines
mm/vmalloc.c +24
-12if (WARN_ON_ONCE (page_shift <PAGE_SHIFT ))return -EINVAL ;if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) -return vmap_small_pages_range_noflush (addr, end, prot, pages); - -for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { -int err; - - err =vmap_range_noflush (addr, addr + (1UL << page_shift), + page_shift ==PAGE_SHIFT ) { + err =vmap_small_pages_range_noflush (addr, end, prot, pages); + }else { +for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { + err =vmap_range_noflush (addr, addr + (1UL << page_shift),page_to_phys (pages[i]), prot, page_shift); -if (err) -return err; +if (err) +break ; - addr +=1UL << page_shift; + addr +=1UL << page_shift; + } } -return 0 ; +/* Undo the PTEs installed before the failure. */ +if (err) +__vunmap_range_noflush (start, end); + +return err; } int vmap_pages_range_noflush(unsigned long addr, unsigned long end,@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,if (ret)return ret; -return __vmap_pages_range_noflush (addr, end, prot, pages, page_shift); + + ret =__vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +/* +* The page tables undo themselves on failure. Tear down the +* metadata that was fully set up before the mapping failed. +*/ +if (ret) +kmsan_vunmap_range_noflush (addr, end); + +return ret; } static int __vmap_pages_range(unsigned long addr, unsigned long end,
Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions
In reply to Hao GeQuoted 264 lines · Hao Ge
Hi Suren and Ulad
Sashiko reported several issues, and they all look real. I have some ideas for fixing the first few;
of course, please correct me if I get anything wrong.I'd like to discuss the last issue with you all.
Quoted 63 lines · On 2026/10/20 13:45, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop invunmap_pte_range()on unaligned error rollback
- [High] mm/vmalloc:__vmap_pages_range_noflush()callscond_resched()in atomic context
- [Medium] mm/vmalloc:vmap_page_range()callsflush_cache_vmap()before error rollback --commit
043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>mm/vmalloc: undo partial mappings inside the mapping functions
This commit adds rollback logic using
__vunmap_range_noflush()andkmsan_vunmap_range_noflush()intovmap_page_range(),__vmap_pages_range_noflush(), andvmap_pages_range_noflush()when page table installation or KMSAN metadata setup fails. It also adds range andpage_shiftvalidation in__vmap_pages_range_noflush().mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370
Quoted 13 lines
mm/vmalloc.c +3
-0@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,if (!err) err =kmsan_ioremap_page_range (addr, end, phys_addr, prot, ioremap_max_page_shift); +if (err) +__vunmap_range_noflush (addr, end); +return err; }[Severity: High]
What happens here whenvmap_range_noflush()returns-EINVALfor a range where end - addr is not PAGE_SIZE-aligned?In
vmap_pte_range(), commita739e6b557af09added a check to bail out on unaligned ranges specifically to avoid an infinite loop:mm/vmalloc.c:vmap_pte_range() { ... if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr))) return -EINVAL; }This can be reached on 64KB page kernels via
pci_remap_iospace()->vmap_page_range()with 16KB/32KB/48KB PCI I/O ranges (for example vialoongson_pci_probe()oracpi_pci_root_remap_iospace()).When
vmap_pte_range()returns-EINVAL,vmap_page_range()now calls__vunmap_range_noflush(addr, end)with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as thePCI_IOBASEPMD populated during early boot byreserve_pio_range(), or an earlier PMD mapped in the same call),__vunmap_range_noflush()descends intovunmap_pte_range(), which has no alignment check:mm/vmalloc.c:vunmap_pte_range() { ... do { ... ptent = ptep_get_and_clear(&init_mm, addr, pte); WARN_ON(!pte_none(ptent) && !pte_present(ptent)); } while (pte += (size >> PAGE_SHIFT), addr += size, addr != end); }Quoted 4 lines
Because end - addr is not a multiple of
PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?That sounds reasonable. I will add a separate patch in the next version to fix this problem.
Quoted 41 lines
[Severity: Medium]
Also invmap_page_range(),flush_cache_vmap(addr, end)is called unconditionally before checking err and before rolling back with__vunmap_range_noflush(addr, end):mm/vmalloc.c:vmap_page_range() { err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot), ioremap_max_page_shift); flush_cache_vmap(addr, end); if (!err) err = kmsan_ioremap_page_range(addr, end, phys_addr, prot, ioremap_max_page_shift); if (err) __vunmap_range_noflush(addr, end); return err; }On UML,
flush_cache_vmapis defined asflush_tlb_kernel_range:
arch/um/include/asm/cacheflush.h:
#defineflush_cache_vmapflush_tlb_kernel_rangeWhen
vmap_range_noflush()fails partway through orkmsan_ioremap_page_range()fails,flush_cache_vmap()runs while the partially installed PTEs are stillpte_present(), causingupdate_pte_range()inarch/um/kernel/tlb.cto map them into the host OS viaops->mmap()and clear_PAGE_NEEDSYNC:arch/um/kernel/tlb.c:update_pte_range() { ... if (pte_present(*pte)) { ... ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE, prot, fd, offset); } else ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE); *pte = pte_mkuptodate(*pte); }Quoted 6 lines
When
__vunmap_range_noflush(addr, end)then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Shouldflush_cache_vmap(addr, end)only be called after bothvmap_range_noflush()andkmsan_ioremap_page_range()succeed?That sounds reasonable. I will add this logic in the next version.
Quoted 56 lines
Quoted 35 lines
mm/vmalloc.c +13
-10@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; +unsigned long start = addr; +int err =0 ; + +if (WARN_ON_ONCE (addr >= end)) +return -EINVAL ;if (WARN_ON_ONCE (page_shift <PAGE_SHIFT ))return -EINVAL ;if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) -return vmap_small_pages_range_noflush (addr, end, prot, pages); - -for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { -int err; - - err =vmap_range_noflush (addr, addr + (1UL << page_shift), + page_shift ==PAGE_SHIFT ) { + err =vmap_small_pages_range_noflush (addr, end, prot, pages); + }else { +for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { + err =vmap_range_noflush (addr, addr + (1UL << page_shift),page_to_phys (pages[i]), prot, page_shift); -if (err) -return err; +if (err) +break ; - addr +=1UL << page_shift; + addr +=1UL << page_shift;mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 718
Quoted 10 lines
mm/vmalloc.c +6
-1+ } } -return 0 ; +/* Undo the PTEs installed before the failure. */ +if (err) +__vunmap_range_noflush (start, end); + +return err; }[Severity: High]
Similarly here,__vmap_pages_range_noflush()checks addr >= end, but does not checkPAGE_ALIGNED(addr)orPAGE_ALIGNED(end).If a caller passes a sub-page or unaligned range (for example, 0 < end - addr <
PAGE_SIZE), nr = (end - addr) >>PAGE_SHIFTevaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely invmap_pages_pte_range(). And if the huge-page path fails on an unaligned range,__vunmap_range_noflush(start, end)will hit the same infinite loop invunmap_pte_range().Quoted 4 lines
Should both
vmap_page_range()and__vmap_pages_range_noflush()validatePAGE_ALIGNED(addr)&&PAGE_ALIGNED(end)upfront before attempting to map or unmap?I think We need to check both
PAGE_ALIGNED(addr)andPAGE_ALIGNED(end). As an example: if addr is PAGE-aligned, end is not, and end - addr <PAGE_SIZE, then nr becomes zero. On the huge page path, we simply return 0 with no work performed.At the same time,
vmap_pages_pte_range()needs the same guard as commita739e6b557af("mm: vmalloc:WARN_ONif mapping size is notPAGE_SIZEaligned"), to protect against unaligned ranges.So will do next version.
Quoted 58 lines
Quoted 8 lines
mm/vmalloc.c +1
-1int vmap_pages_range_noflush(unsigned long addr, unsigned long end,@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,if (ret)return ret; -return __vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 739
Quoted 10 lines
mm/vmalloc.c +9
-0+ ret =__vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +/* +* The page tables undo themselves on failure. Tear down the +* metadata that was fully set up before the mapping failed. +*/ +if (ret) +kmsan_vunmap_range_noflush (addr, end); + +return ret; }[Severity: High]
Can calling__vunmap_range_noflush()andkmsan_vunmap_range_noflush()here trigger a sleeping-in-atomic-context bug when__vmalloc()orkvmalloc()is called withGFP_ATOMICorGFP_NOWAIT?
__vmalloc_area_node()supports non-blocking allocations by wrapping__vmap_pages_range()inmemalloc_apply_gfp_scope(gfp_mask):mm/vmalloc.c:__vmalloc_area_node() { ... flags = memalloc_apply_gfp_scope(gfp_mask); do { ret = __vmap_pages_range(addr, addr + size, prot, area->pages, page_shift, nested_gfp); } while ((gfp_mask & __GFP_NOFAIL) && (ret < 0)); memalloc_restore_gfp_scope(flags); }When a
GFP_ATOMICorGFP_NOWAITallocation runs in atomic context (such as under a spinlock,preempt_disable(), orrcu_read_lock()) and page table allocation fails invmap_small_pages_range_noflush()after at least one PMD in [start, end) is populated (or afterkmsan_vmap_pages_range_noflush()populated the KMSAN shadow/origin ranges),__vmap_pages_range_noflush()andkmsan_vunmap_range_noflush()synchronously call__vunmap_range_noflush().
__vunmap_range_noflush()walks the populated PMDs viavunmap_pmd_range(), which unconditionally callscond_resched():mm/vmalloc.c:vunmap_pmd_range() { ... do { next = pmd_addr_end(addr, end); if (pmd_none_or_clear_bad(pmd)) continue; vunmap_pte_range(pmd, addr, next, mask); cond_resched(); } while (pmd++, addr = next, addr != end); }Quoted 7 lines
Commit
9c47753167a6("mm/vmalloc: defer freeing partly initializedvm_struct") deferred error-path cleanup in__vmalloc_area_node()to a workqueue specifically because unmapping on failure can sleep. Doesn't calling__vunmap_range_noflush()synchronously inside__vmap_pages_range_noflush()andvmap_pages_range_noflush()reintroducecond_resched()in atomic context?
Quoted 3 lines
This issue also appears real. One possible approach is to plumb the caller's
gfp_maskintovunmap_pmd_range()to conditionally skipcond_resched().
Suren, Ulad, do you have any preferred or cleaner way to handle this?
Alternatively, we can follow the approach from commit 9c47753167a6 ("mm/vmalloc: defer freeing partly initialized vm_struct"):
defer the cleanup to a workqueue when |gfp_mask| has GFP_ATOMIC or GFP_NOWAIT,and perform cleanup synchronously
if we are already in process context. Anyway, I will look into this further tomorrow.
Quoted 5 lines
Thanks
Best Regards
Hao
Uladzislau Rezki
maintainerRe: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions
Quoted 265 lines · Hao Ge
Quoted 258 lines · Hao Ge
Hi Suren and Ulad
Sashiko reported several issues, and they all look real. I have some ideas for fixing the first few;
of course, please correct me if I get anything wrong.I'd like to discuss the last issue with you all.
Quoted 63 lines · On 2026/10/20 13:45, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop invunmap_pte_range()on unaligned error rollback
- [High] mm/vmalloc:__vmap_pages_range_noflush()callscond_resched()in atomic context
- [Medium] mm/vmalloc:vmap_page_range()callsflush_cache_vmap()before error rollback --commit
043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>mm/vmalloc: undo partial mappings inside the mapping functions
This commit adds rollback logic using
__vunmap_range_noflush()andkmsan_vunmap_range_noflush()intovmap_page_range(),__vmap_pages_range_noflush(), andvmap_pages_range_noflush()when page table installation or KMSAN metadata setup fails. It also adds range andpage_shiftvalidation in__vmap_pages_range_noflush().mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370
Quoted 13 lines
mm/vmalloc.c +3
-0@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,if (!err) err =kmsan_ioremap_page_range (addr, end, phys_addr, prot, ioremap_max_page_shift); +if (err) +__vunmap_range_noflush (addr, end); +return err; }[Severity: High]
What happens here whenvmap_range_noflush()returns-EINVALfor a range where end - addr is not PAGE_SIZE-aligned?In
vmap_pte_range(), commita739e6b557af09added a check to bail out on unaligned ranges specifically to avoid an infinite loop:mm/vmalloc.c:vmap_pte_range() { ... if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr))) return -EINVAL; }This can be reached on 64KB page kernels via
pci_remap_iospace()->vmap_page_range()with 16KB/32KB/48KB PCI I/O ranges (for example vialoongson_pci_probe()oracpi_pci_root_remap_iospace()).When
vmap_pte_range()returns-EINVAL,vmap_page_range()now calls__vunmap_range_noflush(addr, end)with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as thePCI_IOBASEPMD populated during early boot byreserve_pio_range(), or an earlier PMD mapped in the same call),__vunmap_range_noflush()descends intovunmap_pte_range(), which has no alignment check:mm/vmalloc.c:vunmap_pte_range() { ... do { ... ptent = ptep_get_and_clear(&init_mm, addr, pte); WARN_ON(!pte_none(ptent) && !pte_present(ptent)); } while (pte += (size >> PAGE_SHIFT), addr += size, addr != end); }Quoted 4 lines
Because end - addr is not a multiple of
PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?That sounds reasonable. I will add a separate patch in the next version to fix this problem.
Quoted 41 lines
[Severity: Medium]
Also invmap_page_range(),flush_cache_vmap(addr, end)is called unconditionally before checking err and before rolling back with__vunmap_range_noflush(addr, end):mm/vmalloc.c:vmap_page_range() { err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot), ioremap_max_page_shift); flush_cache_vmap(addr, end); if (!err) err = kmsan_ioremap_page_range(addr, end, phys_addr, prot, ioremap_max_page_shift); if (err) __vunmap_range_noflush(addr, end); return err; }On UML,
flush_cache_vmapis defined asflush_tlb_kernel_range:
arch/um/include/asm/cacheflush.h:
#defineflush_cache_vmapflush_tlb_kernel_rangeWhen
vmap_range_noflush()fails partway through orkmsan_ioremap_page_range()fails,flush_cache_vmap()runs while the partially installed PTEs are stillpte_present(), causingupdate_pte_range()inarch/um/kernel/tlb.cto map them into the host OS viaops->mmap()and clear_PAGE_NEEDSYNC:arch/um/kernel/tlb.c:update_pte_range() { ... if (pte_present(*pte)) { ... ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE, prot, fd, offset); } else ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE); *pte = pte_mkuptodate(*pte); }Quoted 6 lines
When
__vunmap_range_noflush(addr, end)then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Shouldflush_cache_vmap(addr, end)only be called after bothvmap_range_noflush()andkmsan_ioremap_page_range()succeed?That sounds reasonable. I will add this logic in the next version.
Quoted 52 lines
Quoted 31 lines
mm/vmalloc.c +12
-10@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; +unsigned long start = addr; +int err =0 ; + +if (WARN_ON_ONCE (addr >= end)) +return -EINVAL ;if (WARN_ON_ONCE (page_shift <PAGE_SHIFT ))return -EINVAL ;if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) -return vmap_small_pages_range_noflush (addr, end, prot, pages); - -for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { -int err; - - err =vmap_range_noflush (addr, addr + (1UL << page_shift), + page_shift ==PAGE_SHIFT ) { + err =vmap_small_pages_range_noflush (addr, end, prot, pages); + }else { +for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { + err =vmap_range_noflush (addr, addr + (1UL << page_shift),page_to_phys (pages[i]), prot, page_shift); -if (err) -return err; +if (err) +break ; - addr +=1UL << page_shift;mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 714
Quoted 10 lines
mm/vmalloc.c +7
-1+ addr +=1UL << page_shift; + } } -return 0 ; +/* Undo the PTEs installed before the failure. */ +if (err) +__vunmap_range_noflush (start, end); + +return err; }[Severity: High]
Similarly here,__vmap_pages_range_noflush()checks addr >= end, but does not checkPAGE_ALIGNED(addr)orPAGE_ALIGNED(end).If a caller passes a sub-page or unaligned range (for example, 0 < end - addr <
PAGE_SIZE), nr = (end - addr) >>PAGE_SHIFTevaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely invmap_pages_pte_range(). And if the huge-page path fails on an unaligned range,__vunmap_range_noflush(start, end)will hit the same infinite loop invunmap_pte_range().Quoted 4 lines
Should both
vmap_page_range()and__vmap_pages_range_noflush()validatePAGE_ALIGNED(addr)&&PAGE_ALIGNED(end)upfront before attempting to map or unmap?I think We need to check both
PAGE_ALIGNED(addr)andPAGE_ALIGNED(end). As an example: if addr is PAGE-aligned, end is not, and end - addr <PAGE_SIZE, then nr becomes zero. On the huge page path, we simply return 0 with no work performed.At the same time,
vmap_pages_pte_range()needs the same guard as commita739e6b557af("mm: vmalloc:WARN_ONif mapping size is notPAGE_SIZEaligned"), to protect against unaligned ranges.So will do next version.
Quoted 56 lines
Quoted 6 lines
mm/vmalloc.c +1
-1int vmap_pages_range_noflush(unsigned long addr, unsigned long end,@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,if (ret)return ret; -return __vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 738
Quoted 10 lines
mm/vmalloc.c +9
-0+ ret =__vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +/* +* The page tables undo themselves on failure. Tear down the +* metadata that was fully set up before the mapping failed. +*/ +if (ret) +kmsan_vunmap_range_noflush (addr, end); + +return ret; }[Severity: High]
Can calling__vunmap_range_noflush()andkmsan_vunmap_range_noflush()here trigger a sleeping-in-atomic-context bug when__vmalloc()orkvmalloc()is called withGFP_ATOMICorGFP_NOWAIT?
__vmalloc_area_node()supports non-blocking allocations by wrapping__vmap_pages_range()inmemalloc_apply_gfp_scope(gfp_mask):mm/vmalloc.c:__vmalloc_area_node() { ... flags = memalloc_apply_gfp_scope(gfp_mask); do { ret = __vmap_pages_range(addr, addr + size, prot, area->pages, page_shift, nested_gfp); } while ((gfp_mask & __GFP_NOFAIL) && (ret < 0)); memalloc_restore_gfp_scope(flags); }When a
GFP_ATOMICorGFP_NOWAITallocation runs in atomic context (such as under a spinlock,preempt_disable(), orrcu_read_lock()) and page table allocation fails invmap_small_pages_range_noflush()after at least one PMD in [start, end) is populated (or afterkmsan_vmap_pages_range_noflush()populated the KMSAN shadow/origin ranges),__vmap_pages_range_noflush()andkmsan_vunmap_range_noflush()synchronously call__vunmap_range_noflush().
__vunmap_range_noflush()walks the populated PMDs viavunmap_pmd_range(), which unconditionally callscond_resched():mm/vmalloc.c:vunmap_pmd_range() { ... do { next = pmd_addr_end(addr, end); if (pmd_none_or_clear_bad(pmd)) continue; vunmap_pte_range(pmd, addr, next, mask); cond_resched(); } while (pmd++, addr = next, addr != end); }Quoted 7 lines
Commit
9c47753167a6("mm/vmalloc: defer freeing partly initializedvm_struct") deferred error-path cleanup in__vmalloc_area_node()to a workqueue specifically because unmapping on failure can sleep. Doesn't calling__vunmap_range_noflush()synchronously inside__vmap_pages_range_noflush()andvmap_pages_range_noflush()reintroducecond_resched()in atomic context?Quoted 3 lines
This issue also appears real. One possible approach is to plumb the caller's
gfp_maskintovunmap_pmd_range()to conditionally skipcond_resched().
Suren, Ulad, do you have any preferred or cleaner way to handle this?
Quoted 9 lines
Alternatively, we can follow the approach from commit
9c47753167a6("mm/vmalloc: defer freeing partly initializedvm_struct"):defer the cleanup to a workqueue when |gfp_mask| has GFP_ATOMIC or GFP_NOWAIT,and perform cleanup synchronously
if we are already in process context. Anyway, I will look into this further tomorrow.
I will check the AI concern.
--
Uladzislau Rezki
Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions
Hi Uladzislau and Suren
Quoted 269 lines · Uladzislau Rezki
Quoted 268 lines · Hao Ge
Quoted 258 lines · Hao Ge
Hi Suren and Ulad
Sashiko reported several issues, and they all look real. I have some ideas for fixing the first few;
of course, please correct me if I get anything wrong.I'd like to discuss the last issue with you all.
Quoted 63 lines · On 2026/10/20 13:45, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop invunmap_pte_range()on unaligned error rollback
- [High] mm/vmalloc:__vmap_pages_range_noflush()callscond_resched()in atomic context
- [Medium] mm/vmalloc:vmap_page_range()callsflush_cache_vmap()before error rollback --commit
043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>mm/vmalloc: undo partial mappings inside the mapping functions
This commit adds rollback logic using
__vunmap_range_noflush()andkmsan_vunmap_range_noflush()intovmap_page_range(),__vmap_pages_range_noflush(), andvmap_pages_range_noflush()when page table installation or KMSAN metadata setup fails. It also adds range andpage_shiftvalidation in__vmap_pages_range_noflush().mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370
Quoted 13 lines
mm/vmalloc.c +3
-0@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,if (!err) err =kmsan_ioremap_page_range (addr, end, phys_addr, prot, ioremap_max_page_shift); +if (err) +__vunmap_range_noflush (addr, end); +return err; }[Severity: High]
What happens here whenvmap_range_noflush()returns-EINVALfor a range where end - addr is not PAGE_SIZE-aligned?In
vmap_pte_range(), commita739e6b557af09added a check to bail out on unaligned ranges specifically to avoid an infinite loop:mm/vmalloc.c:vmap_pte_range() { ... if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr))) return -EINVAL; }This can be reached on 64KB page kernels via
pci_remap_iospace()->vmap_page_range()with 16KB/32KB/48KB PCI I/O ranges (for example vialoongson_pci_probe()oracpi_pci_root_remap_iospace()).When
vmap_pte_range()returns-EINVAL,vmap_page_range()now calls__vunmap_range_noflush(addr, end)with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as thePCI_IOBASEPMD populated during early boot byreserve_pio_range(), or an earlier PMD mapped in the same call),__vunmap_range_noflush()descends intovunmap_pte_range(), which has no alignment check:mm/vmalloc.c:vunmap_pte_range() { ... do { ... ptent = ptep_get_and_clear(&init_mm, addr, pte); WARN_ON(!pte_none(ptent) && !pte_present(ptent)); } while (pte += (size >> PAGE_SHIFT), addr += size, addr != end); }Quoted 4 lines
Because end - addr is not a multiple of
PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?That sounds reasonable. I will add a separate patch in the next version to fix this problem.
Quoted 41 lines
[Severity: Medium]
Also invmap_page_range(),flush_cache_vmap(addr, end)is called unconditionally before checking err and before rolling back with__vunmap_range_noflush(addr, end):mm/vmalloc.c:vmap_page_range() { err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot), ioremap_max_page_shift); flush_cache_vmap(addr, end); if (!err) err = kmsan_ioremap_page_range(addr, end, phys_addr, prot, ioremap_max_page_shift); if (err) __vunmap_range_noflush(addr, end); return err; }On UML,
flush_cache_vmapis defined asflush_tlb_kernel_range:
arch/um/include/asm/cacheflush.h:
#defineflush_cache_vmapflush_tlb_kernel_rangeWhen
vmap_range_noflush()fails partway through orkmsan_ioremap_page_range()fails,flush_cache_vmap()runs while the partially installed PTEs are stillpte_present(), causingupdate_pte_range()inarch/um/kernel/tlb.cto map them into the host OS viaops->mmap()and clear_PAGE_NEEDSYNC:arch/um/kernel/tlb.c:update_pte_range() { ... if (pte_present(*pte)) { ... ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE, prot, fd, offset); } else ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE); *pte = pte_mkuptodate(*pte); }Quoted 6 lines
When
__vunmap_range_noflush(addr, end)then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Shouldflush_cache_vmap(addr, end)only be called after bothvmap_range_noflush()andkmsan_ioremap_page_range()succeed?That sounds reasonable. I will add this logic in the next version.
Quoted 52 lines
Quoted 31 lines
mm/vmalloc.c +12
-10@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; +unsigned long start = addr; +int err =0 ; + +if (WARN_ON_ONCE (addr >= end)) +return -EINVAL ;if (WARN_ON_ONCE (page_shift <PAGE_SHIFT ))return -EINVAL ;if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) -return vmap_small_pages_range_noflush (addr, end, prot, pages); - -for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { -int err; - - err =vmap_range_noflush (addr, addr + (1UL << page_shift), + page_shift ==PAGE_SHIFT ) { + err =vmap_small_pages_range_noflush (addr, end, prot, pages); + }else { +for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { + err =vmap_range_noflush (addr, addr + (1UL << page_shift),page_to_phys (pages[i]), prot, page_shift); -if (err) -return err; +if (err) +break ; - addr +=1UL << page_shift;mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 714
Quoted 10 lines
mm/vmalloc.c +7
-1+ addr +=1UL << page_shift; + } } -return 0 ; +/* Undo the PTEs installed before the failure. */ +if (err) +__vunmap_range_noflush (start, end); + +return err; }[Severity: High]
Similarly here,__vmap_pages_range_noflush()checks addr >= end, but does not checkPAGE_ALIGNED(addr)orPAGE_ALIGNED(end).If a caller passes a sub-page or unaligned range (for example, 0 < end - addr <
PAGE_SIZE), nr = (end - addr) >>PAGE_SHIFTevaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely invmap_pages_pte_range(). And if the huge-page path fails on an unaligned range,__vunmap_range_noflush(start, end)will hit the same infinite loop invunmap_pte_range().Quoted 4 lines
Should both
vmap_page_range()and__vmap_pages_range_noflush()validatePAGE_ALIGNED(addr)&&PAGE_ALIGNED(end)upfront before attempting to map or unmap?I think We need to check both
PAGE_ALIGNED(addr)andPAGE_ALIGNED(end). As an example: if addr is PAGE-aligned, end is not, and end - addr <PAGE_SIZE, then nr becomes zero. On the huge page path, we simply return 0 with no work performed.At the same time,
vmap_pages_pte_range()needs the same guard as commita739e6b557af("mm: vmalloc:WARN_ONif mapping size is notPAGE_SIZEaligned"), to protect against unaligned ranges.So will do next version.
Quoted 56 lines
Quoted 6 lines
mm/vmalloc.c +1
-1int vmap_pages_range_noflush(unsigned long addr, unsigned long end,@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,if (ret)return ret; -return __vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 738
Quoted 10 lines
mm/vmalloc.c +9
-0+ ret =__vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +/* +* The page tables undo themselves on failure. Tear down the +* metadata that was fully set up before the mapping failed. +*/ +if (ret) +kmsan_vunmap_range_noflush (addr, end); + +return ret; }[Severity: High]
Can calling__vunmap_range_noflush()andkmsan_vunmap_range_noflush()here trigger a sleeping-in-atomic-context bug when__vmalloc()orkvmalloc()is called withGFP_ATOMICorGFP_NOWAIT?
__vmalloc_area_node()supports non-blocking allocations by wrapping__vmap_pages_range()inmemalloc_apply_gfp_scope(gfp_mask):mm/vmalloc.c:__vmalloc_area_node() { ... flags = memalloc_apply_gfp_scope(gfp_mask); do { ret = __vmap_pages_range(addr, addr + size, prot, area->pages, page_shift, nested_gfp); } while ((gfp_mask & __GFP_NOFAIL) && (ret < 0)); memalloc_restore_gfp_scope(flags); }When a
GFP_ATOMICorGFP_NOWAITallocation runs in atomic context (such as under a spinlock,preempt_disable(), orrcu_read_lock()) and page table allocation fails invmap_small_pages_range_noflush()after at least one PMD in [start, end) is populated (or afterkmsan_vmap_pages_range_noflush()populated the KMSAN shadow/origin ranges),__vmap_pages_range_noflush()andkmsan_vunmap_range_noflush()synchronously call__vunmap_range_noflush().
__vunmap_range_noflush()walks the populated PMDs viavunmap_pmd_range(), which unconditionally callscond_resched():mm/vmalloc.c:vunmap_pmd_range() { ... do { next = pmd_addr_end(addr, end); if (pmd_none_or_clear_bad(pmd)) continue; vunmap_pte_range(pmd, addr, next, mask); cond_resched(); } while (pmd++, addr = next, addr != end); }Quoted 7 lines
Commit
9c47753167a6("mm/vmalloc: defer freeing partly initializedvm_struct") deferred error-path cleanup in__vmalloc_area_node()to a workqueue specifically because unmapping on failure can sleep. Doesn't calling__vunmap_range_noflush()synchronously inside__vmap_pages_range_noflush()andvmap_pages_range_noflush()reintroducecond_resched()in atomic context?Quoted 3 lines
This issue also appears real. One possible approach is to plumb the caller's
gfp_maskintovunmap_pmd_range()to conditionally skipcond_resched().
Suren, Ulad, do you have any preferred or cleaner way to handle this?Alternatively, we can follow the approach from commit
9c47753167a6("mm/vmalloc: defer freeing partly initializedvm_struct"):
Quoted 8 lines
Quoted 6 lines
defer the cleanup to a workqueue when |gfp_mask| has GFP_ATOMIC or GFP_NOWAIT,and perform cleanup synchronously
if we are already in process context. Anyway, I will look into this further tomorrow.
I will check the AI concern.
Thank you very much for the quick reply and for offering to check the AI concern. Much appreciated!
Here is a summary of what I have found over the last few days.
I went through the history first. The cond_resched() in vunmap_pmd_range() was added by commit e47110e90584 ("mm/vunmap: add cond_resched() in vunmap_pmd_range"), modeled after zap_pte_range(). It fixed a real soft lockup: on a non-preemptible kernel, unmapping a large persistent-memory namespace (hence a large I/O map region) kept a CPU busy for 23 seconds inside the walk:
watchdog: BUG: soft lockup - CPU#49 stuck for 23s! [ndctl:50777]
Then I looked at how vfree() handles atomic contexts: it defers the whole free to a workqueue from interrupt context (vfree_atomic()), and 9c47753167a6 does the same for a partly initialized vm_struct.
Unfortunately I do not think deferral works for this rollback case. A deferred rollback returns with the PTEs still installed, and if that range is mapped again before the workqueue runs, the mapping hits the leftovers: a warning and -EBUSY on the small-page path, a BUG() on the huge path.
So I kept it simple: plumb the caller's gfp_mask down to the unmap walk and call cond_resched() conditionally, like
commit 660654f90e7f ("mm/vmalloc.c: add a schedule point to vmalloc()") did:
if (gfp_mask & __GFP_WAIT) cond_resched();
Of course, this may not be the perfect solution either. I would gladly hear your thoughts on it.
Thanks
Best Regards
Hao
Quoted 2 lines
--
Uladzislau Rezki
Uladzislau Rezki
maintainerRe: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions
Quoted 303 lines · Hao Ge
Hi Uladzislau and Suren
Quoted 269 lines · Uladzislau Rezki
Quoted 268 lines · Hao Ge
Quoted 258 lines · Hao Ge
Hi Suren and Ulad
Sashiko reported several issues, and they all look real. I have some ideas for fixing the first few;
of course, please correct me if I get anything wrong.I'd like to discuss the last issue with you all.
Quoted 63 lines · On 2026/10/20 13:45, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop invunmap_pte_range()on unaligned error rollback
- [High] mm/vmalloc:__vmap_pages_range_noflush()callscond_resched()in atomic context
- [Medium] mm/vmalloc:vmap_page_range()callsflush_cache_vmap()before error rollback --commit
043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>mm/vmalloc: undo partial mappings inside the mapping functions
This commit adds rollback logic using
__vunmap_range_noflush()andkmsan_vunmap_range_noflush()intovmap_page_range(),__vmap_pages_range_noflush(), andvmap_pages_range_noflush()when page table installation or KMSAN metadata setup fails. It also adds range andpage_shiftvalidation in__vmap_pages_range_noflush().mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370
Quoted 13 lines
mm/vmalloc.c +3
-0@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,if (!err) err =kmsan_ioremap_page_range (addr, end, phys_addr, prot, ioremap_max_page_shift); +if (err) +__vunmap_range_noflush (addr, end); +return err; }[Severity: High]
What happens here whenvmap_range_noflush()returns-EINVALfor a range where end - addr is not PAGE_SIZE-aligned?In
vmap_pte_range(), commita739e6b557af09added a check to bail out on unaligned ranges specifically to avoid an infinite loop:mm/vmalloc.c:vmap_pte_range() { ... if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr))) return -EINVAL; }This can be reached on 64KB page kernels via
pci_remap_iospace()->vmap_page_range()with 16KB/32KB/48KB PCI I/O ranges (for example vialoongson_pci_probe()oracpi_pci_root_remap_iospace()).When
vmap_pte_range()returns-EINVAL,vmap_page_range()now calls__vunmap_range_noflush(addr, end)with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as thePCI_IOBASEPMD populated during early boot byreserve_pio_range(), or an earlier PMD mapped in the same call),__vunmap_range_noflush()descends intovunmap_pte_range(), which has no alignment check:mm/vmalloc.c:vunmap_pte_range() { ... do { ... ptent = ptep_get_and_clear(&init_mm, addr, pte); WARN_ON(!pte_none(ptent) && !pte_present(ptent)); } while (pte += (size >> PAGE_SHIFT), addr += size, addr != end); }Quoted 4 lines
Because end - addr is not a multiple of
PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?That sounds reasonable. I will add a separate patch in the next version to fix this problem.
Quoted 41 lines
[Severity: Medium]
Also invmap_page_range(),flush_cache_vmap(addr, end)is called unconditionally before checking err and before rolling back with__vunmap_range_noflush(addr, end):mm/vmalloc.c:vmap_page_range() { err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot), ioremap_max_page_shift); flush_cache_vmap(addr, end); if (!err) err = kmsan_ioremap_page_range(addr, end, phys_addr, prot, ioremap_max_page_shift); if (err) __vunmap_range_noflush(addr, end); return err; }On UML,
flush_cache_vmapis defined asflush_tlb_kernel_range:
arch/um/include/asm/cacheflush.h:
#defineflush_cache_vmapflush_tlb_kernel_rangeWhen
vmap_range_noflush()fails partway through orkmsan_ioremap_page_range()fails,flush_cache_vmap()runs while the partially installed PTEs are stillpte_present(), causingupdate_pte_range()inarch/um/kernel/tlb.cto map them into the host OS viaops->mmap()and clear_PAGE_NEEDSYNC:arch/um/kernel/tlb.c:update_pte_range() { ... if (pte_present(*pte)) { ... ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE, prot, fd, offset); } else ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE); *pte = pte_mkuptodate(*pte); }Quoted 6 lines
When
__vunmap_range_noflush(addr, end)then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Shouldflush_cache_vmap(addr, end)only be called after bothvmap_range_noflush()andkmsan_ioremap_page_range()succeed?That sounds reasonable. I will add this logic in the next version.
Quoted 52 lines
Quoted 31 lines
mm/vmalloc.c +12
-10@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; +unsigned long start = addr; +int err =0 ; + +if (WARN_ON_ONCE (addr >= end)) +return -EINVAL ;if (WARN_ON_ONCE (page_shift <PAGE_SHIFT ))return -EINVAL ;if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) -return vmap_small_pages_range_noflush (addr, end, prot, pages); - -for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { -int err; - - err =vmap_range_noflush (addr, addr + (1UL << page_shift), + page_shift ==PAGE_SHIFT ) { + err =vmap_small_pages_range_noflush (addr, end, prot, pages); + }else { +for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { + err =vmap_range_noflush (addr, addr + (1UL << page_shift),page_to_phys (pages[i]), prot, page_shift); -if (err) -return err; +if (err) +break ; - addr +=1UL << page_shift;mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 714
Quoted 10 lines
mm/vmalloc.c +7
-1+ addr +=1UL << page_shift; + } } -return 0 ; +/* Undo the PTEs installed before the failure. */ +if (err) +__vunmap_range_noflush (start, end); + +return err; }[Severity: High]
Similarly here,__vmap_pages_range_noflush()checks addr >= end, but does not checkPAGE_ALIGNED(addr)orPAGE_ALIGNED(end).If a caller passes a sub-page or unaligned range (for example, 0 < end - addr <
PAGE_SIZE), nr = (end - addr) >>PAGE_SHIFTevaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely invmap_pages_pte_range(). And if the huge-page path fails on an unaligned range,__vunmap_range_noflush(start, end)will hit the same infinite loop invunmap_pte_range().Quoted 4 lines
Should both
vmap_page_range()and__vmap_pages_range_noflush()validatePAGE_ALIGNED(addr)&&PAGE_ALIGNED(end)upfront before attempting to map or unmap?I think We need to check both
PAGE_ALIGNED(addr)andPAGE_ALIGNED(end). As an example: if addr is PAGE-aligned, end is not, and end - addr <PAGE_SIZE, then nr becomes zero. On the huge page path, we simply return 0 with no work performed.At the same time,
vmap_pages_pte_range()needs the same guard as commita739e6b557af("mm: vmalloc:WARN_ONif mapping size is notPAGE_SIZEaligned"), to protect against unaligned ranges.So will do next version.
Quoted 56 lines
Quoted 6 lines
mm/vmalloc.c +1
-1int vmap_pages_range_noflush(unsigned long addr, unsigned long end,@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,if (ret)return ret; -return __vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 738
Quoted 10 lines
mm/vmalloc.c +9
-0+ ret =__vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +/* +* The page tables undo themselves on failure. Tear down the +* metadata that was fully set up before the mapping failed. +*/ +if (ret) +kmsan_vunmap_range_noflush (addr, end); + +return ret; }[Severity: High]
Can calling__vunmap_range_noflush()andkmsan_vunmap_range_noflush()here trigger a sleeping-in-atomic-context bug when__vmalloc()orkvmalloc()is called withGFP_ATOMICorGFP_NOWAIT?
__vmalloc_area_node()supports non-blocking allocations by wrapping__vmap_pages_range()inmemalloc_apply_gfp_scope(gfp_mask):mm/vmalloc.c:__vmalloc_area_node() { ... flags = memalloc_apply_gfp_scope(gfp_mask); do { ret = __vmap_pages_range(addr, addr + size, prot, area->pages, page_shift, nested_gfp); } while ((gfp_mask & __GFP_NOFAIL) && (ret < 0)); memalloc_restore_gfp_scope(flags); }When a
GFP_ATOMICorGFP_NOWAITallocation runs in atomic context (such as under a spinlock,preempt_disable(), orrcu_read_lock()) and page table allocation fails invmap_small_pages_range_noflush()after at least one PMD in [start, end) is populated (or afterkmsan_vmap_pages_range_noflush()populated the KMSAN shadow/origin ranges),__vmap_pages_range_noflush()andkmsan_vunmap_range_noflush()synchronously call__vunmap_range_noflush().
__vunmap_range_noflush()walks the populated PMDs viavunmap_pmd_range(), which unconditionally callscond_resched():mm/vmalloc.c:vunmap_pmd_range() { ... do { next = pmd_addr_end(addr, end); if (pmd_none_or_clear_bad(pmd)) continue; vunmap_pte_range(pmd, addr, next, mask); cond_resched(); } while (pmd++, addr = next, addr != end); }Quoted 7 lines
Commit
9c47753167a6("mm/vmalloc: defer freeing partly initializedvm_struct") deferred error-path cleanup in__vmalloc_area_node()to a workqueue specifically because unmapping on failure can sleep. Doesn't calling__vunmap_range_noflush()synchronously inside__vmap_pages_range_noflush()andvmap_pages_range_noflush()reintroducecond_resched()in atomic context?Quoted 3 lines
This issue also appears real. One possible approach is to plumb the caller's
gfp_maskintovunmap_pmd_range()to conditionally skipcond_resched().
Suren, Ulad, do you have any preferred or cleaner way to handle this?Alternatively, we can follow the approach from commit
9c47753167a6("mm/vmalloc: defer freeing partly initializedvm_struct"):Quoted 8 lines
Quoted 6 lines
defer the cleanup to a workqueue when |gfp_mask| has GFP_ATOMIC or GFP_NOWAIT,and perform cleanup synchronously
if we are already in process context. Anyway, I will look into this further tomorrow.
I will check the AI concern.
Thank you very much for the quick reply and for offering to check the AI concern. Much appreciated!
Here is a summary of what I have found over the last few days.
I went through the history first. Thecond_resched()invunmap_pmd_range()was added by commite47110e90584("mm/vunmap: addcond_resched()invunmap_pmd_range"), modeled afterzap_pte_range(). It fixed a real soft lockup: on a non-preemptible kernel, unmapping a large persistent-memory namespace (hence a large I/O map region) kept a CPU busy for 23 seconds inside the walk:watchdog: BUG: soft lockup - CPU#49 stuck for 23s! [ndctl:50777]Then I looked at how
vfree()handles atomic contexts: it defers the whole free to a workqueue from interrupt context (vfree_atomic()), and9c47753167a6does the same for a partly initializedvm_struct.Unfortunately I do not think deferral works for this rollback case. A deferred rollback returns with the PTEs still installed, and if that range is mapped again before the workqueue runs, the mapping hits the leftovers: a warning and
-EBUSYon the small-page path, aBUG()on the huge path.
Quoted 7 lines
So I kept it simple: plumb the caller's
gfp_maskdown to the unmap walk and callcond_resched()conditionally, like
commit660654f90e7f("mm/vmalloc.c: add a schedule point tovmalloc()") did:if (gfp_mask & __GFP_WAIT) cond_resched();
Sounds good to me. But we have the helper: gfpflags_allow_blocking(gfp_mask)) Another option is probably we can remove cond_resched(); from unmap path?
--
Uladzsislau Rezki
Re: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions
Quoted 304 lines · Uladzislau Rezki
Quoted 303 lines · Hao Ge
Hi Uladzislau and Suren
Quoted 269 lines · Uladzislau Rezki
Quoted 268 lines · Hao Ge
Quoted 258 lines · Hao Ge
Hi Suren and Ulad
Sashiko reported several issues, and they all look real. I have some ideas for fixing the first few;
of course, please correct me if I get anything wrong.I'd like to discuss the last issue with you all.
Quoted 63 lines · On 2026/10/20 13:45, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop invunmap_pte_range()on unaligned error rollback
- [High] mm/vmalloc:__vmap_pages_range_noflush()callscond_resched()in atomic context
- [Medium] mm/vmalloc:vmap_page_range()callsflush_cache_vmap()before error rollback --commit
043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>mm/vmalloc: undo partial mappings inside the mapping functions
This commit adds rollback logic using
__vunmap_range_noflush()andkmsan_vunmap_range_noflush()intovmap_page_range(),__vmap_pages_range_noflush(), andvmap_pages_range_noflush()when page table installation or KMSAN metadata setup fails. It also adds range andpage_shiftvalidation in__vmap_pages_range_noflush().mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370
Quoted 13 lines
mm/vmalloc.c +3
-0@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,if (!err) err =kmsan_ioremap_page_range (addr, end, phys_addr, prot, ioremap_max_page_shift); +if (err) +__vunmap_range_noflush (addr, end); +return err; }[Severity: High]
What happens here whenvmap_range_noflush()returns-EINVALfor a range where end - addr is not PAGE_SIZE-aligned?In
vmap_pte_range(), commita739e6b557af09added a check to bail out on unaligned ranges specifically to avoid an infinite loop:mm/vmalloc.c:vmap_pte_range() { ... if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr))) return -EINVAL; }This can be reached on 64KB page kernels via
pci_remap_iospace()->vmap_page_range()with 16KB/32KB/48KB PCI I/O ranges (for example vialoongson_pci_probe()oracpi_pci_root_remap_iospace()).When
vmap_pte_range()returns-EINVAL,vmap_page_range()now calls__vunmap_range_noflush(addr, end)with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as thePCI_IOBASEPMD populated during early boot byreserve_pio_range(), or an earlier PMD mapped in the same call),__vunmap_range_noflush()descends intovunmap_pte_range(), which has no alignment check:mm/vmalloc.c:vunmap_pte_range() { ... do { ... ptent = ptep_get_and_clear(&init_mm, addr, pte); WARN_ON(!pte_none(ptent) && !pte_present(ptent)); } while (pte += (size >> PAGE_SHIFT), addr += size, addr != end); }Quoted 4 lines
Because end - addr is not a multiple of
PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?That sounds reasonable. I will add a separate patch in the next version to fix this problem.
Quoted 41 lines
[Severity: Medium]
Also invmap_page_range(),flush_cache_vmap(addr, end)is called unconditionally before checking err and before rolling back with__vunmap_range_noflush(addr, end):mm/vmalloc.c:vmap_page_range() { err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot), ioremap_max_page_shift); flush_cache_vmap(addr, end); if (!err) err = kmsan_ioremap_page_range(addr, end, phys_addr, prot, ioremap_max_page_shift); if (err) __vunmap_range_noflush(addr, end); return err; }On UML,
flush_cache_vmapis defined asflush_tlb_kernel_range:
arch/um/include/asm/cacheflush.h:
#defineflush_cache_vmapflush_tlb_kernel_rangeWhen
vmap_range_noflush()fails partway through orkmsan_ioremap_page_range()fails,flush_cache_vmap()runs while the partially installed PTEs are stillpte_present(), causingupdate_pte_range()inarch/um/kernel/tlb.cto map them into the host OS viaops->mmap()and clear_PAGE_NEEDSYNC:arch/um/kernel/tlb.c:update_pte_range() { ... if (pte_present(*pte)) { ... ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE, prot, fd, offset); } else ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE); *pte = pte_mkuptodate(*pte); }Quoted 6 lines
When
__vunmap_range_noflush(addr, end)then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Shouldflush_cache_vmap(addr, end)only be called after bothvmap_range_noflush()andkmsan_ioremap_page_range()succeed?That sounds reasonable. I will add this logic in the next version.
Quoted 52 lines
Quoted 31 lines
mm/vmalloc.c +12
-10@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; +unsigned long start = addr; +int err =0 ; + +if (WARN_ON_ONCE (addr >= end)) +return -EINVAL ;if (WARN_ON_ONCE (page_shift <PAGE_SHIFT ))return -EINVAL ;if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) -return vmap_small_pages_range_noflush (addr, end, prot, pages); - -for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { -int err; - - err =vmap_range_noflush (addr, addr + (1UL << page_shift), + page_shift ==PAGE_SHIFT ) { + err =vmap_small_pages_range_noflush (addr, end, prot, pages); + }else { +for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { + err =vmap_range_noflush (addr, addr + (1UL << page_shift),page_to_phys (pages[i]), prot, page_shift); -if (err) -return err; +if (err) +break ; - addr +=1UL << page_shift;mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 714
Quoted 10 lines
mm/vmalloc.c +7
-1+ addr +=1UL << page_shift; + } } -return 0 ; +/* Undo the PTEs installed before the failure. */ +if (err) +__vunmap_range_noflush (start, end); + +return err; }[Severity: High]
Similarly here,__vmap_pages_range_noflush()checks addr >= end, but does not checkPAGE_ALIGNED(addr)orPAGE_ALIGNED(end).If a caller passes a sub-page or unaligned range (for example, 0 < end - addr <
PAGE_SIZE), nr = (end - addr) >>PAGE_SHIFTevaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely invmap_pages_pte_range(). And if the huge-page path fails on an unaligned range,__vunmap_range_noflush(start, end)will hit the same infinite loop invunmap_pte_range().Quoted 4 lines
Should both
vmap_page_range()and__vmap_pages_range_noflush()validatePAGE_ALIGNED(addr)&&PAGE_ALIGNED(end)upfront before attempting to map or unmap?I think We need to check both
PAGE_ALIGNED(addr)andPAGE_ALIGNED(end). As an example: if addr is PAGE-aligned, end is not, and end - addr <PAGE_SIZE, then nr becomes zero. On the huge page path, we simply return 0 with no work performed.At the same time,
vmap_pages_pte_range()needs the same guard as commita739e6b557af("mm: vmalloc:WARN_ONif mapping size is notPAGE_SIZEaligned"), to protect against unaligned ranges.So will do next version.
Quoted 56 lines
Quoted 6 lines
mm/vmalloc.c +1
-1int vmap_pages_range_noflush(unsigned long addr, unsigned long end,@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,if (ret)return ret; -return __vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 738
Quoted 10 lines
mm/vmalloc.c +9
-0+ ret =__vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +/* +* The page tables undo themselves on failure. Tear down the +* metadata that was fully set up before the mapping failed. +*/ +if (ret) +kmsan_vunmap_range_noflush (addr, end); + +return ret; }[Severity: High]
Can calling__vunmap_range_noflush()andkmsan_vunmap_range_noflush()here trigger a sleeping-in-atomic-context bug when__vmalloc()orkvmalloc()is called withGFP_ATOMICorGFP_NOWAIT?
__vmalloc_area_node()supports non-blocking allocations by wrapping__vmap_pages_range()inmemalloc_apply_gfp_scope(gfp_mask):mm/vmalloc.c:__vmalloc_area_node() { ... flags = memalloc_apply_gfp_scope(gfp_mask); do { ret = __vmap_pages_range(addr, addr + size, prot, area->pages, page_shift, nested_gfp); } while ((gfp_mask & __GFP_NOFAIL) && (ret < 0)); memalloc_restore_gfp_scope(flags); }When a
GFP_ATOMICorGFP_NOWAITallocation runs in atomic context (such as under a spinlock,preempt_disable(), orrcu_read_lock()) and page table allocation fails invmap_small_pages_range_noflush()after at least one PMD in [start, end) is populated (or afterkmsan_vmap_pages_range_noflush()populated the KMSAN shadow/origin ranges),__vmap_pages_range_noflush()andkmsan_vunmap_range_noflush()synchronously call__vunmap_range_noflush().
__vunmap_range_noflush()walks the populated PMDs viavunmap_pmd_range(), which unconditionally callscond_resched():mm/vmalloc.c:vunmap_pmd_range() { ... do { next = pmd_addr_end(addr, end); if (pmd_none_or_clear_bad(pmd)) continue; vunmap_pte_range(pmd, addr, next, mask); cond_resched(); } while (pmd++, addr = next, addr != end); }Quoted 7 lines
Commit
9c47753167a6("mm/vmalloc: defer freeing partly initializedvm_struct") deferred error-path cleanup in__vmalloc_area_node()to a workqueue specifically because unmapping on failure can sleep. Doesn't calling__vunmap_range_noflush()synchronously inside__vmap_pages_range_noflush()andvmap_pages_range_noflush()reintroducecond_resched()in atomic context?Quoted 3 lines
This issue also appears real. One possible approach is to plumb the caller's
gfp_maskintovunmap_pmd_range()to conditionally skipcond_resched().
Suren, Ulad, do you have any preferred or cleaner way to handle this?Alternatively, we can follow the approach from commit
9c47753167a6("mm/vmalloc: defer freeing partly initializedvm_struct"):Quoted 8 lines
Quoted 6 lines
defer the cleanup to a workqueue when |gfp_mask| has GFP_ATOMIC or GFP_NOWAIT,and perform cleanup synchronously
if we are already in process context. Anyway, I will look into this further tomorrow.
I will check the AI concern.
Thank you very much for the quick reply and for offering to check the AI concern. Much appreciated!
Here is a summary of what I have found over the last few days.
I went through the history first. Thecond_resched()invunmap_pmd_range()was added by commite47110e90584("mm/vunmap: addcond_resched()invunmap_pmd_range"), modeled afterzap_pte_range(). It fixed a real soft lockup: on a non-preemptible kernel, unmapping a large persistent-memory namespace (hence a large I/O map region) kept a CPU busy for 23 seconds inside the walk:watchdog: BUG: soft lockup - CPU#49 stuck for 23s! [ndctl:50777]Then I looked at how
vfree()handles atomic contexts: it defers the whole free to a workqueue from interrupt context (vfree_atomic()), and9c47753167a6does the same for a partly initializedvm_struct.Unfortunately I do not think deferral works for this rollback case. A deferred rollback returns with the PTEs still installed, and if that range is mapped again before the workqueue runs, the mapping hits the leftovers: a warning and
-EBUSYon the small-page path, aBUG()on the huge path.
Quoted 8 lines
Quoted 7 lines
So I kept it simple: plumb the caller's
gfp_maskdown to the unmap walk and callcond_resched()conditionally, like
commit660654f90e7f("mm/vmalloc.c: add a schedule point tovmalloc()") did:if (gfp_mask & __GFP_WAIT) cond_resched();Sounds good to me. But we have the helper:
gfpflags_allow_blocking(gfp_mask))
Right.
Quoted 1 line
Another option is probably we can remove
cond_resched(); from unmap path?
I'm wondering if removing cond_resched() from the unmap path would reintroduce the softlockup issue fixed by below commit:
commit e47110e90584a22e9980510b00d0dfad3a83354e
Author: Aneesh Kumar K.V <[email protected]> Date: Thu Aug 20 17:42:05 2020 -0700
mm/vunmap: add cond_resched() in vunmap_pmd_range
Like zap_pte_range add cond_resched so that we can avoid softlockups as
reported below. On non-preemptible kernel with large I/O map region (like
the one we get when using persistent memory with sector mode), an unmap of
the namespace can report below softlockups.
22724.027334] watchdog: BUG: soft lockup - CPU#49 stuck for 23s! [ndctl:50777]
NIP [c0000000000dc224] plpar_hcall+0x38/0x58
LR [c0000000000d8898] pSeries_lpar_hpte_invalidate+0x68/0xb0
Call Trace:
flush_hash_page+0x114/0x200
hpte_need_flush+0x2dc/0x540
vunmap_page_range+0x538/0x6f0
free_unmap_vmap_area+0x30/0x70
remove_vm_area+0xfc/0x140
__vunmap+0x68/0x270
__iounmap.part.0+0x34/0x60
memunmap+0x54/0x70
release_nodes+0x28c/0x300
device_release_driver_internal+0x16c/0x280
unbind_store+0x124/0x170
drv_attr_store+0x44/0x60
sysfs_kf_write+0x64/0x90
kernfs_fop_write+0x1b0/0x290
__vfs_write+0x3c/0x70
vfs_write+0xd8/0x260
ksys_write+0xdc/0x130
system_call+0x5c/0x70Even though we gained lazy MMU mode after that issue was fixed, I suspect I may be missing some technical details. Could you walk me through the reasoning for removing this cond_resched()?
Thanks very much!
Thanks
Best Regards
Hao
Quoted 3 lines
--
Uladzsislau Rezki
Uladzislau Rezki
maintainerRe: [PATCH v2 3/5] mm/vmalloc: undo partial mappings inside the mapping functions
Quoted 357 lines · Hao Ge
Quoted 304 lines · Uladzislau Rezki
Quoted 303 lines · Hao Ge
Hi Uladzislau and Suren
Quoted 269 lines · Uladzislau Rezki
Quoted 268 lines · Hao Ge
Quoted 258 lines · Hao Ge
Hi Suren and Ulad
Sashiko reported several issues, and they all look real. I have some ideas for fixing the first few;
of course, please correct me if I get anything wrong.I'd like to discuss the last issue with you all.
Quoted 63 lines · On 2026/10/20 13:45, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm/vmalloc: infinite loop invunmap_pte_range()on unaligned error rollback
- [High] mm/vmalloc:__vmap_pages_range_noflush()callscond_resched()in atomic context
- [Medium] mm/vmalloc:vmap_page_range()callsflush_cache_vmap()before error rollback --commit
043026437ba63b051df5d14470fe69ce8fa8825c
Author: Hao Ge <[email protected]>mm/vmalloc: undo partial mappings inside the mapping functions
This commit adds rollback logic using
__vunmap_range_noflush()andkmsan_vunmap_range_noflush()intovmap_page_range(),__vmap_pages_range_noflush(), andvmap_pages_range_noflush()when page table installation or KMSAN metadata setup fails. It also adds range andpage_shiftvalidation in__vmap_pages_range_noflush().mm/vmalloc.c int vmap_page_range(unsigned long addr, unsigned long end, · line 370
Quoted 13 lines
mm/vmalloc.c +3
-0@@ -363,6 +363,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,if (!err) err =kmsan_ioremap_page_range (addr, end, phys_addr, prot, ioremap_max_page_shift); +if (err) +__vunmap_range_noflush (addr, end); +return err; }[Severity: High]
What happens here whenvmap_range_noflush()returns-EINVALfor a range where end - addr is not PAGE_SIZE-aligned?In
vmap_pte_range(), commita739e6b557af09added a check to bail out on unaligned ranges specifically to avoid an infinite loop:mm/vmalloc.c:vmap_pte_range() { ... if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr))) return -EINVAL; }This can be reached on 64KB page kernels via
pci_remap_iospace()->vmap_page_range()with 16KB/32KB/48KB PCI I/O ranges (for example vialoongson_pci_probe()oracpi_pci_root_remap_iospace()).When
vmap_pte_range()returns-EINVAL,vmap_page_range()now calls__vunmap_range_noflush(addr, end)with the unaligned range. Whenever a PMD in [addr, end) is already populated with a PTE table (such as thePCI_IOBASEPMD populated during early boot byreserve_pio_range(), or an earlier PMD mapped in the same call),__vunmap_range_noflush()descends intovunmap_pte_range(), which has no alignment check:mm/vmalloc.c:vunmap_pte_range() { ... do { ... ptent = ptep_get_and_clear(&init_mm, addr, pte); WARN_ON(!pte_none(ptent) && !pte_present(ptent)); } while (pte += (size >> PAGE_SHIFT), addr += size, addr != end); }Quoted 4 lines
Because end - addr is not a multiple of
PAGE_SIZE, won't addr step past end and loop infinitely, clearing unrelated kernel PTEs and corrupting memory past the PTE table?That sounds reasonable. I will add a separate patch in the next version to fix this problem.
Quoted 41 lines
[Severity: Medium]
Also invmap_page_range(),flush_cache_vmap(addr, end)is called unconditionally before checking err and before rolling back with__vunmap_range_noflush(addr, end):mm/vmalloc.c:vmap_page_range() { err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot), ioremap_max_page_shift); flush_cache_vmap(addr, end); if (!err) err = kmsan_ioremap_page_range(addr, end, phys_addr, prot, ioremap_max_page_shift); if (err) __vunmap_range_noflush(addr, end); return err; }On UML,
flush_cache_vmapis defined asflush_tlb_kernel_range:
arch/um/include/asm/cacheflush.h:
#defineflush_cache_vmapflush_tlb_kernel_rangeWhen
vmap_range_noflush()fails partway through orkmsan_ioremap_page_range()fails,flush_cache_vmap()runs while the partially installed PTEs are stillpte_present(), causingupdate_pte_range()inarch/um/kernel/tlb.cto map them into the host OS viaops->mmap()and clear_PAGE_NEEDSYNC:arch/um/kernel/tlb.c:update_pte_range() { ... if (pte_present(*pte)) { ... ret = ops->mmap(ops->mm_idp, addr, PAGE_SIZE, prot, fd, offset); } else ret = ops->unmap(ops->mm_idp, addr, PAGE_SIZE); *pte = pte_mkuptodate(*pte); }Quoted 6 lines
When
__vunmap_range_noflush(addr, end)then clears the software PTEs without a subsequent TLB flush or unmap notification, won't those host OS mmap mappings be left behind? Shouldflush_cache_vmap(addr, end)only be called after bothvmap_range_noflush()andkmsan_ioremap_page_range()succeed?That sounds reasonable. I will add this logic in the next version.
Quoted 52 lines
Quoted 31 lines
mm/vmalloc.c +12
-10@@ -683,27 +686,35 @@ int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr = (end - addr) >> PAGE_SHIFT; +unsigned long start = addr; +int err =0 ; + +if (WARN_ON_ONCE (addr >= end)) +return -EINVAL ;if (WARN_ON_ONCE (page_shift <PAGE_SHIFT ))return -EINVAL ;if (!IS_ENABLED (CONFIG_HAVE_ARCH_HUGE_VMALLOC ) || - page_shift ==PAGE_SHIFT ) -return vmap_small_pages_range_noflush (addr, end, prot, pages); - -for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { -int err; - - err =vmap_range_noflush (addr, addr + (1UL << page_shift), + page_shift ==PAGE_SHIFT ) { + err =vmap_small_pages_range_noflush (addr, end, prot, pages); + }else { +for (i =0 ; i < nr; i +=1U << (page_shift -PAGE_SHIFT )) { + err =vmap_range_noflush (addr, addr + (1UL << page_shift),page_to_phys (pages[i]), prot, page_shift); -if (err) -return err; +if (err) +break ; - addr +=1UL << page_shift;mm/vmalloc.c int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 714
Quoted 10 lines
mm/vmalloc.c +7
-1+ addr +=1UL << page_shift; + } } -return 0 ; +/* Undo the PTEs installed before the failure. */ +if (err) +__vunmap_range_noflush (start, end); + +return err; }[Severity: High]
Similarly here,__vmap_pages_range_noflush()checks addr >= end, but does not checkPAGE_ALIGNED(addr)orPAGE_ALIGNED(end).If a caller passes a sub-page or unaligned range (for example, 0 < end - addr <
PAGE_SIZE), nr = (end - addr) >>PAGE_SHIFTevaluates to 0, so the huge-page path still maps nothing and returns 0, while the small-page path loops infinitely invmap_pages_pte_range(). And if the huge-page path fails on an unaligned range,__vunmap_range_noflush(start, end)will hit the same infinite loop invunmap_pte_range().Quoted 4 lines
Should both
vmap_page_range()and__vmap_pages_range_noflush()validatePAGE_ALIGNED(addr)&&PAGE_ALIGNED(end)upfront before attempting to map or unmap?I think We need to check both
PAGE_ALIGNED(addr)andPAGE_ALIGNED(end). As an example: if addr is PAGE-aligned, end is not, and end - addr <PAGE_SIZE, then nr becomes zero. On the huge page path, we simply return 0 with no work performed.At the same time,
vmap_pages_pte_range()needs the same guard as commita739e6b557af("mm: vmalloc:WARN_ONif mapping size is notPAGE_SIZEaligned"), to protect against unaligned ranges.So will do next version.
Quoted 56 lines
Quoted 6 lines
mm/vmalloc.c +1
-1int vmap_pages_range_noflush(unsigned long addr, unsigned long end,@@ -715,7 +726,16 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,if (ret)return ret; -return __vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +mm/vmalloc.c int vmap_pages_range_noflush(unsigned long addr, unsigned long end, · line 738
Quoted 10 lines
mm/vmalloc.c +9
-0+ ret =__vmap_pages_range_noflush (addr, end, prot, pages, page_shift); +/* +* The page tables undo themselves on failure. Tear down the +* metadata that was fully set up before the mapping failed. +*/ +if (ret) +kmsan_vunmap_range_noflush (addr, end); + +return ret; }[Severity: High]
Can calling__vunmap_range_noflush()andkmsan_vunmap_range_noflush()here trigger a sleeping-in-atomic-context bug when__vmalloc()orkvmalloc()is called withGFP_ATOMICorGFP_NOWAIT?
__vmalloc_area_node()supports non-blocking allocations by wrapping__vmap_pages_range()inmemalloc_apply_gfp_scope(gfp_mask):mm/vmalloc.c:__vmalloc_area_node() { ... flags = memalloc_apply_gfp_scope(gfp_mask); do { ret = __vmap_pages_range(addr, addr + size, prot, area->pages, page_shift, nested_gfp); } while ((gfp_mask & __GFP_NOFAIL) && (ret < 0)); memalloc_restore_gfp_scope(flags); }When a
GFP_ATOMICorGFP_NOWAITallocation runs in atomic context (such as under a spinlock,preempt_disable(), orrcu_read_lock()) and page table allocation fails invmap_small_pages_range_noflush()after at least one PMD in [start, end) is populated (or afterkmsan_vmap_pages_range_noflush()populated the KMSAN shadow/origin ranges),__vmap_pages_range_noflush()andkmsan_vunmap_range_noflush()synchronously call__vunmap_range_noflush().
__vunmap_range_noflush()walks the populated PMDs viavunmap_pmd_range(), which unconditionally callscond_resched():mm/vmalloc.c:vunmap_pmd_range() { ... do { next = pmd_addr_end(addr, end); if (pmd_none_or_clear_bad(pmd)) continue; vunmap_pte_range(pmd, addr, next, mask); cond_resched(); } while (pmd++, addr = next, addr != end); }Quoted 7 lines
Commit
9c47753167a6("mm/vmalloc: defer freeing partly initializedvm_struct") deferred error-path cleanup in__vmalloc_area_node()to a workqueue specifically because unmapping on failure can sleep. Doesn't calling__vunmap_range_noflush()synchronously inside__vmap_pages_range_noflush()andvmap_pages_range_noflush()reintroducecond_resched()in atomic context?Quoted 3 lines
This issue also appears real. One possible approach is to plumb the caller's
gfp_maskintovunmap_pmd_range()to conditionally skipcond_resched().
Suren, Ulad, do you have any preferred or cleaner way to handle this?Alternatively, we can follow the approach from commit
9c47753167a6("mm/vmalloc: defer freeing partly initializedvm_struct"):Quoted 8 lines
Quoted 6 lines
defer the cleanup to a workqueue when |gfp_mask| has GFP_ATOMIC or GFP_NOWAIT,and perform cleanup synchronously
if we are already in process context. Anyway, I will look into this further tomorrow.
I will check the AI concern.
Thank you very much for the quick reply and for offering to check the AI concern. Much appreciated!
Here is a summary of what I have found over the last few days.
I went through the history first. Thecond_resched()invunmap_pmd_range()was added by commite47110e90584("mm/vunmap: addcond_resched()invunmap_pmd_range"), modeled afterzap_pte_range(). It fixed a real soft lockup: on a non-preemptible kernel, unmapping a large persistent-memory namespace (hence a large I/O map region) kept a CPU busy for 23 seconds inside the walk:watchdog: BUG: soft lockup - CPU#49 stuck for 23s! [ndctl:50777]Then I looked at how
vfree()handles atomic contexts: it defers the whole free to a workqueue from interrupt context (vfree_atomic()), and9c47753167a6does the same for a partly initializedvm_struct.Unfortunately I do not think deferral works for this rollback case. A deferred rollback returns with the PTEs still installed, and if that range is mapped again before the workqueue runs, the mapping hits the leftovers: a warning and
-EBUSYon the small-page path, aBUG()on the huge path.Quoted 8 lines
Quoted 7 lines
So I kept it simple: plumb the caller's
gfp_maskdown to the unmap walk and callcond_resched()conditionally, like
commit660654f90e7f("mm/vmalloc.c: add a schedule point tovmalloc()") did:if (gfp_mask & __GFP_WAIT) cond_resched();Sounds good to me. But we have the helper:
gfpflags_allow_blocking(gfp_mask))Right.
Quoted 1 line
Another option is probably we can remove
cond_resched(); from unmap path?I'm wondering if removing
cond_resched()from the unmap path would reintroduce the softlockup issue fixed by below commit:commit
e47110e90584a22e9980510b00d0dfad3a83354eAuthor: Aneesh Kumar K.V <[email protected]> Date: Thu Aug 20 17:42:05 2020 -0700mm/vunmap: add cond_resched() in vunmap_pmd_range Like zap_pte_range add cond_resched so that we can avoid softlockups as reported below. On non-preemptible kernel with large I/O map region (like the one we get when using persistent memory with sector mode), an unmap of the namespace can report below softlockups. 22724.027334] watchdog: BUG: soft lockup - CPU#49 stuck for 23s! [ndctl:50777] NIP [c0000000000dc224] plpar_hcall+0x38/0x58 LR [c0000000000d8898] pSeries_lpar_hpte_invalidate+0x68/0xb0 Call Trace: flush_hash_page+0x114/0x200 hpte_need_flush+0x2dc/0x540 vunmap_page_range+0x538/0x6f0 free_unmap_vmap_area+0x30/0x70 remove_vm_area+0xfc/0x140 __vunmap+0x68/0x270 __iounmap.part.0+0x34/0x60 memunmap+0x54/0x70 release_nodes+0x28c/0x300 device_release_driver_internal+0x16c/0x280 unbind_store+0x124/0x170 drv_attr_store+0x44/0x60 sysfs_kf_write+0x64/0x90 kernfs_fop_write+0x1b0/0x290 __vfs_write+0x3c/0x70 vfs_write+0xd8/0x260 ksys_write+0xdc/0x130 system_call+0x5c/0x70
Quoted 4 lines
Even though we gained lazy MMU mode after that issue was fixed, I suspect I may be missing some technical details. Could you walk me through the reasoning for removing this
cond_resched()?
This is a good point. And you showed the good example why it is a bad idea to remove it. I was thinking about this as preventing CPU hogging by unmap path and you proved it.
Checking gfp sounds like is a correct away to address AI concern which is correct.
--
Uladzislau Rezki