mm/virtio: skip redundant zeroing of host-zeroed pages
Subsystems (19)
Michael S. Tsirkin
maintainerv8 cover mm/virtio: skip redundant zeroing of host-zeroed pages
When a guest reports free pages to the hypervisor via virtio-balloon's free page reporting, the host typically zeros those pages when reclaiming their backing memory (e.g., via MADV_DONTNEED on anonymous mappings). When the guest later reallocates those pages, the kernel zeros them again, redundantly.
Further, on architectures with aliasing caches, upstream with init_on_alloc double-zeros user pages: once via kernel_init_pages() in post_alloc_hook, and again via clear_user_highpage() at the callsite (because user_alloc_needs_zeroing() returns true). This series eliminates that double-zeroing by moving the zeroing into the post_alloc_hook + propagating the "host
already zeroed this page" information through the buddy allocator.
For page reporting, VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED (bit 6) is used. For the inflate/deflate path, VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE (bit 7) is used.
Virtio spec: https://lore.kernel.org/all/[email protected]
Based on v7.1-rc2. When applying on mm-unstable, two conflicts are expected:
- kernel_init_pages() was renamed to clear_highpages_kasan_tagged() in mm-unstable. Use clear_highpages_kasan_tagged() in the post_alloc_hook else branch.
- FPI_PREPARED uses BIT(3) in mm-unstable. Bump FPI_ZEROED to BIT(4).
Build-tested on mm-unstable at e9dd96806dbc:
https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git zero-mm-unstable
Patches 1-5: fixes/cleanups, dependencies of the zeroing patches. Patches 6-9: thread user_addr through page allocator, contig API,
and gigantic hugetlb allocation.
Patches 10-23: mm rework + cleanups + init_on_alloc double-zeroing
fix + aliasing guard.
Patches 24-32: page reporting zeroing (DEVICE_INIT_REPORTED),
disable indirect descriptors.
Patches 33-37: inflate/deflate zeroing (DEVICE_INIT_ON_INFLATE),
FPI_ZEROED optimization.
-------
Performance with THP enabled on a 2GB VM, 1 vCPU, allocating 256MB of anonymous pages:
metric baseline optimized delta task-clock 232 +- 20 ms 51 +- 26 ms -78% cache-misses 1.20M +- 248K 288K +- 102K -76% instructions 16.3M +- 1.2M 13.8M +- 1.0M -15%
With hugetlb surplus pages:
metric baseline optimized delta task-clock 219 +- 23 ms 65 +- 34 ms -70% cache-misses 1.17M +- 391K 263K +- 36K -78% instructions 17.9M +- 1.2M 15.1M +- 724K -16%
Two flags track known-zero pages:
PG_zeroed (aliased to PG_private) marks buddy allocator pages that are known to contain all zeros, either because the host zeroed them during page reporting, or because they were freed via the balloon deflate path. It lives on free-list pages and is consumed by post_alloc_hook() on allocation.
HPG_zeroed (stored in hugetlb folio->private bits) serves the same purpose for hugetlb pool pages, which are kept in a pool and may be zeroed long after buddy allocation, so PG_zeroed (consumed at allocation time) cannot track their state.
PG_zeroed lifecycle:
Sets PG_zeroed:
- page_reporting_drain: on reported pages when host zeroes them
- __free_pages_ok / __free_frozen_pages: when FPI_ZEROED is set (balloon deflate path)
- buddy merge: on merged page if both buddies were zeroed
- expand(): propagate to split-off buddy sub-pages
Clears PG_zeroed:
- __free_pages_prepare: clears all PAGE_FLAGS_CHECK_AT_PREP flags (PG_zeroed included), preventing PG_private aliasing leaks
- rmqueue_buddy / __rmqueue_pcplist: read-then-clear, passes
zeroed hint to prep_new_page -> post_alloc_hook
- __isolate_free_page: clear (compaction/page_reporting isolation)
- compaction, alloc_contig, split_free_frozen: clear before use
- buddy merge: clear both pages before merge, then conditionally re-set on merged head if both were zeroed
HPG_zeroed lifecycle (hugetlb pool pages, stored in folio->private):
Sets HPG_zeroed:
- alloc_surplus_hugetlb_folio: after buddy allocation with __GFP_ZERO, mark pool page as known-zero
Clears HPG_zeroed:
- free_huge_folio: page was mapped to userspace, no longer known-zero when it returns to the pool
- alloc_hugetlb_folio: cleared unconditionally on output
- alloc_hugetlb_folio_reserve: cleared after checking
- The optimization is most effective with THP, where entire 2MB pages are allocated directly from reported order-9+ buddy pages. Without THP, only ~21% of order-0 allocations come from reported pages due to low-order fragmentation.
- Persistent hugetlb pool pages are not covered: when freed by userspace they return to the hugetlb free pool, not the buddy allocator, so they are never reported to the host. Surplus hugetlb pages are allocated from buddy and do benefit.
- PG_zeroed is aliased to PG_private. __free_pages_prepare() clears it (preventing filesystem PG_private from leaking as false PG_zeroed). FPI_ZEROED re-sets it after prepare for balloon deflate pages. Is aliasing PG_private acceptable, or should a different bit be used?
- With __GFP_ZERO, the folio is zeroed before mem_cgroup_charge(). If the charge fails (cgroup at limit), the zeroing work is wasted and the folio is freed and retried at a smaller order. Previously, zeroing was done after a successful charge. This is inherent to the __GFP_ZERO approach. Is this acceptable?
- On architectures with aliasing caches, upstream with init_on_alloc double-zeros user pages: once via kernel_init_pages() in post_alloc_hook, and again via clear_user_highpage() at the callsite (because user_alloc_needs_zeroing() returns true). Our patches eliminate this by zeroing once via folio_zero_user() in post_alloc_hook. Not a critical fix (people who set init_on_alloc know they are paying performance) but a nice cleanup anyway.
Test program:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#ifndef MADV_POPULATE_WRITE
#define MADV_POPULATE_WRITE 23
#endif
#ifndef MAP_HUGETLB
#define MAP_HUGETLB 0x40000
#endif
int main(int argc, char **argv)
{
unsigned long size;
int flags = MAP_PRIVATE | MAP_ANONYMOUS;
void *p;
int r; if (argc < 2) {
fprintf(stderr, "usage: %s <size_mb> [huge]\n", argv[0]);
return 1;
}
size = atol(argv[1]) * 1024UL * 1024;
if (argc >= 3 && strcmp(argv[2], "huge") == 0)
flags |= MAP_HUGETLB;
p = mmap(NULL, size, PROT_READ | PROT_WRITE, flags, -1, 0);
if (p == MAP_FAILED) {
perror("mmap");
return 1;
}
r = madvise(p, size, MADV_POPULATE_WRITE);
if (r) {
perror("madvise");
return 1;
}
munmap(p, size);
return 0;}
Test script (bench.sh):
#!/bin/bash
# Usage: bench.sh <size_mb> <iterations> [huge]
# Feature negotiation (DEVICE_INIT_REPORTED/ON_INFLATE) is
# handled by QEMU command line flags, not module parameters.
SZ=${1:-256}; ITER=${2:-10}; HUGE=${3:-}
FLUSH=/sys/module/page_reporting/parameters/flush
CSV=/tmp/perf.csv
rmmod virtio_balloon 2>/dev/null
insmod /mnt/share/virtio_balloon.ko
echo 512 > $FLUSH
[ "$HUGE" = "huge" ] && echo $((SZ/2)) > /proc/sys/vm/nr_overcommit_hugepages
rm -f $CSV
echo "=== sz=${SZ}MB iter=$ITER $HUGE ==="
for i in $(seq 1 $ITER); do
echo 3 > /proc/sys/vm/drop_caches
echo 512 > $FLUSH
perf stat -e task-clock,instructions,cache-misses \
-x, -o $CSV --append -- /mnt/share/alloc_once $SZ $HUGE
done
[ "$HUGE" = "huge" ] && echo 0 > /proc/sys/vm/nr_overcommit_hugepages
rmmod virtio_balloon
awk -F, '/^#/||/^$/{next}{v=$1+0;e=$3;gsub(/ /,"",e);s[e]+=v;ss[e]+=v*v;n[e]++}
END{for(e in s){a=s[e]/n[e];d=sqrt(ss[e]/n[e]-a*a);printf " %-16s %10.0f +- %8.0f (n=%d)\n",e,a,d,n[e]}}' $CSVCompile and run:
gcc -static -O2 -o alloc_once alloc_once.c
bash bench.sh 256 10 # regular pages
bash bench.sh 256 10 huge # hugetlb surplus
Changes since v7 (address Sashiko AI review findings):
- Fix dcache flush on VIPT aliasing architectures: add user_alloc_needs_zeroing() guard in post_alloc_hook to force folio_zero_user for user pages when cache aliasing requires it. Host-zeroed pages excluded (!zeroed). Optimization preserved.
- Fix folio_zero_user stub: replace macro with non-inline function in mm/memory.c to avoid double-evaluation and missing include.
- Fix C89 declaration-after-statement in free_huge_folio.
- Fix CMA __GFP_ZERO: pass through to cma_alloc_frozen_compound so HPG_zeroed accurately reflects whether page was zeroed.
- Fix big-endian bitmap: use test_bit_le() for inflate_bitmap.
- Fix migratepage: clear PageZeroed on old page before deflation.
- Fix page_reporting flush: overflow-safe loop, add -EINTR on signal, add code comment explaining double flush_delayed_work.
- Add atomic ClearPageZeroed (CLEARPAGEFLAG) for balloon migration path where zone->lock is not held.
- Add VM_WARN_ON_ONCE for order>0 without __GFP_COMP in post_alloc_hook (folio_zero_user requires compound metadata).
- Add _noprof pattern for vma_alloc_zeroed_movable_folio to preserve memory allocation profiling attribution.
- Add PageReported propagation in split_large_buddy (was missing from patch 2).
- Add FPI_ZEROED guard: skip PageZeroed when page_poisoning enabled and init_on_free disabled (poison overwrites zeroes).
- Add DMA alignment comment for inflate_bitmap (ACCESS_PLATFORM cleared, so not needed now).
- Restore tell_host comment explaining vq buffer assumption.
- Various code comments documenting design decisions.
- Drop __GFP_ZERO from gather_surplus_pages: avoid shifting zeroing from fault time to reservation time (mmap/fallocate). Pool pages are zeroed at fault time via alloc_hugetlb_folio. Fresh surplus allocations at fault time still benefit from __GFP_ZERO + HPG_zeroed.
- New patch: add alloc_contig_frozen_pages_user API with user_addr for cache-friendly zeroing in the contiguous allocation path.
- New patch: thread user_addr through gigantic hugetlb allocation via alloc_contig_frozen_pages_user.
- New patch: replace user_alloc_needs_zeroing() with aliasing-only checks (cpu_dcache_is_aliasing || cpu_icache_is_aliasing) in the post_alloc_hook guard. Avoids redundant re-zero on non-aliasing.
- New patch: serialize TestSetPageHWPoison with zone->lock in memory_failure to fix pre-existing race with non-atomic buddy flag operations (e.g. page->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP).
- New patch: disable VIRTIO_RING_F_INDIRECT_DESC in balloon to prevent GFP_KERNEL allocation under balloon_lock (OOM deadlock).
- New patch: skip kernel_init_pages for FPI_ZEROED when page poisoning is not enabled (page already zero, skip redundant work).
Also since v7 (address review by Gregory Price):
- Drop from_pool bool in alloc_hugetlb_folio: use folio_test_hugetlb_zeroed directly. HPG_zeroed is set by alloc_surplus_hugetlb_folio for fresh allocations, so the check handles both pool and fresh pages.
- Drop bool *zeroed output parameter from alloc_hugetlb_folio:
sink zeroing inside the function. When __GFP_ZERO is set and !folio_test_hugetlb_zeroed, call folio_zero_user internally.
- Rename addr to user_addr in alloc_hugetlb_folio, align internally with huge_page_mask.
- Add Reviewed-by: Gregory Price tags on reviewed patches.
New patches since v7:
- mm: memory-failure: serialize TestSetPageHWPoison with zone->lock
- mm: add alloc_contig_frozen_pages_user for cache-friendly zeroing
- mm: hugetlb: thread user_addr through gigantic page allocation
- mm: page_alloc: use aliasing checks instead of user_alloc_needs_zeroing
- virtio_balloon: disable indirect descriptors
- mm: page_alloc: skip kernel_init_pages for FPI_ZEROED when safe
Changes since v6 (address review by Gregory Price):
- Rework hugetlb: use gfp_t parameter instead of bool zero / bool *zeroed. Sink zeroing inside alloc_hugetlb_folio(). Pass raw fault address (user_addr) for cache-friendly zeroing on both pool-page and fresh allocation
paths. (Suggested by Gregory Price)
- Reorder compaction_alloc_noprof() to call prep_compound_page before post_alloc_hook for consistency.
(Suggested by Gregory Price)
- Reorder: interleave fix first, PageReported propagation and capacity fix moved to front as dependencies.
- Add USER_ADDR_NONE comments in mmap.c and internal.h explaining why -1 is never a valid userspace address.
- Fix err uninitialized warning in virtballoon_free_page_report().
- Lots of commit log tweaks.
Also in v7:
- Fix hugetlb pool page zeroing to use vmf->real_address (the actual faulting subpage) instead of vmf->address (hugepage-aligned), preserving cache-friendly zeroing locality that upstream had at the callsite.
- Remove dead/broken alloc_hugetlb_folio !CONFIG_HUGETLB_PAGE stub (returned NULL but callers check IS_ERR).
Changes since v5:
- Rebased onto v7.1-rc2.
- Split alloc_anon_folio and alloc_swap_folio raw fault address changes into separate patches.
- In virtio, move PAGE_POISON check for DEVICE_INIT_REPORTED from probe() to validate(), clearing the feature instead of just gating host_zeroes_pages. Same for confidential computing check.
- Fix bisectability: FPI_ZEROED definition and usage now in the same patch.
- Lots of commit log tweaks.
- Reorder: REPORTED before ON_INFLATE.
- Kerneldoc fixes.
Changes since v4:
With virtio spec posted, update to latest spec:
- Add VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED (bit 6) for reporting.
- Add VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE (bit 7) for inflate.
- Per-page virtqueue submission, per-page used_len feedback.
- Balloon migration preserves PageZeroed hint.
- Page_reporting capacity bugfix for small virtqueues.
- PG_zeroed propagation in split_large_buddy.
- Disable both features for confidential computing guests.
- Gate host_zeroes_pages on PAGE_POISON/poison_val: when PAGE_POISON is negotiated with non-zero poison_val, device fills with poison not zeros, so host_zeroes_pages must be false.
- Disable ON_INFLATE when PAGE_POISON with non-zero poison_val.
- Bound inflate bitmap reads by used_len from device.
- Move ON_INFLATE poison_val check to validate() for proper feature negotiation.
- Fix NUMA interleave index for unaligned VMA start (new patch 1).
- Drop vma_alloc_folio_user_addr: with the ilx fix, callers can pass raw fault address to vma_alloc_folio directly.
- Tested with DEBUG_VM, INIT_ON_ALLOC/FREE enabled.
Changes since v3 (address review by Gregory Price and David Hildenbrand):
- Keep user_addr threading internal: public APIs (__alloc_pages, __folio_alloc, folio_alloc_mpol) are unchanged. Only internal functions (__alloc_frozen_pages_noprof, __alloc_pages_mpol) carry user_addr. This eliminates all API churn for external callers.
- Add vma_alloc_folio_user_addr() (2/22) to separate NUMA policy address from the zeroing hint address. Fixes NUMA interleave index corruption when passing unaligned fault address for higher-order allocations.
- Add per-page zeroed_bitmap to page_reporting_dev_info (17/22). The driver's report() callback manages the bitmap. Drain checks it gated by the host_zeroes_pages static key. This matches the proposed virtio balloon extension at
https://lore.kernel.org/all/[email protected]/
- Clear PG_zeroed in __isolate_free_page() to prevent the aliased PG_private flag from leaking to compaction/alloc_contig paths.
- Do not exclude PG_zeroed from PAGE_FLAGS_CHECK_AT_PREP macro. Instead, __free_pages_prepare() clears it (preventing filesystem PG_private leaking as false PG_zeroed), and FPI_ZEROED sets it after prepare. Only buddy merge assertion is relaxed.
- Initialize alloc_context.user_addr in alloc_pages_bulk_noprof.
- Deflate and hugetlb changes are much smaller now. Still, the patchset can be merged gradually, if desired.
Changes since v2 (address review by Gregory Price and David Hildenbrand):
- v2 used pghint_t / vma_alloc_folio_hints API. v3 switches to threading user_addr through the page allocator and using __GFP_ZERO, so post_alloc_hook() can use folio_zero_user() for cache-friendly zeroing when the user fault address is known.
- Use FPI_ZEROED to set PG_zeroed after __free_pages_prepare() instead of runtime masking in __free_one_page (further refined in v4).
- Drop redundant page_poisoning_enabled() check from mm core free path, already guarded at feature negotiation time in virtio_balloon_validate. The balloon driver keeps its own page_poisoning_enabled_static() check as defense in depth.
- Split free_frozen_pages_zeroed and put_page_zeroed into separate patches. David Hildenbrand indicated he intends to rework balloon pages to be frozen (no refcount), at which point put_page_zeroed (21/22) can be dropped and the balloon can call free_frozen_pages_zeroed directly.
- Use HPG_zeroed flag (in hugetlb folio->private) for hugetlb pool pages instead of PG_zeroed, since pool pages are zeroed long after buddy allocation and PG_zeroed is consumed at allocation time.
- syzbot CI found a PF_NO_COMPOUND BUG in the v2 pghint_t approach where __ClearPageZeroed was called on compound hugetlb pages in free_huge_folio. The v3 HPG_zeroed approach avoids this.
- Remove redundant arch vma_alloc_zeroed_movable_folio overrides on x86, s390, m68k, and alpha (12/22). Suggested by David Hildenbrand.
- Updated benchmarking script to compute per-run avg +- stddev via awk on CSV output.
Changes v1->v2:
- Replaced __GFP_PREZEROED with PG_zeroed page flag (aliased PG_private)
- Added pghint_t type and vma_alloc_folio_hints() API
- Track PG_zeroed across buddy merges and splits
- Added post_alloc_hook integration (single consume/clear point)
- Added hugetlb support (pool pages + memfd)
- Added page_reporting flush parameter for deterministic testing
- Added free_frozen_pages_hint/put_page_hint for balloon deflate path
- Added try_to_claim_block PG_zeroed preservation
- Updated perf numbers with per-iteration flush methodology
Written with assistance from Claude (claude-opus-4-6). Reviewed by cursor-agent (GPT-5.4-xhigh).
Everything manually read, patchset split and commit logs edited manually.
Michael S. Tsirkin (37):
mm: mempolicy: fix interleave index for unaligned VMA start
mm: memory-failure: serialize TestSetPageHWPoison with zone->lock
mm: page_alloc: propagate PageReported flag across buddy splits
mm: page_reporting: allow driver to set batch capacity
mm: hugetlb: remove dead alloc_hugetlb_folio stub
mm: move vma_alloc_folio_noprof to page_alloc.c
mm: thread user_addr through page allocator for cache-friendly zeroing
mm: add alloc_contig_frozen_pages_user for cache-friendly zeroing
mm: hugetlb: thread user_addr through gigantic page allocation
mm: add folio_zero_user stub for configs without THP/HUGETLBFS
mm: page_alloc: move prep_compound_page before post_alloc_hook
mm: use folio_zero_user for user pages in post_alloc_hook
mm: use __GFP_ZERO in vma_alloc_zeroed_movable_folio
mm: remove arch vma_alloc_zeroed_movable_folio overrides
mm: alloc_anon_folio: pass raw fault address to vma_alloc_folio
mm: alloc_swap_folio: pass raw fault address to vma_alloc_folio
mm: use __GFP_ZERO in alloc_anon_folio
mm: vma_alloc_anon_folio_pmd: pass raw fault address to
vma_alloc_folio
mm: use __GFP_ZERO in vma_alloc_anon_folio_pmd
mm: hugetlb: add gfp parameter and skip zeroing for zeroed pages
mm: memfd: skip zeroing for zeroed hugetlb pool pages
mm: page_reporting: skip redundant zeroing of host-zeroed reported
pages
mm: page_alloc: use aliasing checks instead of
user_alloc_needs_zeroing
mm: page_reporting: add per-page zeroed bitmap for host feedback
mm: page_alloc: clear PG_zeroed on buddy merge if not both zero
mm: page_alloc: preserve PG_zeroed in page_del_and_expand
virtio_balloon: submit reported pages as individual buffers
virtio_balloon: disable indirect descriptors
mm: page_reporting: add flush parameter with page budget
mm: page_alloc: propagate PG_zeroed in split_large_buddy
virtio_balloon: skip zeroing for host-zeroed reported pages
virtio_balloon: disable reporting zeroed optimization for confidential
guests
mm: add free_frozen_pages_zeroed
mm: page_alloc: skip kernel_init_pages for FPI_ZEROED when safe
mm: add put_page_zeroed and folio_put_zeroed
virtio_balloon: implement VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE
mm: balloon: use put_page_zeroed for zeroed balloon pages
arch/alpha/include/asm/page.h | 3 - arch/m68k/include/asm/page_no.h | 3 - arch/s390/include/asm/page.h | 3 - arch/x86/include/asm/page.h | 3 - drivers/virtio/virtio_balloon.c | 177 +++++++++++++++--- fs/hugetlbfs/inode.c | 3 +- include/linux/cma.h | 3 +- include/linux/gfp.h | 18 +- include/linux/highmem.h | 15 +- include/linux/hugetlb.h | 18 +- include/linux/mm.h | 15 ++ include/linux/page-flags.h | 10 + include/linux/page_reporting.h | 13 ++ include/uapi/linux/virtio_balloon.h | 2 + mm/balloon.c | 10 +- mm/cma.c | 6 +- mm/compaction.c | 9 +- mm/huge_memory.c | 12 +- mm/hugetlb.c | 102 ++++++---- mm/hugetlb_cma.c | 4 +- mm/internal.h | 21 ++- mm/memfd.c | 14 +- mm/memory-failure.c | 6 + mm/memory.c | 26 ++- mm/mempolicy.c | 73 +++----- mm/mmap.c | 6 + mm/page_alloc.c | 281 ++++++++++++++++++++++++---- mm/page_reporting.c | 102 ++++++++-- mm/page_reporting.h | 12 ++ mm/slub.c | 4 +- mm/swap.c | 20 +- 31 files changed, 747 insertions(+), 247 deletions(-)
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 1/37 mm: mempolicy: fix interleave index for unaligned VMA start
The NUMA interleave index formula (addr - vm_start) >> shift gives wrong results when vm_start is not aligned to the folio
size: the subtraction before the shift allows low bits to affect the result via borrows.
Use (addr >> shift) - (vm_start >> shift) instead, which independently aligns both values before computing the difference.
No functional change for current callers: the fix only affects NUMA interleave and weighted-interleave policies. Current large-order callers either pre-align the address (vma_alloc_anon_folio_pmd) or do not use NUMA interleave (drm_pagemap). All other callers use order 0 where the old and new formulas are equivalent. However subsequent patches in this series add large-order callers that pass unaligned fault addresses, making this fix necessary.
--- mm/mempolicy.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/mempolicy.c +2 -1
@@ -2049,7 +2049,8 @@ struct mempolicy *get_vma_policy(struct vm_area_struct *vma,if (pol->mode ==MPOL_INTERLEAVE || pol->mode ==MPOL_WEIGHTED_INTERLEAVE ) { *ilx += vma->vm_pgoff >> order; - *ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order); + *ilx += (addr >> (PAGE_SHIFT + order)) - + (vma->vm_start >> (PAGE_SHIFT + order)); }return pol; }
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 2/37 mm: memory-failure: serialize TestSetPageHWPoison with zone->lock
In reply to Michael S. TsirkinTestSetPageHWPoison() is called without zone->lock, so its atomic update to page->flags can race with non-atomic flag operations that run under zone->lock in the buddy allocator.
In particular, __free_pages_prepare() does:
page->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP;
This non-atomic read-modify-write, while correctly excluding __PG_HWPOISON from the mask, can still lose a concurrent TestSetPageHWPoison if the read happens before the poison bit is set and the write happens after. Follow-up patches in this series add similar non-atomic flag operations as well.
Fix by acquiring zone->lock around TestSetPageHWPoison. This serializes with all buddy flag manipulation. The cost is
negligible: one lock/unlock in an extremely rare path (hardware memory errors).
--- mm/memory-failure.c | 6 ++++++ 1 file changed, 6 insertions(+)
mm/memory-failure.c +6 -0
@@ -2348,6 +2348,8 @@ int memory_failure(unsigned long pfn, int flags)unsigned long page_flags;bool retry = true;int hugetlb =0 ; +struct zone *zone; +unsigned long mf_flags;if (!sysctl_memory_failure_recovery)panic ("Memory failure on page %lx" , pfn);
@@ -2390,7 +2392,10 @@ int memory_failure(unsigned long pfn, int flags)if (hugetlb) goto unlock_mutex; + zone =page_zone (p); +spin_lock_irqsave (&zone->lock, mf_flags);if (TestSetPageHWPoison (p)) { +spin_unlock_irqrestore (&zone->lock, mf_flags); res = -EHWPOISON ;if (flags &MF_ACTION_REQUIRED ) res =kill_accessing_process (current, pfn, flags);
@@ -2399,6 +2404,7 @@ int memory_failure(unsigned long pfn, int flags)action_result (pfn,MF_MSG_ALREADY_POISONED ,MF_FAILED ); goto unlock_mutex; } +spin_unlock_irqrestore (&zone->lock, mf_flags);/* * We need/can do nothing about count=0 pages.
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 3/37 mm: page_alloc: propagate PageReported flag across buddy splits
In reply to Michael S. TsirkinWhen a reported free page is split via expand() to satisfy a smaller allocation, the sub-pages placed back on the free lists lose the PageReported flag. This means they will be unnecessarily re-reported to the hypervisor in the next reporting cycle, wasting work.
While I was unable to quantify the performance difference, it is an obvious waste, even if small.
Propagate the PageReported flag to sub-pages during expand(), both in page_del_and_expand() and try_to_claim_block(), so that they are recognized as already-reported.
--- mm/page_alloc.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)
mm/page_alloc.c +18 -3
@@ -1510,6 +1510,7 @@ static void split_large_buddy(struct zone *zone, struct page *page, unsigned long pfn, int order, fpi_t fpi) { unsigned long end = pfn + (1 << order); +bool reported =PageReported (page);VM_WARN_ON_ONCE (!IS_ALIGNED (pfn,1 << order));/* Caller removed page from freelist, buddy info cleared! */
@@ -1521,6 +1522,8 @@ static void split_large_buddy(struct zone *zone, struct page *page, do { int mt = get_pfnblock_migratetype(page, pfn); + if (reported) + __SetPageReported(page); __free_one_page(page, pfn, zone, order, mt, fpi); pfn += 1 << order; if (pfn == end)
@@ -1699,7 +1702,7 @@ struct page *__pageblock_pfn_to_page(unsigned long start_pfn,* -- nyc */ static inline unsigned int expand (struct zone *zone,struct page *page,int low, -int high,int migratetype) +int high,int migratetype,bool reported) {unsigned int size =1 << high;unsigned int nr_added =0 ;
@@ -1721,6 +1724,15 @@ static inline unsigned int expand(struct zone *zone, struct page *page, int low,__add_to_free_list (&page[size], zone, high, migratetype, false);set_buddy_order (&page[size], high); nr_added += size; + +/* +* The parent page has been reported to the host. The +* sub-pages are part of the same reported block, so mark +* them reported too. This avoids re-reporting pages that +* the host already knows about. +*/ +if (reported) +__SetPageReported (&page[size]); }return nr_added;
@@ -1731,9 +1743,10 @@ static __always_inline void page_del_and_expand(struct zone *zone, int high, int migratetype) { int nr_pages = 1 << high; +bool was_reported =page_reported (page);__del_page_from_free_list (page, zone, high, migratetype); - nr_pages -=expand (zone, page, low, high, migratetype); + nr_pages -=expand (zone, page, low, high, migratetype, was_reported);account_freepages (zone, -nr_pages, migratetype); }
@@ -2288,10 +2301,12 @@ try_to_claim_block(struct zone *zone, struct page *page,/* Take ownership for orders >= pageblock_order */ if (current_order >= pageblock_order) {unsigned int nr_added; +bool was_reported =page_reported (page);del_page_from_free_list (page, zone, current_order, block_type);change_pageblock_range (page, current_order, start_type); - nr_added =expand (zone, page, order, current_order, start_type); + nr_added =expand (zone, page, order, current_order, start_type, + was_reported);account_freepages (zone, nr_added, start_type);return page; }
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 4/37 mm: page_reporting: allow driver to set batch capacity
In reply to Michael S. TsirkinAdd a capacity field to page_reporting_dev_info so drivers can control the maximum number of pages per report batch. This is useful when the driver needs to reserve virtqueue descriptors for metadata (e.g., a bitmap buffer) alongside the page buffers.
The value is capped at PAGE_REPORTING_CAPACITY and rounded down to a power of 2. If unset (0), defaults to PAGE_REPORTING_CAPACITY.
The virtio_balloon driver sets capacity to the reporting virtqueue size, letting page_reporting adapt to whatever the device provides.
--- drivers/virtio/virtio_balloon.c | 5 +---- include/linux/page_reporting.h | 3 +++ mm/page_reporting.c | 28 +++++++++++++++++----------- 3 files changed, 21 insertions(+), 15 deletions(-)
drivers/virtio/virtio_balloon.c +1 -4
@@ -1017,10 +1017,6 @@ static int virtballoon_probe(struct virtio_device *vdev)unsigned int capacity; capacity =virtqueue_get_vring_size (vb->reporting_vq); -if (capacity <PAGE_REPORTING_CAPACITY ) { - err = -ENOSPC ; - goto out_unregister_oom; - } vb->pr_dev_info.order =PAGE_REPORTING_ORDER_UNSPECIFIED ;
@@ -1041,6 +1037,7 @@ static int virtballoon_probe(struct virtio_device *vdev) vb->pr_dev_info.order =5 ;#endif + vb->pr_dev_info.capacity = capacity; err =page_reporting_register (&vb->pr_dev_info);if (err) goto out_unregister_oom;
include/linux/page_reporting.h +3 -0
@@ -22,6 +22,9 @@ struct page_reporting_dev_info {/* Minimal order of page reporting */ unsigned int order; + +/* Max pages per report batch (default PAGE_REPORTING_CAPACITY) */ +unsigned int capacity; };/* Tear-down and bring-up for page reporting devices */
mm/page_reporting.c +17 -11
@@ -174,10 +174,10 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,* list processed. This should result in us reporting all pages on * an idle system in about 30 seconds. * -* The division here should be cheap since PAGE_REPORTING_CAPACITY -* should always be a power of 2. +* The division here should be cheap since capacity should +* always be a power of 2. */ - budget =DIV_ROUND_UP (area->nr_free,PAGE_REPORTING_CAPACITY *16 ); + budget =DIV_ROUND_UP (area->nr_free, prdev->capacity *16 );/* loop through free list adding unreported pages to sg list */ list_for_each_entry_safe (page, next, list, lru) {
@@ -222,10 +222,10 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,spin_unlock_irq (&zone->lock);/* begin processing pages in local list */ - err = prdev->report (prdev, sgl,PAGE_REPORTING_CAPACITY ); + err = prdev->report (prdev, sgl, prdev->capacity);/* reset offset since the full list was reported */ - *offset =PAGE_REPORTING_CAPACITY ; + *offset = prdev->capacity;/* update budget to reflect call to report function */ budget--;
@@ -234,7 +234,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,spin_lock_irq (&zone->lock);/* flush reported pages from the sg list */ -page_reporting_drain (prdev, sgl,PAGE_REPORTING_CAPACITY , !err); +page_reporting_drain (prdev, sgl, prdev->capacity, !err);/* * Reset next to first entry, the old next isn't valid
@@ -260,13 +260,13 @@ static int page_reporting_process_zone(struct page_reporting_dev_info *prdev, struct scatterlist *sgl, struct zone *zone) { - unsigned int order, mt, leftover, offset = PAGE_REPORTING_CAPACITY; + unsigned int order, mt, leftover, offset = prdev->capacity;unsigned long watermark;int err =0 ;/* Generate minimum watermark to be able to guarantee progress */ watermark =low_wmark_pages (zone) + - (PAGE_REPORTING_CAPACITY << page_reporting_order); + (prdev->capacity << page_reporting_order);/* * Cancel request if insufficient free memory or if we failed
@@ -290,7 +290,7 @@ page_reporting_process_zone(struct page_reporting_dev_info *prdev, }/* report the leftover pages before going idle */ - leftover =PAGE_REPORTING_CAPACITY - offset; + leftover = prdev->capacity - offset;if (leftover) { sgl = &sgl[offset]; err = prdev->report (prdev, sgl, leftover);
@@ -322,11 +322,11 @@ static void page_reporting_process(struct work_struct *work)atomic_set (&prdev->state, state);/* allocate scatterlist to store pages being reported on */ - sgl =kmalloc_objs (*sgl,PAGE_REPORTING_CAPACITY ); + sgl =kmalloc_objs (*sgl, prdev->capacity);if (!sgl) goto err_out; -sg_init_table (sgl,PAGE_REPORTING_CAPACITY ); +sg_init_table (sgl, prdev->capacity);for_each_zone (zone) { err =page_reporting_process_zone (prdev, sgl, zone);
@@ -377,6 +377,12 @@ int page_reporting_register(struct page_reporting_dev_info *prdev) page_reporting_order = pageblock_order; } +if (!prdev->capacity || prdev->capacity >PAGE_REPORTING_CAPACITY ) + prdev->capacity =PAGE_REPORTING_CAPACITY ; + +/* Power of 2 so division by capacity in the budget calc is cheap */ + prdev->capacity =rounddown_pow_of_two (prdev->capacity); +/* initialize state and work structures */ atomic_set (&prdev->state,PAGE_REPORTING_IDLE );INIT_DELAYED_WORK (&prdev->work, &page_reporting_process);
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 5/37 mm: hugetlb: remove dead alloc_hugetlb_folio stub
In reply to Michael S. TsirkinRemove the !CONFIG_HUGETLB_PAGE stub for alloc_hugetlb_folio().
The stub is dead code: all callers are in mm/hugetlb.c (CONFIG_HUGETLB_PAGE) or fs/hugetlbfs/inode.c (CONFIG_HUGETLBFS), and CONFIG_HUGETLB_PAGE is def_bool HUGETLBFS with nothing selecting it independently.
The stub is also broken: it returns NULL, but all callers check IS_ERR(folio), so a NULL return would not be caught and would crash on the subsequent folio dereference.
Remove it now since follow-up patches change the signature of alloc_hugetlb_folio and would otherwise need to update the broken stub too.
--- include/linux/hugetlb.h | 7 ------- 1 file changed, 7 deletions(-)
include/linux/hugetlb.h +0 -7
@@ -1129,13 +1129,6 @@ static inline void wait_for_freed_hugetlb_folios(void) { } -static inline struct folio *alloc_hugetlb_folio (struct vm_area_struct *vma, -unsigned long addr, -bool cow_from_owner) -{ -return NULL ; -} -static inline struct folio *alloc_hugetlb_folio_reserve (struct hstate *h,int preferred_nid,nodemask_t *nmask,gfp_t gfp_mask)
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 6/37 mm: move vma_alloc_folio_noprof to page_alloc.c
In reply to Michael S. TsirkinMove vma_alloc_folio_noprof() from an inline in gfp.h (for !NUMA) and mempolicy.c (for NUMA) to page_alloc.c.
This prepares for a subsequent patch that will thread user_addr through the allocator: having vma_alloc_folio_noprof in page_alloc.c means user_addr can be passed to the internal allocation path without changing public API signatures or duplicating plumbing in both gfp.h and mempolicy.c.
The !NUMA path gains the VM_DROPPABLE -> __GFP_NOWARN check that the NUMA path already had.
--- include/linux/gfp.h | 9 ++------- mm/mempolicy.c | 32 -------------------------------- mm/page_alloc.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 45 insertions(+), 39 deletions(-)
include/linux/gfp.h +2 -7
@@ -318,13 +318,13 @@ static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask,#define alloc_pages_node (...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__)) +struct folio *vma_alloc_folio_noprof (gfp_t gfp,int order, +struct vm_area_struct *vma,unsigned long addr);#ifdef CONFIG_NUMA struct page *alloc_pages_noprof (gfp_t gfp,unsigned int order);struct folio *folio_alloc_noprof (gfp_t gfp,unsigned int order);struct folio *folio_alloc_mpol_noprof (gfp_t gfp,unsigned int order,struct mempolicy *mpol,pgoff_t ilx,int nid); -struct folio *vma_alloc_folio_noprof (gfp_t gfp,int order,struct vm_area_struct *vma, -unsigned long addr);#else static inline struct page *alloc_pages_noprof (gfp_t gfp_mask,unsigned int order) {
@@ -339,11 +339,6 @@ static inline struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int orde {return folio_alloc_noprof (gfp, order); } -static inline struct folio *vma_alloc_folio_noprof (gfp_t gfp,int order, -struct vm_area_struct *vma,unsigned long addr) -{ -return folio_alloc_noprof (gfp, order); -}#endif #define alloc_pages (...) alloc_hooks(alloc_pages_noprof(__VA_ARGS__))
mm/mempolicy.c +0 -32
@@ -2516,38 +2516,6 @@ struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order,return page_rmappable_folio (page); } -/** -* vma_alloc_folio - Allocate a folio for a VMA. -* @gfp: GFP flags. -* @order: Order of the folio. -* @vma: Pointer to VMA. -* @addr: Virtual address of the allocation. Must be inside @vma. -* -* Allocate a folio for a specific address in @vma, using the appropriate -* NUMA policy. The caller must hold the mmap_lock of the mm_struct of the -* VMA to prevent it from going away. Should be used for all allocations -* for folios that will be mapped into user space, excepting hugetlbfs, and -* excepting where direct use of folio_alloc_mpol() is more appropriate. -* -* Return: The folio on success or NULL if allocation fails. -*/ -struct folio *vma_alloc_folio_noprof (gfp_t gfp,int order,struct vm_area_struct *vma, -unsigned long addr) -{ -struct mempolicy *pol; -pgoff_t ilx; -struct folio *folio; - -if (vma->vm_flags &VM_DROPPABLE ) - gfp |= __GFP_NOWARN; - - pol =get_vma_policy (vma, addr, order, &ilx); - folio =folio_alloc_mpol_noprof (gfp, order, pol, ilx,numa_node_id ()); -mpol_cond_put (pol); -return folio; -} -EXPORT_SYMBOL (vma_alloc_folio_noprof); -struct page *alloc_frozen_pages_noprof (gfp_t gfp,unsigned order) {struct mempolicy *pol = &default_policy;
mm/page_alloc.c +43 -0
49 lines not on this page · raw message
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 7/37 mm: thread user_addr through page allocator for cache-friendly zeroing
In reply to Michael S. TsirkinThread a user virtual address from vma_alloc_folio() down through the page allocator to post_alloc_hook(). This is plumbing preparation for a subsequent patch that will use user_addr to call folio_zero_user() for cache-friendly zeroing of user pages.
The user_addr is stored in struct alloc_context and flows through:
vma_alloc_folio -> folio_alloc_mpol -> __alloc_pages_mpol -> __alloc_frozen_pages -> get_page_from_freelist -> prep_new_page -> post_alloc_hook
USER_ADDR_NONE ((unsigned long)-1) is used for non-user allocations, since address 0 is a valid userspace mapping.
--- include/linux/gfp.h | 2 +- mm/compaction.c | 5 ++--- mm/hugetlb.c | 36 ++++++++++++++++++++---------------- mm/internal.h | 22 +++++++++++++++++++--- mm/mempolicy.c | 44 ++++++++++++++++++++++++++++++++------------ mm/mmap.c | 6 ++++++ mm/page_alloc.c | 44 +++++++++++++++++++++++++++++--------------- mm/slub.c | 4 ++-- 8 files changed, 111 insertions(+), 52 deletions(-)
include/linux/gfp.h +1 -1
@@ -337,7 +337,7 @@ static inline struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order)static inline struct folio *folio_alloc_mpol_noprof (gfp_t gfp,unsigned int order,struct mempolicy *mpol,pgoff_t ilx,int nid) { -return folio_alloc_noprof (gfp, order); +return __folio_alloc_noprof (gfp, order,numa_node_id (),NULL ); }#endif
mm/compaction.c +2 -3
@@ -82,7 +82,7 @@ static inline bool is_via_compact_memory(int order) { return false; }static struct page *mark_allocated_noprof (struct page *page,unsigned int order,gfp_t gfp_flags) { -post_alloc_hook (page, order, __GFP_MOVABLE); +post_alloc_hook (page, order, __GFP_MOVABLE,USER_ADDR_NONE );set_page_refcounted (page);return page; }
@@ -1849,8 +1849,7 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long daset_page_private (&freepage[size], start_order); } dst = (struct folio *)freepage; - -post_alloc_hook (&dst->page, order, __GFP_MOVABLE); +post_alloc_hook (&dst->page, order, __GFP_MOVABLE,USER_ADDR_NONE );set_page_refcounted (&dst->page);if (order)prep_compound_page (&dst->page, order);
mm/hugetlb.c +20 -16
@@ -1806,7 +1806,8 @@ struct address_space *hugetlb_folio_mapping_lock_write(struct folio *folio) }static struct folio *alloc_buddy_frozen_folio (int order,gfp_t gfp_mask, -int nid,nodemask_t *nmask,nodemask_t *node_alloc_noretry) +int nid,nodemask_t *nmask,nodemask_t *node_alloc_noretry, +unsigned long addr) {struct folio *folio;bool alloc_try_hard = true;
@@ -1823,7 +1824,7 @@ static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask,if (alloc_try_hard) gfp_mask |= __GFP_RETRY_MAYFAIL; - folio = (struct folio *)__alloc_frozen_pages (gfp_mask, order, nid, nmask); + folio = (struct folio *)__alloc_frozen_pages (gfp_mask, order, nid, nmask, addr);/* * If we did not specify __GFP_RETRY_MAYFAIL, but still got a
@@ -1852,7 +1853,7 @@ static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask,static struct folio *only_alloc_fresh_hugetlb_folio (struct hstate *h,gfp_t gfp_mask,int nid,nodemask_t *nmask, -nodemask_t *node_alloc_noretry) +nodemask_t *node_alloc_noretry,unsigned long addr) {struct folio *folio;int order =huge_page_order (h);
@@ -1864,7 +1865,7 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, folio =alloc_gigantic_frozen_folio (order, gfp_mask, nid, nmask); else folio =alloc_buddy_frozen_folio (order, gfp_mask, nid, nmask, - node_alloc_noretry); + node_alloc_noretry, addr);if (folio)init_new_hugetlb_folio (folio);return folio;
@@ -1878,11 +1879,12 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h,* pages is zero, and the accounting must be done in the caller. */ static struct folio *alloc_fresh_hugetlb_folio (struct hstate *h, -gfp_t gfp_mask,int nid,nodemask_t *nmask) +gfp_t gfp_mask,int nid,nodemask_t *nmask, +unsigned long addr) {struct folio *folio; - folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask,NULL ); + folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask,NULL , addr);if (folio)hugetlb_vmemmap_optimize_folio (h, folio);return folio;
@@ -1922,7 +1924,7 @@ static struct folio *alloc_pool_huge_folio(struct hstate *h,struct folio *folio; folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, node, - nodes_allowed, node_alloc_noretry); + nodes_allowed, node_alloc_noretry,USER_ADDR_NONE );if (folio)return folio; }
@@ -2091,7 +2093,8 @@ int dissolve_free_hugetlb_folios(unsigned long start_pfn, unsigned long end_pfn)* Allocates a fresh surplus page from the page allocator. */ static struct folio *alloc_surplus_hugetlb_folio (struct hstate *h, -gfp_t gfp_mask,int nid,nodemask_t *nmask) +gfp_t gfp_mask,int nid,nodemask_t *nmask, +unsigned long addr) {struct folio *folio =NULL ;
@@ -2103,7 +2106,7 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h, goto out_unlock;spin_unlock_irq (&hugetlb_lock); - folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask); + folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask, addr);if (!folio)return NULL ;
@@ -2146,7 +2149,7 @@ static struct folio *alloc_migrate_hugetlb_folio(struct hstate *h, gfp_t gfp_masif (hstate_is_gigantic (h))return NULL ; - folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask); + folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask,USER_ADDR_NONE );if (!folio)return NULL ;
@@ -2182,14 +2185,14 @@ struct folio *alloc_buddy_hugetlb_folio_with_mpol(struct hstate *h,if (mpol_is_preferred_many (mpol)) {gfp_t gfp = gfp_mask & ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - folio =alloc_surplus_hugetlb_folio (h, gfp, nid, nodemask); + folio =alloc_surplus_hugetlb_folio (h, gfp, nid, nodemask, addr);/* Fallback to all nodes if page==NULL */ nodemask =NULL ; }if (!folio) - folio =alloc_surplus_hugetlb_folio (h, gfp_mask, nid, nodemask); + folio =alloc_surplus_hugetlb_folio (h, gfp_mask, nid, nodemask, addr);mpol_cond_put (mpol);return folio; }
@@ -2296,7 +2299,8 @@ static int gather_surplus_pages(struct hstate *h, long delta)* down the road to pick the current node if that is the case. */ folio =alloc_surplus_hugetlb_folio (h,htlb_alloc_mask (h), -NUMA_NO_NODE , &alloc_nodemask); +NUMA_NO_NODE , &alloc_nodemask, +USER_ADDR_NONE );if (!folio) { alloc_ok = false;break ;
@@ -2702,7 +2706,7 @@ static int alloc_and_dissolve_hugetlb_folio(struct folio *old_folio,spin_unlock_irq (&hugetlb_lock); gfp_mask =htlb_alloc_mask (h) | __GFP_THISNODE; new_folio =alloc_fresh_hugetlb_folio (h, gfp_mask, - nid,NULL ); + nid,NULL ,USER_ADDR_NONE );if (!new_folio)return -ENOMEM ; goto retry;
@@ -3400,13 +3404,13 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)gfp_t gfp_mask =htlb_alloc_mask (h) | __GFP_THISNODE; folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, - &node_states[N_MEMORY ],NULL ); + &node_states[N_MEMORY ],NULL ,USER_ADDR_NONE );if (!folio && !list_empty (&folio_list) &&hugetlb_vmemmap_optimizable_size (h)) {prep_and_add_allocated_folios (h, &folio_list);INIT_LIST_HEAD (&folio_list); folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, - &node_states[N_MEMORY ],NULL ); + &node_states[N_MEMORY ],NULL ,USER_ADDR_NONE ); }if (!folio)break ;
mm/internal.h +19 -3
@@ -662,6 +662,16 @@ void calculate_min_free_kbytes(void);int __meminit init_per_zone_wmark_min (void );void page_alloc_sysctl_init (void ); +/* +* Sentinel for user_addr: indicates a non-user allocation. +* Cannot use 0 because address 0 is a valid userspace mapping. +* (unsigned long)-1 is safe because: +* 1. vm_end = addr + len <= TASK_SIZE, and vm_end is exclusive, +* so -1 is never inside any VMA. +* 2. It will only be compared to page-aligned addresses. +*/ +#define USER_ADDR_NONE ((unsigned long)-1) +/* * Structure for holding the mostly immutable allocation parameters passed * between functions involved in allocations, including the alloc_pages*
@@ -693,6 +703,7 @@ struct alloc_context {*/ enum zone_type highest_zoneidx;bool spread_dirty_pages; +unsigned long user_addr; };/*
@@ -916,24 +927,29 @@ static inline void init_compound_tail(struct page *tail,prep_compound_tail (tail, head, order); } -void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags); +void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags, +unsigned long user_addr);extern bool free_pages_prepare (struct page *page,unsigned int order);extern int user_min_free_kbytes;struct page *__alloc_frozen_pages_noprof (gfp_t ,unsigned int order,int nid, -nodemask_t *); +nodemask_t *,unsigned long user_addr);#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))void free_frozen_pages (struct page *page,unsigned int order); +void free_frozen_pages_zeroed (struct page *page,unsigned int order);void free_unref_folios (struct folio_batch *fbatch);#ifdef CONFIG_NUMA struct page *alloc_frozen_pages_noprof (gfp_t ,unsigned int order); +struct folio *folio_alloc_mpol_user_noprof (gfp_t gfp,unsigned int order, +struct mempolicy *pol,pgoff_t ilx,int nid, +unsigned long user_addr);#else static inline struct page *alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order) { -return __alloc_frozen_pages_noprof (gfp, order,numa_node_id (),NULL ); +return __alloc_frozen_pages_noprof (gfp, order,numa_node_id (),NULL ,USER_ADDR_NONE ); }#endif
mm/mempolicy.c +32 -12
102 lines not on this page · raw message
mm/mmap.c +6 -0
@@ -855,6 +855,12 @@ __get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,if (IS_ERR_VALUE (addr))return addr; +/* +* The check below ensures vm_end = addr + len <= TASK_SIZE. +* Since (unsigned long)-1 (USER_ADDR_NONE) >= TASK_SIZE and +* vm_end is exclusive, USER_ADDR_NONE is thus never a valid +* userspace address. +*/ if (addr >TASK_SIZE - len)return -ENOMEM ;if (offset_in_page (addr))
mm/page_alloc.c +29 -15
133 lines not on this page · raw message
mm/slub.c +2 -2
@@ -3275,7 +3275,7 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node,else if (node ==NUMA_NO_NODE ) page =alloc_frozen_pages (flags, order);else - page =__alloc_frozen_pages (flags, order, node,NULL ); + page =__alloc_frozen_pages (flags, order, node,NULL ,USER_ADDR_NONE );if (!page)return NULL ;
@@ -5235,7 +5235,7 @@ static void *___kmalloc_large_node(size_t size, gfp_t flags, int node)if (node ==NUMA_NO_NODE ) page =alloc_frozen_pages_noprof (flags, order);else - page =__alloc_frozen_pages_noprof (flags, order, node,NULL ); + page =__alloc_frozen_pages_noprof (flags, order, node,NULL ,USER_ADDR_NONE );if (page) { ptr =page_address (page);
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 8/37 mm: add alloc_contig_frozen_pages_user for cache-friendly zeroing
In reply to Michael S. TsirkinAdd a _user variant of alloc_contig_frozen_pages that accepts a user_addr parameter for cache-friendly zeroing of contiguous allocations.
No functional change; all existing callers continue to pass USER_ADDR_NONE.
--- include/linux/gfp.h | 6 ++++++ mm/page_alloc.c | 32 +++++++++++++++++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-)
include/linux/gfp.h +6 -0
@@ -453,6 +453,12 @@ struct page *alloc_contig_frozen_pages_noprof(unsigned long nr_pages,#define alloc_contig_frozen_pages (...) \ alloc_hooks(alloc_contig_frozen_pages_noprof(__VA_ARGS__)) +struct page *alloc_contig_frozen_pages_user_noprof (unsigned long nr_pages, +gfp_t gfp_mask,int nid,nodemask_t *nodemask, +unsigned long user_addr); +#define alloc_contig_frozen_pages_user (...) \ + alloc_hooks(alloc_contig_frozen_pages_user_noprof(__VA_ARGS__)) +struct page *alloc_contig_pages_noprof (unsigned long nr_pages,gfp_t gfp_mask,int nid,nodemask_t *nodemask);#define alloc_contig_pages (...) \
mm/page_alloc.c +25 -7
@@ -6999,8 +6999,9 @@ static void __free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages* * Return: zero on success or negative error code. */ -int alloc_contig_frozen_range_noprof (unsigned long start,unsigned long end, -acr_flags_t alloc_flags,gfp_t gfp_mask) +static int __alloc_contig_frozen_range (unsigned long start,unsigned long end, +acr_flags_t alloc_flags,gfp_t gfp_mask, +unsigned long user_addr) {const unsigned int order =ilog2 (end - start);unsigned long outer_start, outer_end;
@@ -7127,7 +7128,7 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,struct page *head =pfn_to_page (start);check_new_pages (head, order); -prep_new_page (head, order, gfp_mask,0 ,USER_ADDR_NONE ); +prep_new_page (head, order, gfp_mask,0 , user_addr); }else { ret = -EINVAL ; WARN(true, "PFN range: requested [%lu, %lu), allocated [%lu, %lu)\n",
@@ -7137,6 +7138,13 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,undo_isolate_page_range (start, end);return ret; } + +int alloc_contig_frozen_range_noprof (unsigned long start,unsigned long end, +acr_flags_t alloc_flags,gfp_t gfp_mask) +{ +return __alloc_contig_frozen_range (start, end, alloc_flags, gfp_mask, +USER_ADDR_NONE ); +}EXPORT_SYMBOL (alloc_contig_frozen_range_noprof);/**
@@ -7255,8 +7263,9 @@ static bool zone_spans_last_pfn(const struct zone *zone,* * Return: pointer to contiguous frozen pages on success, or NULL if not successful. */ -struct page *alloc_contig_frozen_pages_noprof (unsigned long nr_pages, -gfp_t gfp_mask,int nid,nodemask_t *nodemask) +struct page *alloc_contig_frozen_pages_user_noprof (unsigned long nr_pages, +gfp_t gfp_mask,int nid,nodemask_t *nodemask, +unsigned long user_addr) {unsigned long ret, pfn, flags;struct zonelist *zonelist;
@@ -7284,10 +7293,11 @@ struct page *alloc_contig_frozen_pages_noprof(unsigned long nr_pages,* win the race and cause allocation to fail. */ spin_unlock_irqrestore (&zone->lock, flags); - ret =alloc_contig_frozen_range_noprof (pfn, + ret =__alloc_contig_frozen_range (pfn, pfn + nr_pages,ACR_FLAGS_NONE , - gfp_mask); + gfp_mask, + user_addr);if (!ret)return pfn_to_page (pfn);spin_lock_irqsave (&zone->lock, flags);
@@ -7309,6 +7319,14 @@ struct page *alloc_contig_frozen_pages_noprof(unsigned long nr_pages, }return NULL ; } +EXPORT_SYMBOL (alloc_contig_frozen_pages_user_noprof); + +struct page *alloc_contig_frozen_pages_noprof (unsigned long nr_pages, +gfp_t gfp_mask,int nid,nodemask_t *nodemask) +{ +return alloc_contig_frozen_pages_user_noprof (nr_pages, gfp_mask, nid, + nodemask,USER_ADDR_NONE ); +}EXPORT_SYMBOL (alloc_contig_frozen_pages_noprof);/**
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 9/37 mm: hugetlb: thread user_addr through gigantic page allocation
In reply to Michael S. TsirkinThread the user_addr parameter through alloc_gigantic_frozen_folio so that gigantic page allocations can benefit from cache-friendly zeroing.
--- mm/hugetlb.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
mm/hugetlb.c +8 -5
@@ -1375,7 +1375,7 @@ static struct folio *dequeue_hugetlb_folio_vma(struct hstate *h,#if defined(CONFIG_ARCH_HAS_GIGANTIC_PAGE ) && defined(CONFIG_CONTIG_ALLOC )static struct folio *alloc_gigantic_frozen_folio (int order,gfp_t gfp_mask, -int nid,nodemask_t *nodemask) +int nid,nodemask_t *nodemask,unsigned long addr) {struct folio *folio;
@@ -1386,13 +1386,15 @@ static struct folio *alloc_gigantic_frozen_folio(int order, gfp_t gfp_mask,if (hugetlb_cma_exclusive_alloc ())return NULL ; - folio = (struct folio *)alloc_contig_frozen_pages (1 << order, gfp_mask, - nid, nodemask); + folio = (struct folio *)alloc_contig_frozen_pages_user (1 << order, + gfp_mask, + nid, nodemask, + addr); return folio; }#else /* !CONFIG_ARCH_HAS_GIGANTIC_PAGE || !CONFIG_CONTIG_ALLOC */ static struct folio *alloc_gigantic_frozen_folio (int order,gfp_t gfp_mask,int nid, -nodemask_t *nodemask) +nodemask_t *nodemask,unsigned long addr) {return NULL ; }
@@ -1862,7 +1864,8 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, nid =numa_mem_id ();if (order_is_gigantic (order)) - folio =alloc_gigantic_frozen_folio (order, gfp_mask, nid, nmask); + folio =alloc_gigantic_frozen_folio (order, gfp_mask, nid, nmask, + addr);else folio =alloc_buddy_frozen_folio (order, gfp_mask, nid, nmask, node_alloc_noretry, addr);
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 10/37 mm: add folio_zero_user stub for configs without THP/HUGETLBFS
In reply to Michael S. Tsirkinfolio_zero_user() is defined in mm/memory.c under CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS. A subsequent patch will call it from post_alloc_hook() for all user page zeroing, so configs without THP or HUGETLBFS will need a stub.
Add a stub that uses clear_user_highpages() with aligned addr_hint.
Without THP/HUGETLBFS, only order-0 user pages are allocated, so the locality optimization in the real folio_zero_user() (zero near the faulting address last) is not needed.
This also matches what vma_alloc_zeroed_movable_folio currently does.
--- include/linux/mm.h | 2 ++ mm/memory.c | 9 +++++++++ 2 files changed, 11 insertions(+)
include/linux/mm.h +2 -0
@@ -5070,6 +5070,8 @@ long copy_folio_from_user(struct folio *dst_folio,const void __user *usr_src, bool allow_pagefault); +#else /* !CONFIG_TRANSPARENT_HUGEPAGE && !CONFIG_HUGETLBFS */ +void folio_zero_user (struct folio *folio,unsigned long addr_hint);#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS */ #if MAX_NUMNODES >1
mm/memory.c +9 -0
@@ -7546,6 +7546,15 @@ long copy_folio_from_user(struct folio *dst_folio, }#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS */ +#if !defined(CONFIG_TRANSPARENT_HUGEPAGE ) && !defined(CONFIG_HUGETLBFS ) +void folio_zero_user (struct folio *folio,unsigned long addr_hint) +{ +unsigned long base =ALIGN_DOWN (addr_hint,folio_size (folio)); + +clear_user_highpages (&folio->page, base,folio_nr_pages (folio)); +} +#endif +#if defined(CONFIG_SPLIT_PTE_PTLOCKS ) &&ALLOC_SPLIT_PTLOCKS static struct kmem_cache *page_ptl_cachep;
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 11/37 mm: page_alloc: move prep_compound_page before post_alloc_hook
In reply to Michael S. TsirkinMove prep_compound_page() before post_alloc_hook() in prep_new_page().
The next patch adds a folio_zero_user() call to post_alloc_hook(), which uses folio_nr_pages() to determine how many pages to zero. Without compound metadata set up first, folio_nr_pages() returns 1 for higher-order allocations, so only the first page would be zeroed.
All other operations in post_alloc_hook() (arch_alloc_page, KASAN, debug, page owner, etc.) use raw page pointers with explicit order counts and are unaffected by this reordering.
Also reorder compaction_alloc_noprof() for consistency. Compaction currently passes USER_ADDR_NONE so folio_zero_user() is not called there, but keeping the same ordering avoids a future tripping hazard.
--- mm/compaction.c | 4 ++-- mm/page_alloc.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
mm/compaction.c +2 -2
@@ -1849,10 +1849,10 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long daset_page_private (&freepage[size], start_order); } dst = (struct folio *)freepage; -post_alloc_hook (&dst->page, order, __GFP_MOVABLE,USER_ADDR_NONE ); -set_page_refcounted (&dst->page);if (order)prep_compound_page (&dst->page, order); +post_alloc_hook (&dst->page, order, __GFP_MOVABLE,USER_ADDR_NONE ); +set_page_refcounted (&dst->page); cc->nr_freepages -=1 << order; cc->nr_migratepages -=1 << order;return page_rmappable_folio (&dst->page);
mm/page_alloc.c +2 -2
@@ -1877,11 +1877,11 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags unsigned int alloc_flags, unsigned long user_addr) { -post_alloc_hook (page ,order ,gfp_flags ,user_addr ); -if (order && (gfp_flags & __GFP_COMP))prep_compound_page (page, order); +post_alloc_hook (page, order, gfp_flags, user_addr); +/* * page is set pfmemalloc when ALLOC_NO_WATERMARKS was necessary to * allocate the page. The expectation is that the caller is taking
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 12/37 mm: use folio_zero_user for user pages in post_alloc_hook
In reply to Michael S. TsirkinWhen post_alloc_hook() needs to zero a page for an explicit __GFP_ZERO allocation for a user page (user_addr is set), use folio_zero_user() instead of kernel_init_pages(). This zeros near the faulting address last, keeping those cachelines hot for the impending
user access.
folio_zero_user() is only used for explicit __GFP_ZERO, not for init_on_alloc. On architectures with virtually-indexed caches (e.g., ARM), clear_user_highpage() performs per-line cache operations; using it for init_on_alloc would add overhead that kernel_init_pages() avoids (the page fault path flushes the cache at PTE installation time regardless).
No functional change yet: current callers do not pass __GFP_ZERO for user pages (they zero at the callsite instead). Subsequent patches will convert them.
--- mm/page_alloc.c | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-)
mm/page_alloc.c +32 -3
@@ -1864,9 +1864,38 @@ inline void post_alloc_hook(struct page *page, unsigned int order,for (i =0 ; i !=1 << order; ++i)page_kasan_tag_reset (page + i); } -/* If memory is still not initialized, initialize it now. */ -if (init) -kernel_init_pages (page,1 << order); +/* +* On architectures with cache aliasing, pages zeroed via the +* kernel direct map (e.g. init_on_free) must be re-zeroed +* through a user-congruent mapping. Host-zeroed pages +* (zeroed flag) don't need this: physical RAM is clean. +*/ +if (!init && (gfp_flags & __GFP_ZERO) && + user_addr !=USER_ADDR_NONE && +user_alloc_needs_zeroing ()) + init = true; +/* +* If memory is still not initialized, initialize it now. +* When __GFP_ZERO was explicitly requested and user_addr is set, +* use folio_zero_user() which zeros near the faulting address +* last, keeping those cachelines hot. For init_on_alloc, use +* kernel_init_pages() to avoid unnecessary cache flush overhead +* on architectures with virtually-indexed caches. +*/ +if (init) { +if ((gfp_flags & __GFP_ZERO) && user_addr !=USER_ADDR_NONE ) { +/* +* folio_zero_user relies on folio_nr_pages which +* requires __GFP_COMP for order > 0. All user folio +* allocations set __GFP_COMP via __folio_alloc. +* user_addr != USER_ADDR_NONE implies sleepable +* context (user page fault). +*/ +VM_WARN_ON_ONCE (order && !(gfp_flags & __GFP_COMP)); +folio_zero_user (page_folio (page), user_addr); + }else +kernel_init_pages (page,1 << order); + }set_page_owner (page, order, gfp_flags);page_table_check_alloc (page, order);
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 13/37 mm: use __GFP_ZERO in vma_alloc_zeroed_movable_folio
In reply to Michael S. TsirkinNow that post_alloc_hook() handles cache-friendly user page zeroing via folio_zero_user(), convert vma_alloc_zeroed_movable_folio() to pass __GFP_ZERO instead of zeroing at the callsite.
--- include/linux/highmem.h | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
include/linux/highmem.h +2 -7
@@ -320,13 +320,8 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); -if (folio &&user_alloc_needs_zeroing ()) -clear_user_highpage (&folio->page, vaddr); - -return folio; +return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO, +0 , vma, vaddr); }#endif
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 14/37 mm: remove arch vma_alloc_zeroed_movable_folio overrides
In reply to Michael S. TsirkinNow that the generic vma_alloc_zeroed_movable_folio() uses __GFP_ZERO, the arch-specific macros on alpha, m68k, s390, and x86 that did the same thing are redundant. Remove them.
arm64 is not affected: it has a real function override that handles MTE tag zeroing, not just __GFP_ZERO.
--- arch/alpha/include/asm/page.h | 3 --- arch/m68k/include/asm/page_no.h | 3 --- arch/s390/include/asm/page.h | 3 --- arch/x86/include/asm/page.h | 3 --- include/linux/highmem.h | 8 +++++--- 5 files changed, 5 insertions(+), 15 deletions(-)
arch/alpha/include/asm/page.h +0 -3
@@ -12,9 +12,6 @@extern void clear_page (void *page); -#define vma_alloc_zeroed_movable_folio (vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr) -extern void copy_page (void * _to,void * _from);#define copy_user_page (to, from, vaddr, pg) copy_page(to, from)
arch/m68k/include/asm/page_no.h +0 -3
@@ -12,9 +12,6 @@ extern unsigned long memory_end;#define copy_user_page (to, from, vaddr, pg) copy_page(to, from) -#define vma_alloc_zeroed_movable_folio (vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr) -#define __pa (vaddr) ((unsigned long)(vaddr))#define __va (paddr) ((void *)((unsigned long)(paddr)))
arch/s390/include/asm/page.h +0 -3
@@ -67,9 +67,6 @@ static inline void copy_page(void *to, void *from)#define copy_user_page (to, from, vaddr, pg) copy_page(to, from) -#define vma_alloc_zeroed_movable_folio (vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr) -#ifdef CONFIG_STRICT_MM_TYPECHECKS #define STRICT_MM_TYPECHECKS #endif
arch/x86/include/asm/page.h +0 -3
@@ -28,9 +28,6 @@ static inline void copy_user_page(void *to, void *from, unsigned long vaddr,copy_page (to, from); } -#define vma_alloc_zeroed_movable_folio (vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr) -#ifndef __pa#define __pa (x) __phys_addr((unsigned long)(x))#endif
include/linux/highmem.h +5 -3
@@ -303,7 +303,6 @@ static inline void clear_user_highpages(struct page *page, unsigned long vaddr,#endif } -#ifndef vma_alloc_zeroed_movable_folio/** * vma_alloc_zeroed_movable_folio - Allocate a zeroed page for a VMA. * @vma: The VMA the page is to be allocated for.
@@ -317,12 +316,15 @@ static inline void clear_user_highpages(struct page *page, unsigned long vaddr,* we are out of memory. */ static inline -struct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma, +struct folio *vma_alloc_zeroed_movable_folio_noprof (struct vm_area_struct *vma,unsigned long vaddr) { -return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO, +return vma_alloc_folio_noprof (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); } +#ifndef vma_alloc_zeroed_movable_folio +#define vma_alloc_zeroed_movable_folio (...) \ + alloc_hooks(vma_alloc_zeroed_movable_folio_noprof(__VA_ARGS__))#endif static inline void clear_highpage (struct page *page)
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 15/37 mm: alloc_anon_folio: pass raw fault address to vma_alloc_folio
In reply to Michael S. TsirkinPass vmf->address directly instead of ALIGN_DOWN(vmf->address, ...). vma_alloc_folio_noprof now aligns internally for NUMA interleave, and post_alloc_hook will use the raw address for cache-friendly zeroing via folio_zero_user().
--- mm/memory.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
mm/memory.c +1 -2
@@ -5252,8 +5252,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)/* Try allocating the highest of the remaining orders. */ gfp =vma_thp_gfp_mask (vma);while (orders) { - addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << order); - folio =vma_alloc_folio (gfp, order, vma, addr); + folio =vma_alloc_folio (gfp, order, vma, vmf->address);if (folio) {if (mem_cgroup_charge (folio, vma->vm_mm, gfp)) {count_mthp_stat (order,MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE );
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 16/37 mm: alloc_swap_folio: pass raw fault address to vma_alloc_folio
In reply to Michael S. TsirkinSame change as the previous patch but for alloc_swap_folio:
pass vmf->address directly instead of ALIGN_DOWN(vmf->address, ...).
--- mm/memory.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
mm/memory.c +1 -2
@@ -4734,8 +4734,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)/* Try allocating the highest of the remaining orders. */ gfp =vma_thp_gfp_mask (vma);while (orders) { - addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << order); - folio =vma_alloc_folio (gfp, order, vma, addr); + folio =vma_alloc_folio (gfp, order, vma, vmf->address);if (folio) {if (!mem_cgroup_swapin_charge_folio (folio, vma->vm_mm, gfp, entry))
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 17/37 mm: use __GFP_ZERO in alloc_anon_folio
In reply to Michael S. TsirkinConvert alloc_anon_folio() to pass __GFP_ZERO instead of zeroing at the callsite. post_alloc_hook uses the fault address passed through vma_alloc_folio for cache-friendly zeroing.
Note: with __GFP_ZERO, the folio is zeroed before mem_cgroup_charge(). If the charge fails, the zeroing work is wasted. Previously zeroing was done after a successful charge. This is inherent to moving zeroing into the allocator.
--- mm/memory.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-)
mm/memory.c +1 -10
@@ -5249,7 +5249,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto fallback;/* Try allocating the highest of the remaining orders. */ - gfp =vma_thp_gfp_mask (vma); + gfp =vma_thp_gfp_mask (vma) | __GFP_ZERO;while (orders) { folio =vma_alloc_folio (gfp, order, vma, vmf->address);if (folio) {
@@ -5259,15 +5259,6 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto next; }folio_throttle_swaprate (folio, gfp); -/* -* When a folio is not zeroed during allocation -* (__GFP_ZERO not used) or user folios require special -* handling, folio_zero_user() is used to make sure -* that the page corresponding to the faulting address -* will be hot in the cache after zeroing. -*/ -if (user_alloc_needs_zeroing ()) -folio_zero_user (folio, vmf->address);return folio; } next:
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 18/37 mm: vma_alloc_anon_folio_pmd: pass raw fault address to vma_alloc_folio
In reply to Michael S. TsirkinNow that vma_alloc_folio aligns the address internally, drop the redundant HPAGE_PMD_MASK alignment at the callsite.
--- mm/huge_memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/huge_memory.c +1 -1
@@ -1337,7 +1337,7 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma,const int order =HPAGE_PMD_ORDER ;struct folio *folio; - folio =vma_alloc_folio (gfp, order, vma, addr &HPAGE_PMD_MASK ); + folio =vma_alloc_folio (gfp, order, vma, addr);if (unlikely (!folio)) {count_vm_event (THP_FAULT_FALLBACK );
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 19/37 mm: use __GFP_ZERO in vma_alloc_anon_folio_pmd
In reply to Michael S. TsirkinConvert vma_alloc_anon_folio_pmd() to pass __GFP_ZERO instead of zeroing at the callsite. post_alloc_hook uses the fault address passed through vma_alloc_folio for cache-friendly zeroing.
Note: with __GFP_ZERO, the folio is zeroed before mem_cgroup_charge(). If the charge fails, the zeroing work is wasted. Previously zeroing was done after a successful charge. This is inherent to moving zeroing into the allocator.
--- mm/huge_memory.c | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-)
mm/huge_memory.c +1 -9
@@ -1333,7 +1333,7 @@ EXPORT_SYMBOL_GPL(thp_get_unmapped_area);static struct folio *vma_alloc_anon_folio_pmd (struct vm_area_struct *vma,unsigned long addr) { -gfp_t gfp =vma_thp_gfp_mask (vma); +gfp_t gfp =vma_thp_gfp_mask (vma) | __GFP_ZERO;const int order =HPAGE_PMD_ORDER ;struct folio *folio;
@@ -1356,14 +1356,6 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, }folio_throttle_swaprate (folio, gfp); -/* -* When a folio is not zeroed during allocation (__GFP_ZERO not used) -* or user folios require special handling, folio_zero_user() is used to -* make sure that the page corresponding to the faulting address will be -* hot in the cache after zeroing. -*/ -if (user_alloc_needs_zeroing ()) -folio_zero_user (folio, addr);/* * The memory barrier inside __folio_mark_uptodate makes sure that * folio_zero_user writes become visible before the set_pmd_at()
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 20/37 mm: hugetlb: add gfp parameter and skip zeroing for zeroed pages
In reply to Michael S. TsirkinAdd a gfp_t parameter to alloc_hugetlb_folio(). When __GFP_ZERO is set, the function guarantees the returned folio is zeroed:
- Fresh allocations (buddy or gigantic): zeroed by post_alloc_hook via __GFP_ZERO, HPG_zeroed set by alloc_surplus_hugetlb_folio.
- Pool pages with HPG_zeroed set: already zeroed, skip.
- Pool pages without HPG_zeroed: zeroed via folio_zero_user().
The address parameter is renamed to user_addr; the function aligns it internally for reservation and NUMA policy lookups. For pages that need zeroing, user_addr is passed to folio_zero_user() for cache-friendly zeroing near the faulting subpage. All callers pass a page-aligned address; the hugetlb_no_page caller passes vmf->real_address & PAGE_MASK for consistency.
HPG_zeroed (stored in hugetlb folio->private bits) tracks known-zero pool pages. It is set when alloc_surplus_hugetlb_folio allocates with __GFP_ZERO, and cleared in free_huge_folio when the page returns to the pool after userspace use.
--- fs/hugetlbfs/inode.c | 3 +-- include/linux/hugetlb.h | 5 ++++- mm/hugetlb.c | 42 +++++++++++++++++++++++++++++------------ 3 files changed, 35 insertions(+), 15 deletions(-)
fs/hugetlbfs/inode.c +1 -2
@@ -810,13 +810,12 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,* folios in these areas, we need to consume the reserves * to keep reservation accounting consistent. */ - folio =alloc_hugetlb_folio (&pseudo_vma, addr, false); + folio =alloc_hugetlb_folio (&pseudo_vma, addr, false, __GFP_ZERO);if (IS_ERR (folio)) {mutex_unlock (&hugetlb_fault_mutex_table[hash]); error =PTR_ERR (folio); goto out; } -folio_zero_user (folio, addr);__folio_mark_uptodate (folio); error =hugetlb_add_to_page_cache (folio, mapping, index);if (unlikely (error)) {
include/linux/hugetlb.h +4 -1
@@ -599,6 +599,7 @@ enum hugetlb_page_flags { HPG_vmemmap_optimized, HPG_raw_hwp_unreliable, HPG_cma, + HPG_zeroed, __NR_HPAGEFLAGS, };
@@ -659,6 +660,7 @@ HPAGEFLAG(Freed, freed) HPAGEFLAG(VmemmapOptimized, vmemmap_optimized) HPAGEFLAG(RawHwpUnreliable, raw_hwp_unreliable) HPAGEFLAG(Cma, cma) +HPAGEFLAG(Zeroed, zeroed)#ifdef CONFIG_HUGETLB_PAGE
@@ -706,7 +708,8 @@ int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *list);int replace_free_hugepage_folios (unsigned long start_pfn,unsigned long end_pfn);void wait_for_freed_hugetlb_folios (void );struct folio *alloc_hugetlb_folio (struct vm_area_struct *vma, -unsigned long addr,bool cow_from_owner); +unsigned long user_addr,bool cow_from_owner, +gfp_t gfp);struct folio *alloc_hugetlb_folio_nodemask (struct hstate *h,int preferred_nid,nodemask_t *nmask,gfp_t gfp_mask,bool allow_alloc_fallback);
mm/hugetlb.c +30 -12
130 lines not on this page · raw message
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 21/37 mm: memfd: skip zeroing for zeroed hugetlb pool pages
In reply to Michael S. TsirkinAdd bool *zeroed output to alloc_hugetlb_folio_reserve() so callers can check whether the pool page is known-zero. memfd's memfd_alloc_folio() uses this to skip the explicit folio_zero_user() when the page is already zero.
This avoids redundant zeroing for memfd hugetlb pages that were pre-allocated into the pool and never mapped to userspace.
--- include/linux/cma.h | 3 ++- include/linux/hugetlb.h | 6 ++++-- mm/cma.c | 6 ++++-- mm/hugetlb.c | 11 +++++++++-- mm/hugetlb_cma.c | 4 ++-- mm/memfd.c | 14 ++++++++------ 6 files changed, 29 insertions(+), 15 deletions(-)
include/linux/cma.h +2 -1
@@ -53,7 +53,8 @@ extern bool cma_release(struct cma *cma, const struct page *pages, unsigned longstruct page *cma_alloc_frozen (struct cma *cma,unsigned long count,unsigned int align,bool no_warn); -struct page *cma_alloc_frozen_compound (struct cma *cma,unsigned int order); +struct page *cma_alloc_frozen_compound (struct cma *cma,unsigned int order, +gfp_t caller_gfp);bool cma_release_frozen (struct cma *cma,const struct page *pages,unsigned long count);
include/linux/hugetlb.h +4 -2
@@ -714,7 +714,8 @@ struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, nodemask_t *nmask, gfp_t gfp_mask, bool allow_alloc_fallback);struct folio *alloc_hugetlb_folio_reserve (struct hstate *h,int preferred_nid, -nodemask_t *nmask,gfp_t gfp_mask); +nodemask_t *nmask,gfp_t gfp_mask, +bool *zeroed);int hugetlb_add_to_page_cache (struct folio *folio,struct address_space *mapping,pgoff_t idx);
@@ -1134,7 +1135,8 @@ static inline void wait_for_freed_hugetlb_folios(void)static inline struct folio *alloc_hugetlb_folio_reserve (struct hstate *h,int preferred_nid, -nodemask_t *nmask,gfp_t gfp_mask) +nodemask_t *nmask,gfp_t gfp_mask, +bool *zeroed) {return NULL ; }
mm/cma.c +4 -2
@@ -924,9 +924,11 @@ struct page *cma_alloc_frozen(struct cma *cma, unsigned long count,return __cma_alloc_frozen (cma, count, align, gfp); } -struct page *cma_alloc_frozen_compound (struct cma *cma,unsigned int order) +struct page *cma_alloc_frozen_compound (struct cma *cma,unsigned int order, +gfp_t caller_gfp) { -gfp_t gfp =GFP_KERNEL | __GFP_COMP | __GFP_NOWARN; +gfp_t gfp =GFP_KERNEL | __GFP_COMP | __GFP_NOWARN | + (caller_gfp & __GFP_ZERO);return __cma_alloc_frozen (cma,1 << order, order, gfp); }
mm/hugetlb.c +9 -2
@@ -2208,7 +2208,7 @@ struct folio *alloc_buddy_hugetlb_folio_with_mpol(struct hstate *h, }struct folio *alloc_hugetlb_folio_reserve (struct hstate *h,int preferred_nid, -nodemask_t *nmask,gfp_t gfp_mask) +nodemask_t *nmask,gfp_t gfp_mask,bool *zeroed) {struct folio *folio;
@@ -2224,6 +2224,12 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid, h->resv_huge_pages--;spin_unlock_irq (&hugetlb_lock); + +if (zeroed && folio) { + *zeroed =folio_test_hugetlb_zeroed (folio); +folio_clear_hugetlb_zeroed (folio); + } +return folio; }
@@ -2308,7 +2314,8 @@ static int gather_surplus_pages(struct hstate *h, long delta)* It is okay to use NUMA_NO_NODE because we use numa_mem_id() * down the road to pick the current node if that is the case. */ - folio =alloc_surplus_hugetlb_folio (h,htlb_alloc_mask (h), + folio =alloc_surplus_hugetlb_folio (h, +htlb_alloc_mask (h),NUMA_NO_NODE , &alloc_nodemask,USER_ADDR_NONE );if (!folio) {
mm/hugetlb_cma.c +2 -2
@@ -35,14 +35,14 @@ struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask,return NULL ;if (hugetlb_cma[nid]) - page =cma_alloc_frozen_compound (hugetlb_cma[nid], order); + page =cma_alloc_frozen_compound (hugetlb_cma[nid], order, gfp_mask);if (!page && !(gfp_mask & __GFP_THISNODE)) {for_each_node_mask (node, *nodemask) {if (node == nid || !hugetlb_cma[node])continue ; - page =cma_alloc_frozen_compound (hugetlb_cma[node], order); + page =cma_alloc_frozen_compound (hugetlb_cma[node], order, gfp_mask);if (page)break ; }
mm/memfd.c +8 -6
@@ -69,6 +69,7 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx)#ifdef CONFIG_HUGETLB_PAGE struct folio *folio;gfp_t gfp_mask; +bool zeroed;if (is_file_hugepages (memfd)) {/*
@@ -93,17 +94,18 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx) folio =alloc_hugetlb_folio_reserve (h,numa_node_id (),NULL , - gfp_mask); + gfp_mask, + &zeroed);if (folio) {u32 hash;/* -* Zero the folio to prevent information leaks to userspace. -* Use folio_zero_user() which is optimized for huge/gigantic -* pages. Pass 0 as addr_hint since this is not a faulting path -* and we don't have a user virtual address yet. +* Zero the folio to prevent information leaks to +* userspace. Skip if the pool page is known-zero +* (HPG_zeroed set during pool pre-allocation). */ -folio_zero_user (folio,0 ); +if (!zeroed) +folio_zero_user (folio,0 );/* * Mark the folio uptodate before adding to page cache,
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 22/37 mm: page_reporting: skip redundant zeroing of host-zeroed reported pages
In reply to Michael S. TsirkinWhen a guest reports free pages to the hypervisor via the page reporting framework (used by virtio-balloon and hv_balloon), the host typically zeros those pages when reclaiming their backing memory. However, when those pages are later allocated in the guest, post_alloc_hook() unconditionally zeros them again if __GFP_ZERO is set. This double-zeroing is wasteful, especially for large pages.
Avoid redundant zeroing:
- Add a host_zeroes_pages flag to page_reporting_dev_info, allowing drivers to declare that their host zeros reported pages on reclaim. A static key (page_reporting_host_zeroes) gates the fast path.
- Add PG_zeroed page flag (sharing PG_private bit) to mark pages that have been zeroed by the host. Set it in page_reporting_drain() after the host reports them.
- Thread the zeroed bool through rmqueue -> prep_new_page -> post_alloc_hook, where it skips redundant zeroing for __GFP_ZERO allocations.
No driver sets host_zeroes_pages yet; a follow-up patch to virtio_balloon is needed to opt in.
--- include/linux/page-flags.h | 9 +++++ include/linux/page_reporting.h | 3 ++ mm/compaction.c | 6 ++- mm/internal.h | 2 +- mm/page_alloc.c | 68 +++++++++++++++++++++++----------- mm/page_reporting.c | 14 ++++++- mm/page_reporting.h | 12 ++++++ 7 files changed, 88 insertions(+), 26 deletions(-)
include/linux/page-flags.h +9 -0
@@ -135,6 +135,8 @@ enum pageflags { PG_swapcache = PG_owner_priv_1,/* Swap page: swp_entry_t in private */ /* Some filesystems */ PG_checked = PG_owner_priv_1, +/* Page contents are known to be zero */ + PG_zeroed = PG_private,/* * Depending on the way an anonymous folio can be mapped into a page
@@ -673,6 +675,13 @@ FOLIO_TEST_CLEAR_FLAG_FALSE(young) FOLIO_FLAG_FALSE(idle)#endif +/* +* PageZeroed() tracks pages known to be zero. The allocator +* uses this to skip redundant zeroing in post_alloc_hook(). +*/ +__PAGEFLAG(Zeroed, zeroed, PF_NO_COMPOUND) +#define __PG_ZEROED (1UL << PG_zeroed) +/* * PageReported() is used to track reported free pages within the Buddy * allocator. We can use the non-atomic version of the test and set
include/linux/page_reporting.h +3 -0
@@ -14,6 +14,9 @@ struct page_reporting_dev_info {int (*report)(struct page_reporting_dev_info *prdev,struct scatterlist *sg,unsigned int nents); +/* If true, host zeros reported pages on reclaim */ +bool host_zeroes_pages; +/* work struct for processing reports */ struct delayed_work work;
mm/compaction.c +4 -2
@@ -82,7 +82,8 @@ static inline bool is_via_compact_memory(int order) { return false; }static struct page *mark_allocated_noprof (struct page *page,unsigned int order,gfp_t gfp_flags) { -post_alloc_hook (page, order, __GFP_MOVABLE,USER_ADDR_NONE ); +__ClearPageZeroed (page); +post_alloc_hook (page, order, __GFP_MOVABLE, false,USER_ADDR_NONE );set_page_refcounted (page);return page; }
@@ -1849,9 +1850,10 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long daset_page_private (&freepage[size], start_order); } dst = (struct folio *)freepage; +__ClearPageZeroed (&dst->page);if (order)prep_compound_page (&dst->page, order); -post_alloc_hook (&dst->page, order, __GFP_MOVABLE,USER_ADDR_NONE ); +post_alloc_hook (&dst->page, order, __GFP_MOVABLE, false,USER_ADDR_NONE );set_page_refcounted (&dst->page); cc->nr_freepages -=1 << order; cc->nr_migratepages -=1 << order;
mm/internal.h +1 -1
@@ -928,7 +928,7 @@ static inline void init_compound_tail(struct page *tail, }void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags, -unsigned long user_addr); +bool zeroed,unsigned long user_addr);extern bool free_pages_prepare (struct page *page,unsigned int order);extern int user_min_free_kbytes;
mm/page_alloc.c +46 -22
214 lines not on this page · raw message
mm/page_reporting.c +13 -1
@@ -50,6 +50,8 @@ EXPORT_SYMBOL_GPL(page_reporting_order);#define PAGE_REPORTING_DELAY (2 * HZ)static struct page_reporting_dev_info __rcu *pr_dev_info__read_mostly ; +DEFINE_STATIC_KEY_FALSE (page_reporting_host_zeroes); +enum {PAGE_REPORTING_IDLE =0 ,PAGE_REPORTING_REQUESTED ,
@@ -129,8 +131,11 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,* report on the new larger page when we make our way * up to that higher order. */ -if (PageBuddy (page) &&buddy_order (page) == order) +if (PageBuddy (page) &&buddy_order (page) == order) {__SetPageReported (page); +if (page_reporting_host_zeroes_pages ()) +__SetPageZeroed (page); + } }while ((sg =sg_next (sg)));/* reinitialize scatterlist now that it is empty */
@@ -393,6 +398,10 @@ int page_reporting_register(struct page_reporting_dev_info *prdev)/* Assign device to allow notifications */ rcu_assign_pointer (pr_dev_info, prdev); +/* enable zeroed page optimization if host zeroes reported pages */ +if (prdev->host_zeroes_pages) +static_branch_enable (&page_reporting_host_zeroes); +/* enable page reporting notification */ if (!static_key_enabled (&page_reporting_enabled)) {static_branch_enable (&page_reporting_enabled);
@@ -417,6 +426,9 @@ void page_reporting_unregister(struct page_reporting_dev_info *prdev)/* Flush any existing work, and lock it out */ cancel_delayed_work_sync (&prdev->work); + +if (prdev->host_zeroes_pages) +static_branch_disable (&page_reporting_host_zeroes); }mutex_unlock (&page_reporting_mutex);
mm/page_reporting.h +12 -0
@@ -15,6 +15,13 @@ DECLARE_STATIC_KEY_FALSE(page_reporting_enabled);extern unsigned int page_reporting_order;void __page_reporting_notify (void ); +DECLARE_STATIC_KEY_FALSE (page_reporting_host_zeroes); + +static inline bool page_reporting_host_zeroes_pages (void ) +{ +return static_branch_unlikely (&page_reporting_host_zeroes); +} +static inline bool page_reported (struct page *page) { return static_branch_unlikely(&page_reporting_enabled) &&
@@ -46,6 +53,11 @@ static inline void page_reporting_notify_free(unsigned int order)#else /* CONFIG_PAGE_REPORTING */ #define page_reported (_page) false +static inline bool page_reporting_host_zeroes_pages (void ) +{ +return false; +} +static inline void page_reporting_notify_free (unsigned int order) { }
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 23/37 mm: page_alloc: use aliasing checks instead of user_alloc_needs_zeroing
In reply to Michael S. TsirkinReplace user_alloc_needs_zeroing() with the direct aliasing checks (cpu_dcache_is_aliasing() || cpu_icache_is_aliasing()) in the post_alloc_hook aliasing guard.
user_alloc_needs_zeroing() includes a !init_on_alloc term that means "allocator didn't zero this page." But in this guard's context (!zeroed && !init && __GFP_ZERO), we already know the page is zero; init incorporates init_on_alloc via want_init_on_alloc(). The only question left is whether the cache architecture needs the data re-zeroed through a congruent mapping, which is purely cpu_dcache_is_aliasing() || cpu_icache_is_aliasing().
On non-aliasing architectures with init_on_free=true and init_on_alloc=false, this avoids a redundant re-zero of an already-zero page.
--- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c +1 -1
@@ -1883,7 +1883,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order,*/ if (!zeroed && !init && (gfp_flags & __GFP_ZERO) && user_addr !=USER_ADDR_NONE && -user_alloc_needs_zeroing ()) + (cpu_dcache_is_aliasing () ||cpu_icache_is_aliasing ())) init = true;/* * If memory is still not initialized, initialize it now.
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 24/37 mm: page_reporting: add per-page zeroed bitmap for host feedback
In reply to Michael S. TsirkinThe host may skip zeroing some reported pages (e.g., due to alignment constraints or bounce buffer fallback in QEMU). Currently, when host_zeroes_pages is set, all reported pages are unconditionally marked PG_zeroed - even ones the host did not actually zero.
Add a zeroed_bitmap to page_reporting_dev_info that the report() callback can use to indicate which pages were actually zeroed. The driver's report() callback is responsible for managing the
bitmap: zeroing it before sending pages to the host, then setting bits for pages the host actually zeroed.
page_reporting_drain() checks the bitmap per-page in addition to the global host_zeroes_pages flag.
No driver sets host_zeroes_pages yet, so the static key is off and the bitmap is never read. Behavior is unchanged.
--- include/linux/page_reporting.h | 7 +++++++ mm/page_reporting.c | 8 ++++++-- 2 files changed, 13 insertions(+), 2 deletions(-)
include/linux/page_reporting.h +7 -0
@@ -17,6 +17,13 @@ struct page_reporting_dev_info {/* If true, host zeros reported pages on reclaim */ bool host_zeroes_pages; +/* +* Per-page zeroed status, indexed by scatterlist position. +* The driver's report() callback must clear the bitmap, +* then set bits for pages that were actually zeroed. +*/ +DECLARE_BITMAP (zeroed_bitmap,PAGE_REPORTING_CAPACITY ); +/* work struct for processing reports */ struct delayed_work work;
mm/page_reporting.c +6 -2
@@ -108,6 +108,7 @@ page_reporting_drain(struct page_reporting_dev_info *prdev, struct scatterlist *sgl, unsigned int nents, bool reported) { struct scatterlist *sg = sgl; +unsigned int i =0 ;/* * Drain the now reported pages back into their respective
@@ -122,7 +123,7 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,/* If the pages were not reported due to error skip flagging */ if (!reported) -continue ; + goto next;/* * If page was not commingled with another page we can
@@ -133,9 +134,12 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,*/ if (PageBuddy (page) &&buddy_order (page) == order) {__SetPageReported (page); -if (page_reporting_host_zeroes_pages ()) +if (page_reporting_host_zeroes_pages () && +test_bit (i, prdev->zeroed_bitmap))__SetPageZeroed (page); } +next: + i++; }while ((sg =sg_next (sg)));/* reinitialize scatterlist now that it is empty */
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 25/37 mm: page_alloc: clear PG_zeroed on buddy merge if not both zero
In reply to Michael S. TsirkinWhen two buddy pages merge in __free_one_page(), preserve PG_zeroed on the merged page only if both buddies have the flag set. Otherwise clear it.
The merged page would inherit PG_zeroed, and a later __GFP_ZERO allocation would skip zeroing stale data in the non-zero half.
--- include/linux/page-flags.h | 1 + mm/page_alloc.c | 15 ++++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-)
include/linux/page-flags.h +1 -0
@@ -680,6 +680,7 @@ FOLIO_FLAG_FALSE(idle)* uses this to skip redundant zeroing in post_alloc_hook(). */ __PAGEFLAG(Zeroed, zeroed, PF_NO_COMPOUND) +CLEARPAGEFLAG(Zeroed, zeroed, PF_NO_COMPOUND)#define __PG_ZEROED (1UL << PG_zeroed)/*
mm/page_alloc.c +14 -1
@@ -940,10 +940,14 @@ static inline void __free_one_page(struct page *page,unsigned long buddy_pfn =0 ;unsigned long combined_pfn;struct page *buddy; +bool buddy_zeroed; +bool page_zeroed;bool to_tail;VM_BUG_ON (!zone_is_initialized (zone)); -VM_BUG_ON_PAGE (page->flags.f &PAGE_FLAGS_CHECK_AT_PREP , page); +/* PG_zeroed (aliased to PG_private) is valid on free-list pages */ +VM_BUG_ON_PAGE (page->flags.f & + (PAGE_FLAGS_CHECK_AT_PREP & ~__PG_ZEROED), page);VM_BUG_ON (migratetype ==-1 );VM_BUG_ON_PAGE (pfn & ((1 << order) -1 ), page);
@@ -978,6 +982,8 @@ static inline void __free_one_page(struct page *page, goto done_merging; } + buddy_zeroed =PageZeroed (buddy); +/* * Our buddy is free or it is CONFIG_DEBUG_PAGEALLOC guard page, * merge with it and move up one order.
@@ -996,10 +1002,17 @@ static inline void __free_one_page(struct page *page,change_pageblock_range (buddy, order, migratetype); } + page_zeroed =PageZeroed (page); +__ClearPageZeroed (page); +__ClearPageZeroed (buddy); + combined_pfn = buddy_pfn & pfn; page = page + (combined_pfn - pfn); pfn = combined_pfn; order++; + +if (page_zeroed && buddy_zeroed) +__SetPageZeroed (page); } done_merging:
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 26/37 mm: page_alloc: preserve PG_zeroed in page_del_and_expand
In reply to Michael S. TsirkinPropagate PG_zeroed through buddy splits in page_del_and_expand() and try_to_claim_block(). When a zeroed high-order page is split to satisfy a smaller allocation, the sub-pages placed back on the free lists keep PG_zeroed.
--- mm/page_alloc.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
mm/page_alloc.c +9 -3
@@ -1715,7 +1715,8 @@ struct page *__pageblock_pfn_to_page(unsigned long start_pfn,* -- nyc */ static inline unsigned int expand (struct zone *zone,struct page *page,int low, -int high,int migratetype,bool reported) +int high,int migratetype,bool reported, +bool zeroed) {unsigned int size =1 << high;unsigned int nr_added =0 ;
@@ -1746,6 +1747,8 @@ static inline unsigned int expand(struct zone *zone, struct page *page, int low,*/ if (reported)__SetPageReported (&page[size]); +if (zeroed) +__SetPageZeroed (&page[size]); }return nr_added;
@@ -1757,10 +1760,12 @@ static __always_inline void page_del_and_expand(struct zone *zone, {int nr_pages =1 << high;bool was_reported =page_reported (page); +bool was_zeroed =PageZeroed (page);__del_page_from_free_list (page, zone, high, migratetype); - nr_pages -=expand (zone, page, low, high, migratetype, was_reported); + nr_pages -=expand (zone, page, low, high, migratetype, was_reported, + was_zeroed);account_freepages (zone, -nr_pages, migratetype); }
@@ -2356,11 +2361,12 @@ try_to_claim_block(struct zone *zone, struct page *page,if (current_order >= pageblock_order) {unsigned int nr_added;bool was_reported =page_reported (page); +bool was_zeroed =PageZeroed (page);del_page_from_free_list (page, zone, current_order, block_type);change_pageblock_range (page, current_order, start_type); nr_added =expand (zone, page, order, current_order, start_type, - was_reported); + was_reported, was_zeroed);account_freepages (zone, nr_added, start_type);return page; }
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 27/37 virtio_balloon: submit reported pages as individual buffers
In reply to Michael S. TsirkinSubmit each reported page as a separate virtqueue buffer instead of one buffer with an sg list of all pages. This avoids indirect descriptor allocation (kmalloc in the reporting path) and gives per-page used length feedback from the device.
On error, the already-queued pages are kicked and drained before the error is returned. The caller (page_reporting_drain) then marks the batch as unreported, which is conservative but correct.
Note: if the virtqueue is broken, wait_event on virtqueue_get_buf hangs. This is a pre-existing issue:
the old single-buffer code had the same hang. EVENT_IDX is not a concern: callbacks were never disabled, so the virtqueue manages used_event automatically.
--- drivers/virtio/virtio_balloon.c | 40 +++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 14 deletions(-)
drivers/virtio/virtio_balloon.c +26 -14
@@ -187,7 +187,9 @@ static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq)sg_init_one (&sg, vb->pfns,sizeof (vb->pfns[0 ]) * vb->num_pfns); -/* We should always be able to add one buffer to an empty queue. */ +/* We made sure the vq is large enough so we should always +* be able to add one buffer to an empty queue. +*/ virtqueue_add_outbuf (vq, &sg,1 , vb,GFP_KERNEL );virtqueue_kick (vq);
@@ -202,25 +204,35 @@ static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_istruct virtio_balloon *vb = container_of(pr_dev_info, struct virtio_balloon, pr_dev_info);struct virtqueue *vq = vb->reporting_vq; -unsigned int unused, err; +unsigned int i, err =0 ;/* We should always be able to add these buffers to an empty queue. */ - err =virtqueue_add_inbuf (vq, sg, nents, vb,GFP_NOWAIT ); +for (i =0 ; i < nents; i++) { +struct scatterlist one; -/* -* In the extremely unlikely case that something has occurred and we -* are able to trigger an error we will simply display a warning -* and exit without actually processing the pages. -*/ -if (WARN_ON_ONCE (err)) -return err; +sg_init_table (&one,1 ); +sg_set_page (&one,sg_page (&sg[i]), sg[i].length, + sg[i].offset); + err =virtqueue_add_inbuf (vq, &one,1 , &sg[i],GFP_NOWAIT ); +if (WARN_ON_ONCE (err)) { + nents = i; +break ; + } + } -virtqueue_kick (vq); +if (nents) { +virtqueue_kick (vq); -/* When host has read buffer, this completes via balloon_ack */ -wait_event (vb->acked,virtqueue_get_buf (vq, &unused)); +/* When host has read buffer, this completes via balloon_ack */ +for (i =0 ; i < nents; i++) { +unsigned int unused; -return 0 ; +wait_event (vb->acked, +virtqueue_get_buf (vq, &unused)); + } + } + +return err; } static void set_page_pfns(struct virtio_balloon *vb,
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 28/37 virtio_balloon: disable indirect descriptors
In reply to Michael S. TsirkinBalloon submits at most 1-2 scatter-gather entries per buffer and virtqueue sizes are 128+, so indirect descriptors are unnecessary.
Disabling them avoids a GFP_KERNEL allocation inside virtqueue_add_sgs when VIRTIO_RING_F_INDIRECT_DESC is negotiated. This allocation could trigger OOM reclaim while balloon_lock is held, leading to a deadlock: OOM notifier -> leak_balloon -> balloon_lock.
With single-buffer submissions (previous patch) and no indirect descriptors, virtqueue_add never allocates memory.
--- drivers/virtio/virtio_balloon.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/virtio/virtio_balloon.c +8 -0
@@ -7,6 +7,7 @@*/ #include <linux/virtio.h> +#include <uapi/linux/virtio_ring.h> #include <linux/virtio_balloon.h> #include <linux/swap.h> #include <linux/workqueue.h>
@@ -1175,6 +1176,13 @@ static int virtballoon_validate(struct virtio_device *vdev)else if (!virtio_has_feature (vdev,VIRTIO_BALLOON_F_PAGE_POISON ))__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_REPORTING ); +/* +* Balloon submits 1-2 sg entries max per buffer, virtqueue +* sizes are 128+. Disable indirect descriptors to avoid +* GFP_KERNEL allocation in virtqueue_add under balloon_lock, +* which could deadlock via OOM -> oom_notify -> leak_balloon. +*/ +__virtio_clear_bit (vdev,VIRTIO_RING_F_INDIRECT_DESC );__virtio_clear_bit (vdev,VIRTIO_F_ACCESS_PLATFORM );return 0 ; }
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 29/37 mm: page_reporting: add flush parameter with page budget
In reply to Michael S. TsirkinAdd a write-only module parameter 'flush' that triggers immediate page reporting. The value specifies a page budget: at least this many pages (at page_reporting_order) will be reported, or all unreported pages if fewer remain. The actual number reported may exceed the budget since each reporting pass processes a full cycle across all zones.
This is helpful when there is a lot of memory freed quickly, and a single cycle may not process all free pages due to internal budget limits.
echo 512 > /sys/module/page_reporting/parameters/flush
Note: the set callback runs under kernel param_lock, so writing this parameter blocks other built-in parameter writes until the flush loop completes. This is acceptable for a privileged debug/test parameter.
--- mm/page_reporting.c | 54 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+)
mm/page_reporting.c +54 -0
@@ -358,6 +358,60 @@ static void page_reporting_process(struct work_struct *work)static DEFINE_MUTEX (page_reporting_mutex);DEFINE_STATIC_KEY_FALSE (page_reporting_enabled); +static int page_reporting_flush_set (const char *val, +const struct kernel_param *kp) +{ +struct page_reporting_dev_info *prdev; +unsigned int budget; +int err; + + err =kstrtouint (val,0 , &budget); +if (err) +return err; +if (!budget) +return 0 ; + +mutex_lock (&page_reporting_mutex); + prdev =rcu_dereference_protected (pr_dev_info, +lockdep_is_held (&page_reporting_mutex)); +if (prdev) { +unsigned int reported; +bool interrupted = false; + +for (reported =0 ; reported < budget; + reported +=min (prdev->capacity, budget - reported)) { +/* +* First flush completes any previously scheduled +* reporting work. Then request a new reporting +* cycle and flush again to execute it. +*/ +flush_delayed_work (&prdev->work); +__page_reporting_request (prdev); +flush_delayed_work (&prdev->work); +if (atomic_read (&prdev->state) ==PAGE_REPORTING_IDLE ) +break ; +if (signal_pending (current)) { + interrupted = true; +break ; + } + } +if (interrupted) { +mutex_unlock (&page_reporting_mutex); +return -EINTR ; + } + } +mutex_unlock (&page_reporting_mutex); +return 0 ; +} + +static const struct kernel_param_ops flush_ops = { + .set = page_reporting_flush_set, + .get = param_get_uint, +}; +static unsigned int page_reporting_flush; +module_param_cb (flush, &flush_ops, &page_reporting_flush,0200 ); +MODULE_PARM_DESC (flush,"Report at least N pages at page_reporting_order, or until all reported" ); +int page_reporting_register (struct page_reporting_dev_info *prdev) {int err =0 ;
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 30/37 mm: page_alloc: propagate PG_zeroed in split_large_buddy
In reply to Michael S. TsirkinWhen splitting a large buddy page, propagate the PG_zeroed flag to each sub-page before freeing it. __free_pages_prepare clears all flags (including PG_zeroed), so the flag must be re-set on each fragment after the split. This ensures that the buddy merge logic can see PG_zeroed on pages that were part of a larger zeroed block.
--- mm/page_alloc.c | 3 +++ 1 file changed, 3 insertions(+)
mm/page_alloc.c +3 -0
@@ -1524,6 +1524,7 @@ static void split_large_buddy(struct zone *zone, struct page *page, {unsigned long end = pfn + (1 << order);bool reported =PageReported (page); +bool zeroed =PageZeroed (page);VM_WARN_ON_ONCE (!IS_ALIGNED (pfn,1 << order));/* Caller removed page from freelist, buddy info cleared! */
@@ -1537,6 +1538,8 @@ static void split_large_buddy(struct zone *zone, struct page *page,if (reported)__SetPageReported (page); +if (zeroed) +__SetPageZeroed (page);__free_one_page (page, pfn, zone, order, mt, fpi); pfn +=1 << order;if (pfn == end)
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 31/37 virtio_balloon: skip zeroing for host-zeroed reported pages
In reply to Michael S. TsirkinImplement VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED (per virtio spec proposal): when negotiated, the device initializes reported pages (zeros, or poison_val if PAGE_POISON).
Check per-page used length returned by the device to determine which reported pages were zeroed. If used_len matches the page size, the device successfully initialized the page (e.g. via MADV_DONTNEED), and we set the corresponding zeroed_bitmap bit.
Gate host_zeroes_pages on the feature bit and page content:
when PAGE_POISON is negotiated with non-zero poison_val, the device fills with poison not zeros, so pages are not zeroed.
Clear the feature in validate() if REPORTING is not present or if PAGE_POISON is active with non-zero poison_val.
See the virtio spec change:
https://github.com/oasis-tcs/virtio-spec/issues/244
--- drivers/virtio/virtio_balloon.c | 22 ++++++++++++++++++++-- include/uapi/linux/virtio_balloon.h | 1 + 2 files changed, 21 insertions(+), 2 deletions(-)
drivers/virtio/virtio_balloon.c +20 -2
@@ -207,6 +207,8 @@ static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_istruct virtqueue *vq = vb->reporting_vq;unsigned int i, err =0 ; +bitmap_zero (pr_dev_info->zeroed_bitmap, nents); +/* We should always be able to add these buffers to an empty queue. */ for (i =0 ; i < nents; i++) {struct scatterlist one;
@@ -226,10 +228,14 @@ static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_i/* When host has read buffer, this completes via balloon_ack */ for (i =0 ; i < nents; i++) { -unsigned int unused; +struct scatterlist *entry; +unsigned int used_len;wait_event (vb->acked, -virtqueue_get_buf (vq, &unused)); + (entry =virtqueue_get_buf (vq, &used_len))); +if (used_len == entry->length) +set_bit (entry - sg, + pr_dev_info->zeroed_bitmap); } }
@@ -1051,6 +1057,9 @@ static int virtballoon_probe(struct virtio_device *vdev)#endif vb->pr_dev_info.capacity = capacity; + vb->pr_dev_info.host_zeroes_pages = +virtio_has_feature (vdev, +VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED ); err =page_reporting_register (&vb->pr_dev_info);if (err) goto out_unregister_oom;
@@ -1176,6 +1185,14 @@ static int virtballoon_validate(struct virtio_device *vdev)else if (!virtio_has_feature (vdev,VIRTIO_BALLOON_F_PAGE_POISON ))__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_REPORTING ); +if (!virtio_has_feature (vdev,VIRTIO_BALLOON_F_REPORTING )) +__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED ); + +/* Device fills with poison_val, not zeros; disable zeroed hint */ +if (virtio_has_feature (vdev,VIRTIO_BALLOON_F_PAGE_POISON ) && + !want_init_on_free ()) +__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED ); +/* * Balloon submits 1-2 sg entries max per buffer, virtqueue * sizes are 128+. Disable indirect descriptors to avoid
@@ -1194,6 +1211,7 @@ static unsigned int features[] = {VIRTIO_BALLOON_F_FREE_PAGE_HINT ,VIRTIO_BALLOON_F_PAGE_POISON ,VIRTIO_BALLOON_F_REPORTING , + VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED, }; static struct virtio_driver virtio_balloon_driver = {
include/uapi/linux/virtio_balloon.h +1 -0
@@ -37,6 +37,7 @@#define VIRTIO_BALLOON_F_FREE_PAGE_HINT 3/* VQ to report free pages */ #define VIRTIO_BALLOON_F_PAGE_POISON 4/* Guest is using page poisoning */ #define VIRTIO_BALLOON_F_REPORTING 5/* Page reporting virtqueue */ +#define VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED 6/* Device initializes reported pages */ /* Size of a PFN in the balloon interface. */ #define VIRTIO_BALLOON_PFN_SHIFT 12
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 32/37 virtio_balloon: disable reporting zeroed optimization for confidential guests
In reply to Michael S. TsirkinIn confidential computing environments (TDX, SEV-SNP), the host is untrusted and may lie about zeroing reported pages. Clear DEVICE_INIT_REPORTED in validate() so the guest does not skip re-zeroing based on hints from an untrusted device.
Note: currently REPORTING remains enabled and VIRTIO_F_ACCESS_PLATFORM is cleared in CC environments. This is known to leak free page physical addresses to the host. Whether that, or ballooning in general, is a security concern in CC is up to the user. This patch only disables our new zeroed-page hints where the host is untrusted.
--- drivers/virtio/virtio_balloon.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/virtio/virtio_balloon.c +3 -0
@@ -19,6 +19,7 @@#include <linux/wait.h> #include <linux/mm.h> #include <linux/page_reporting.h> +#include <linux/cc_platform.h> /* * Balloon device works in 4K page units. So each page is pointed to by
@@ -1193,6 +1194,8 @@ static int virtballoon_validate(struct virtio_device *vdev) !want_init_on_free())__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED ); +if (cc_platform_has (CC_ATTR_GUEST_MEM_ENCRYPT )) +__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED );/* * Balloon submits 1-2 sg entries max per buffer, virtqueue * sizes are 128+. Disable indirect descriptors to avoid
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 33/37 mm: add free_frozen_pages_zeroed
In reply to Michael S. TsirkinAdd free_frozen_pages_zeroed(page, order) to free a frozen page while marking it as zeroed, so the next allocation can skip redundant zeroing.
An FPI_ZEROED internal flag carries the hint through the free path. PageZeroed is set after __free_pages_prepare() clears all flags, so the hint survives on the free list.
--- include/linux/gfp.h | 1 + mm/internal.h | 1 - mm/page_alloc.c | 29 ++++++++++++++++++++++++++++- 3 files changed, 29 insertions(+), 2 deletions(-)
include/linux/gfp.h +1 -0
@@ -384,6 +384,7 @@ __meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_masextern void __free_pages (struct page *page,unsigned int order);extern void free_pages_nolock (struct page *page,unsigned int order);extern void free_pages (unsigned long addr,unsigned int order); +void free_frozen_pages_zeroed (struct page *page,unsigned int order);#define __free_page (page) __free_pages((page), 0)#define free_page (addr) free_pages((addr), 0)
mm/internal.h +0 -1
@@ -938,7 +938,6 @@ struct page *__alloc_frozen_pages_noprof(gfp_t, unsigned int order, int nid,#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))void free_frozen_pages (struct page *page,unsigned int order); -void free_frozen_pages_zeroed (struct page *page,unsigned int order);void free_unref_folios (struct folio_batch *fbatch);#ifdef CONFIG_NUMA
mm/page_alloc.c +28 -1
@@ -90,6 +90,13 @@ typedef int __bitwise fpi_t;/* Free the page without taking locks. Rely on trylock only. */ #define FPI_TRYLOCK ((__force fpi_t)BIT(2)) +/* +* The page contents are known to be zero (e.g., the host zeroed them +* during balloon deflate). Set PageZeroed after free so the next +* allocation can skip redundant zeroing. +*/ +#define FPI_ZEROED ((__force fpi_t)BIT(3)) +/* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX (pcp_batch_high_lock);#define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8)
@@ -1599,8 +1606,19 @@ static void __free_pages_ok(struct page *page, unsigned int order,unsigned long pfn =page_to_pfn (page);struct zone *zone =page_zone (page); -if (__free_pages_prepare (page, order, fpi_flags)) +if (__free_pages_prepare (page, order, fpi_flags)) { +/* +* Only mark as zeroed if the page is actually still zero. +* kernel_poison_pages() overwrites with PAGE_POISON (0xAA) +* when CONFIG_PAGE_POISONING is enabled. With init_on_free, +* kernel_init_pages() re-zeroes after poisoning, so the +* page is still zero. +*/ +if ((fpi_flags &FPI_ZEROED ) && + (!page_poisoning_enabled_static () ||want_init_on_free ())) +__SetPageZeroed (page);free_one_page (zone, page, pfn, order, fpi_flags); + } } void __meminit __free_pages_core(struct page *page, unsigned int order,
@@ -3021,6 +3039,9 @@ static void __free_frozen_pages(struct page *page, unsigned int order,if (!__free_pages_prepare (page, order, fpi_flags))return ; +if (fpi_flags &FPI_ZEROED ) +__SetPageZeroed (page); +/* * We only track unmovable, reclaimable and movable on pcp lists. * Place ISOLATE pages on the isolated list because they are being
@@ -3059,6 +3080,12 @@ void free_frozen_pages(struct page *page, unsigned int order)__free_frozen_pages (page, order,FPI_NONE ); } +void free_frozen_pages_zeroed (struct page *page,unsigned int order) +{ +__free_frozen_pages (page, order,FPI_ZEROED ); +} +EXPORT_SYMBOL (free_frozen_pages_zeroed); +void free_frozen_pages_nolock (struct page *page,unsigned int order) {__free_frozen_pages (page, order,FPI_TRYLOCK );
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 34/37 mm: page_alloc: skip kernel_init_pages for FPI_ZEROED when safe
In reply to Michael S. TsirkinIn __free_pages_prepare(), when FPI_ZEROED is set the page is already known to be zero. We can skip kernel_init_pages() if page poisoning is not enabled (because poison would overwrite the zeroes).
This avoids redundant zeroing work when freeing pages that are already known to contain all zeros.
--- mm/page_alloc.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
mm/page_alloc.c +8 -1
@@ -1447,7 +1447,14 @@ __always_inline bool __free_pages_prepare(struct page *page,if (kasan_has_integrated_init ()) init = false; } -if (init) +/* +* Skip redundant zeroing when the page is already known-zero +* (FPI_ZEROED) and page poisoning did not overwrite it. +* When page_poisoning is enabled, kernel_poison_pages above +* wrote PAGE_POISON (0xAA), so we must re-zero. +*/ +if (init && !((fpi_flags &FPI_ZEROED ) && + !page_poisoning_enabled_static ()))kernel_init_pages (page,1 << order);/*
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 35/37 mm: add put_page_zeroed and folio_put_zeroed
In reply to Michael S. TsirkinAdd put_page_zeroed() / folio_put_zeroed() for callers that hold a reference to a page known to be zeroed.
If this drops the last reference, the zeroed hint is propagated to the buddy allocator. If someone else still holds a reference, the hint is simply lost - this is best-effort.
This is useful for balloon drivers during deflation: the host has already zeroed the pages, and the balloon is typically the sole owner. But if the page happens to be shared, silently dropping the hint is safe and avoids the need for callers to check the refcount.
--- include/linux/mm.h | 13 +++++++++++++ mm/swap.c | 20 ++++++++++++++++++-- 2 files changed, 31 insertions(+), 2 deletions(-)
include/linux/mm.h +13 -0
@@ -1913,6 +1913,7 @@ static inline struct folio *virt_to_folio(const void *x) }void __folio_put (struct folio *folio); +void __folio_put_zeroed (struct folio *folio);void split_page (struct page *page,unsigned int order);void folio_copy (struct folio *dst,struct folio *src);
@@ -2090,6 +2091,18 @@ static inline void folio_put(struct folio *folio)__folio_put (folio); } +/* Caller must be sole owner to guarantee page is still zero */ +static inline void folio_put_zeroed (struct folio *folio) +{ +if (folio_put_testzero (folio)) +__folio_put_zeroed (folio); +} + +static inline void put_page_zeroed (struct page *page) +{ +folio_put_zeroed (page_folio (page)); +} +/** * folio_put_refs - Reduce the reference count on a folio. * @folio: The folio.
mm/swap.c +18 -2
@@ -94,13 +94,15 @@ static void page_cache_release(struct folio *folio)lruvec_unlock_irqrestore (lruvec, flags); } -void __folio_put (struct folio *folio) +static void ___folio_put (struct folio *folio,bool zeroed) { +/* zeroed hint ignored for now, no current user */ if (unlikely (folio_is_zone_device (folio))) {free_zone_device_folio (folio);return ; } +/* zeroed hint ignored for now, no current user */ if (folio_test_hugetlb (folio)) {free_huge_folio (folio);return ;
@@ -109,10 +111,24 @@ void __folio_put(struct folio *folio)page_cache_release (folio);folio_unqueue_deferred_split (folio);mem_cgroup_uncharge (folio); -free_frozen_pages (&folio->page,folio_order (folio)); +if (zeroed) +free_frozen_pages_zeroed (&folio->page,folio_order (folio)); +else +free_frozen_pages (&folio->page,folio_order (folio)); +} + +void __folio_put (struct folio *folio) +{ +___folio_put (folio, false); }EXPORT_SYMBOL (__folio_put); +void __folio_put_zeroed (struct folio *folio) +{ +___folio_put (folio, true); +} +EXPORT_SYMBOL (__folio_put_zeroed); +typedef void (*move_fn_t )(struct lruvec *lruvec,struct folio *folio);static void lru_add (struct lruvec *lruvec,struct folio *folio)
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 36/37 virtio_balloon: implement VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE
In reply to Michael S. TsirkinWhen the device offers DEVICE_INIT_ON_INFLATE (bit 7), the device initializes inflated pages and returns a per-page bitmap indicating which pages were successfully initialized.
The driver appends a device-writable bitmap buffer to each inflate descriptor chain via virtqueue_add_sgs. After the host acknowledges, the driver checks bitmap bits (bounded by used_len) and marks pages with SetPageZeroed.
tell_host() returns used_len from virtqueue_get_buf(). Bitmap reads are bounded: fill_balloon() and virtballoon_migratepage() only trust bits within the used_len range.
On deflate, release_pages_balloon checks PageZeroed per page and uses put_page_zeroed for pages the host initialized, propagating the zeroed hint to the buddy allocator.
If inflate_vq has fewer than 2 descriptors, probe fails with -ENOSPC. If PAGE_POISON is negotiated with non-zero poison_val, the feature is cleared in validate().
See the virtio spec change:
https://lore.kernel.org/all/9c69b992c3dd83dfef3db92cd86b2fd8a0730d48.1777731396.git.mst@redhat.com
--- drivers/virtio/virtio_balloon.c | 107 ++++++++++++++++++++++++---- include/uapi/linux/virtio_balloon.h | 1 + 2 files changed, 96 insertions(+), 12 deletions(-)
drivers/virtio/virtio_balloon.c +95 -12
@@ -20,6 +20,7 @@#include <linux/mm.h> #include <linux/page_reporting.h> #include <linux/cc_platform.h> +#include <asm-generic/bitops/le.h> /* * Balloon device works in 4K page units. So each page is pointed to by
@@ -122,6 +123,13 @@ struct virtio_balloon {struct virtqueue *reporting_vq;struct page_reporting_dev_info pr_dev_info; +/* No DMA alignment needed: ACCESS_PLATFORM is cleared, +* so virtio bypasses the DMA API. If this ever changes, +* add ____dma_from_device_aligned here. +*/ +/* Bitmap returned by host for DEVICE_INIT_ON_INFLATE */ +DECLARE_BITMAP (inflate_bitmap,VIRTIO_BALLOON_ARRAY_PFNS_MAX ); +/* State for keeping the wakeup_source active while adjusting the balloon */ spinlock_t wakeup_lock;bool processing_wakeup_event;
@@ -182,22 +190,33 @@ static void balloon_ack(struct virtqueue *vq)wake_up (&vb->acked); } -static void tell_host (struct virtio_balloon *vb,struct virtqueue *vq) +static unsigned int tell_host (struct virtio_balloon *vb,struct virtqueue *vq) { -struct scatterlist sg; +struct scatterlist sg_out, sg_in; +struct scatterlist *sgs[] = { &sg_out, &sg_in };unsigned int len; -sg_init_one (&sg, vb->pfns,sizeof (vb->pfns[0 ]) * vb->num_pfns); +sg_init_one (&sg_out, vb->pfns,sizeof (vb->pfns[0 ]) * vb->num_pfns);/* We made sure the vq is large enough so we should always * be able to add one buffer to an empty queue. */ -virtqueue_add_outbuf (vq, &sg,1 , vb,GFP_KERNEL ); +if (vq == vb->inflate_vq && +virtio_has_feature (vb->vdev, +VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE )) { +unsigned int bitmap_bytes; + + bitmap_bytes =DIV_ROUND_UP (vb->num_pfns,8 ); +bitmap_zero (vb->inflate_bitmap, vb->num_pfns); +sg_init_one (&sg_in, vb->inflate_bitmap, bitmap_bytes); +virtqueue_add_sgs (vq, sgs,1 ,1 , vb,GFP_KERNEL ); + }else { +virtqueue_add_outbuf (vq, &sg_out,1 , vb,GFP_KERNEL ); + }virtqueue_kick (vq); -/* When host has read buffer, this completes via balloon_ack */ wait_event (vb->acked,virtqueue_get_buf (vq, &len)); - +return len; } static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_info,
@@ -300,8 +319,37 @@ static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num) num_allocated_pages = vb->num_pfns;/* Did we get any? */ -if (vb->num_pfns !=0 ) -tell_host (vb, vb->inflate_vq); +if (vb->num_pfns !=0 ) { +unsigned int used_len =tell_host (vb, vb->inflate_vq); + +if (virtio_has_feature (vb->vdev, +VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE )) { +unsigned int i; +unsigned int valid_bits = used_len *8 ; + +for (i =0 ; i < vb->num_pfns; + i +=VIRTIO_BALLOON_PAGES_PER_PAGE ) { +unsigned int pfn, j; +bool zeroed = true; + +if (i +VIRTIO_BALLOON_PAGES_PER_PAGE > valid_bits) +break ; +for (j =0 ; j <VIRTIO_BALLOON_PAGES_PER_PAGE ; j++) { +if (!test_bit_le (i + j, vb->inflate_bitmap)) { + zeroed = false; +break ; + } + } +if (zeroed) { + pfn =virtio32_to_cpu (vb->vdev, + vb->pfns[i]); +__SetPageZeroed (pfn_to_page (pfn >> + (PAGE_SHIFT - +VIRTIO_BALLOON_PFN_SHIFT ))); + } + } + } + }mutex_unlock (&vb->balloon_lock);return num_allocated_pages;
@@ -314,7 +362,12 @@ static void release_pages_balloon(struct virtio_balloon *vb,list_for_each_entry_safe (page, next, pages, lru) {list_del (&page->lru); -put_page (page);/* balloon reference */ +if (PageZeroed (page)) { +__ClearPageZeroed (page); +put_page_zeroed (page); + }else { +put_page (page); + } } }
@@ -861,8 +914,27 @@ static int virtballoon_migratepage(struct balloon_dev_info *vb_dev_info,/* balloon's page migration 1st step -- inflate "newpage" */ vb->num_pfns =VIRTIO_BALLOON_PAGES_PER_PAGE ;set_page_pfns (vb, vb->pfns, newpage); -tell_host (vb, vb->inflate_vq); + { +unsigned int used_len =tell_host (vb, vb->inflate_vq); +if (virtio_has_feature (vb->vdev, +VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE ) && + used_len >=DIV_ROUND_UP (VIRTIO_BALLOON_PAGES_PER_PAGE ,8 )) { +unsigned int j; +bool zeroed = true; + +for (j =0 ; j <VIRTIO_BALLOON_PAGES_PER_PAGE ; j++) { +if (!test_bit_le (j, vb->inflate_bitmap)) { + zeroed = false; +break ; + } + } +if (zeroed) +__SetPageZeroed (newpage); + } + } + +__ClearPageZeroed (page);/* balloon's page migration 2nd step -- deflate "page" */ vb->num_pfns =VIRTIO_BALLOON_PAGES_PER_PAGE ;set_page_pfns (vb, vb->pfns, page);
@@ -966,6 +1038,12 @@ static int virtballoon_probe(struct virtio_device *vdev)if (err) goto out_free_vb; +if (virtio_has_feature (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE ) && +virtqueue_get_vring_size (vb->inflate_vq) <2 ) { + err = -ENOSPC ; + goto out_del_vqs; + } +if (!virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_DEFLATE_ON_OOM )) vb->vb_dev_info.adjust_managed_page_count = true;#ifdef CONFIG_BALLOON_MIGRATION
@@ -1191,11 +1269,15 @@ static int virtballoon_validate(struct virtio_device *vdev)/* Device fills with poison_val, not zeros; disable zeroed hint */ if (virtio_has_feature (vdev,VIRTIO_BALLOON_F_PAGE_POISON ) && - !want_init_on_free ()) + !want_init_on_free ()) {__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED ); +__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE ); + } -if (cc_platform_has (CC_ATTR_GUEST_MEM_ENCRYPT )) +if (cc_platform_has (CC_ATTR_GUEST_MEM_ENCRYPT )) {__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED ); +__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE ); + }/* * Balloon submits 1-2 sg entries max per buffer, virtqueue * sizes are 128+. Disable indirect descriptors to avoid
@@ -1215,6 +1297,7 @@ static unsigned int features[] = {VIRTIO_BALLOON_F_PAGE_POISON ,VIRTIO_BALLOON_F_REPORTING ,VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED , + VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE, }; static struct virtio_driver virtio_balloon_driver = {
include/uapi/linux/virtio_balloon.h +1 -0
@@ -38,6 +38,7 @@#define VIRTIO_BALLOON_F_PAGE_POISON 4/* Guest is using page poisoning */ #define VIRTIO_BALLOON_F_REPORTING 5/* Page reporting virtqueue */ #define VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED 6/* Device initializes reported pages */ +#define VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE 7/* Device initializes pages on inflate */ /* Size of a PFN in the balloon interface. */ #define VIRTIO_BALLOON_PFN_SHIFT 12
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v8 37/37 mm: balloon: use put_page_zeroed for zeroed balloon pages
In reply to Michael S. TsirkinWhen a balloon page marked PageZeroed is freed during migration, use put_page_zeroed() to propagate the zeroed hint to the buddy allocator. Previously the hint was silently lost via plain put_page().
--- mm/balloon.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
mm/balloon.c +9 -1
@@ -324,7 +324,15 @@ static int balloon_page_migrate(struct page *newpage, struct page *page,balloon_page_finalize (page);spin_unlock_irqrestore (&balloon_pages_lock, flags); -put_page (page); +if (PageZeroed (page)) { +/* Atomic to serialize with memory_failure's +* TestSetPageHWPoison; not under zone->lock here. +*/ +ClearPageZeroed (page); +put_page_zeroed (page); + }else { +put_page (page); + }return 0 ; }
Signature · 1 line
MST
Michael S. Tsirkin
maintainerQuoted 5 lines · Michael S. Tsirkin
When a guest reports free pages to the hypervisor via virtio-balloon's free page reporting, the host typically zeros those pages when reclaiming their backing memory (e.g., via
MADV_DONTNEEDon anonymous mappings). When the guest later reallocates those pages, the kernel zeros them again, redundantly.
FYI Sashiko reported a couple of issues. Mostly false positives but I tweaked commit log to make it clearer. But also a couple of real issues. So far I have this diff on top:
--->
Changes from v8 to v9 candidate (code only):
1. mm/mempolicy.c (patch 1: mm: mempolicy: fix interleave index calculation) Combine vm_pgoff and VMA offset into a single expression before shifting, fixing carry loss for file-backed VMAs with unaligned vm_pgoff.
2. mm/memory-failure.c (patch 2: mm: memory-failure: serialize TestSetPageHWPoison with zone->lock) Wrap ClearPageHWPoison in retry path with zone->lock too.
3. mm/huge_memory.c (patch 19: mm: use __GFP_ZERO in vma_alloc_anon_folio_pmd) Fix stale comment: "folio_zero_user writes" -> "page zeroing".
4. mm/page_reporting.c (patch 4: mm: page_reporting: allow driver to set batch capacity) Drop rounddown_pow_of_two: compiler cannot optimize division
by runtime variable anyway, and rounding halves batch size for non-power-of-2 capacity values.
Commit log improvements (no code change):
- Patch 1: rewritten to explain new single-expression formula (fixes patch 1)
- Patch 2: mention ClearPageHWPoison in retry path (fixes patch 2)
- Patch 4: drop "rounded down to a power of 2" (fixes patch 4)
- Patch 10: "stub" wording simplified (fixes patch 10)
- Patch 22: note PG_zeroed hint loss is harmless (fixes patch 22)
- Patch 23: add PowerPC note (fixes patch 23)
- Patch 28: explain DEVICE_INIT_ON_INFLATE is a follow-up (fixes patch 28)
- Patch 29: explain flush over-reporting is by design (fixes patch 29)
- Patch 33: note __SetPageZeroed is safe on frozen pages (fixes patch 33)
- Patch 36: note __SetPageZeroed is safe on balloon-owned pages (fixes patch 36)
---
mm/huge_memory.c +1 -1
@@ -1358,7 +1358,7 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma,/* * The memory barrier inside __folio_mark_uptodate makes sure that -* folio_zero_user writes become visible before the set_pmd_at() +* page zeroing becomes visible before the set_pmd_at() * write. */ __folio_mark_uptodate (folio);
mm/memory-failure.c +2 -0
@@ -2426,7 +2426,9 @@ int memory_failure(unsigned long pfn, int flags) }else {/* We lost the race, try again */ if (retry) { +spin_lock_irqsave (&zone->lock, mf_flags);ClearPageHWPoison (p); +spin_unlock_irqrestore (&zone->lock, mf_flags); retry = false; goto try_again; }
mm/mempolicy.c +3 -3
@@ -2048,9 +2048,9 @@ struct mempolicy *get_vma_policy(struct vm_area_struct *vma, pol =get_task_policy (current);if (pol->mode ==MPOL_INTERLEAVE || pol->mode ==MPOL_WEIGHTED_INTERLEAVE ) { - *ilx += vma->vm_pgoff >> order; - *ilx += (addr >> (PAGE_SHIFT + order)) - - (vma->vm_start >> (PAGE_SHIFT + order)); + *ilx += (vma->vm_pgoff + + (addr >>PAGE_SHIFT ) - + (vma->vm_start >>PAGE_SHIFT )) >> order; }return pol; }
mm/page_reporting.c +0 -3
@@ -443,9 +443,6 @@ int page_reporting_register(struct page_reporting_dev_info *prdev)if (!prdev->capacity || prdev->capacity >PAGE_REPORTING_CAPACITY ) prdev->capacity =PAGE_REPORTING_CAPACITY ; -/* Power of 2 so division by capacity in the budget calc is cheap */ - prdev->capacity =rounddown_pow_of_two (prdev->capacity); -/* initialize state and work structures */ atomic_set (&prdev->state,PAGE_REPORTING_IDLE );INIT_DELAYED_WORK (&prdev->work, &page_reporting_process);
Miaohe Lin
maintainerQuoted 15 lines · Michael S. Tsirkin
Quoted 5 lines · Michael S. Tsirkin
When a guest reports free pages to the hypervisor via virtio-balloon's free page reporting, the host typically zeros those pages when reclaiming their backing memory (e.g., via
MADV_DONTNEEDon anonymous mappings). When the guest later reallocates those pages, the kernel zeros them again, redundantly.FYI Sashiko reported a couple of issues. Mostly false positives but I tweaked commit log to make it clearer. But also a couple of real issues. So far I have this diff on top:
--->
Changes from v8 to v9 candidate (code only):
Quoted 7 lines
1.
mm/mempolicy.c(patch 1: mm: mempolicy: fix interleave index calculation) Combinevm_pgoffand VMA offset into a single expression before shifting, fixing carry loss for file-backed VMAs with unalignedvm_pgoff.2.
mm/memory-failure.c(patch 2: mm: memory-failure: serialize TestSetPageHWPoison withzone->lock) Wrap ClearPageHWPoison in retry path withzone->locktoo.
It seems TestClearPageHWPoison and SetPageHWPoison are missing. Should we serialize them with zone->lock too?
Thanks.
.
Michael S. Tsirkin
maintainerQuoted 21 lines · Miaohe Lin
Quoted 20 lines · Michael S. Tsirkin
Quoted 5 lines · Michael S. Tsirkin
When a guest reports free pages to the hypervisor via virtio-balloon's free page reporting, the host typically zeros those pages when reclaiming their backing memory (e.g., via
MADV_DONTNEEDon anonymous mappings). When the guest later reallocates those pages, the kernel zeros them again, redundantly.FYI Sashiko reported a couple of issues. Mostly false positives but I tweaked commit log to make it clearer. But also a couple of real issues. So far I have this diff on top:
--->
Changes from v8 to v9 candidate (code only):
1.
mm/mempolicy.c(patch 1: mm: mempolicy: fix interleave index calculation) Combinevm_pgoffand VMA offset into a single expression before shifting, fixing carry loss for file-backed VMAs with unalignedvm_pgoff.
Quoted 7 lines
Quoted 2 lines
2.
mm/memory-failure.c(patch 2: mm: memory-failure: serialize TestSetPageHWPoison withzone->lock) Wrap ClearPageHWPoison in retry path withzone->locktoo.It seems TestClearPageHWPoison and SetPageHWPoison are missing. Should we serialize them with
zone->locktoo?Thanks.
.
I don't think so. As far as I could tell, none of these are called on pages that could be in the buddy. And all the non atomic page flags games we play are only on pages in the buddy.
Signature · 1 line
MST
Miaohe Lin
maintainerQuoted 25 lines · Michael S. Tsirkin
Quoted 24 lines · Miaohe Lin
Quoted 22 lines · Michael S. Tsirkin
Quoted 5 lines · Michael S. Tsirkin
When a guest reports free pages to the hypervisor via virtio-balloon's free page reporting, the host typically zeros those pages when reclaiming their backing memory (e.g., via
MADV_DONTNEEDon anonymous mappings). When the guest later reallocates those pages, the kernel zeros them again, redundantly.FYI Sashiko reported a couple of issues. Mostly false positives but I tweaked commit log to make it clearer. But also a couple of real issues. So far I have this diff on top:
--->
Changes from v8 to v9 candidate (code only):
1.
mm/mempolicy.c(patch 1: mm: mempolicy: fix interleave index calculation) Combinevm_pgoffand VMA offset into a single expression before shifting, fixing carry loss for file-backed VMAs with unalignedvm_pgoff.2.
mm/memory-failure.c(patch 2: mm: memory-failure: serialize TestSetPageHWPoison withzone->lock) Wrap ClearPageHWPoison in retry path withzone->locktoo.
Quoted 8 lines
Quoted 4 lines
It seems TestClearPageHWPoison and SetPageHWPoison are missing. Should we serialize them with
zone->locktoo?Thanks.
.I don't think so. As far as I could tell, none of these are called on pages that could be in the buddy. And all the non atomic page flags games we play are only on pages in the buddy.
I scanned the code and it seems no one breaks the game. So I agree with you.
Thanks.
.
Michael S. Tsirkin
maintainerv9 cover mm/virtio: skip redundant zeroing of host-zeroed pages
In reply to Michael S. TsirkinWhen a guest reports free pages to the hypervisor via virtio-balloon's free page reporting, the host typically zeros those pages when reclaiming their backing memory (e.g., via MADV_DONTNEED on anonymous mappings). When the guest later reallocates those pages, the kernel zeros them again, redundantly.
Further, on architectures with aliasing caches, upstream with init_on_alloc double-zeros user pages: once via kernel_init_pages() in post_alloc_hook, and again via clear_user_highpage() at the callsite (because user_alloc_needs_zeroing() returns true). This series eliminates that double-zeroing by moving the zeroing into the post_alloc_hook + propagating the "host
already zeroed this page" information through the buddy allocator.
For page reporting, VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED (bit 6) is used. For the inflate/deflate path, VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE (bit 7) is used.
Virtio spec: https://lore.kernel.org/all/[email protected]
Based on v7.1-rc2. When applying on mm-unstable, two conflicts are expected:
- kernel_init_pages() was renamed to clear_highpages_kasan_tagged() in mm-unstable. Use clear_highpages_kasan_tagged() in the post_alloc_hook else branch.
- FPI_PREPARED uses BIT(3) in mm-unstable. Bump FPI_ZEROED to BIT(4).
Build-tested on mm-unstable at e9dd96806dbc:
https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git zero-mm-unstable
Patches 1-5: fixes/cleanups, dependencies of the zeroing patches. Patches 6-9: thread user_addr through page allocator, contig API,
and gigantic hugetlb allocation.
Patches 10-23: mm rework + cleanups + init_on_alloc double-zeroing
fix + aliasing guard.
Patches 24-32: page reporting zeroing (DEVICE_INIT_REPORTED),
disable indirect descriptors.
Patches 33-37: inflate/deflate zeroing (DEVICE_INIT_ON_INFLATE),
FPI_ZEROED optimization.
-------
Performance with THP enabled on a 2GB VM, 1 vCPU, allocating 256MB of anonymous pages:
metric baseline optimized delta task-clock 232 +- 20 ms 51 +- 26 ms -78% cache-misses 1.20M +- 248K 288K +- 102K -76% instructions 16.3M +- 1.2M 13.8M +- 1.0M -15%
With hugetlb surplus pages:
metric baseline optimized delta task-clock 219 +- 23 ms 65 +- 34 ms -70% cache-misses 1.17M +- 391K 263K +- 36K -78% instructions 17.9M +- 1.2M 15.1M +- 724K -16%
Two flags track known-zero pages:
PG_zeroed (aliased to PG_private) marks buddy allocator pages that are known to contain all zeros, either because the host zeroed them during page reporting, or because they were freed via the balloon deflate path. It lives on free-list pages and is consumed by post_alloc_hook() on allocation.
HPG_zeroed (stored in hugetlb folio->private bits) serves the same purpose for hugetlb pool pages, which are kept in a pool and may be zeroed long after buddy allocation, so PG_zeroed (consumed at allocation time) cannot track their state.
PG_zeroed lifecycle:
Sets PG_zeroed:
- page_reporting_drain: on reported pages when host zeroes them
- __free_pages_ok / __free_frozen_pages: when FPI_ZEROED is set (balloon deflate path)
- buddy merge: on merged page if both buddies were zeroed
- expand(): propagate to split-off buddy sub-pages
Clears PG_zeroed:
- __free_pages_prepare: clears all PAGE_FLAGS_CHECK_AT_PREP flags (PG_zeroed included), preventing PG_private aliasing leaks
- rmqueue_buddy / __rmqueue_pcplist: read-then-clear, passes
zeroed hint to prep_new_page -> post_alloc_hook
- __isolate_free_page: clear (compaction/page_reporting isolation)
- compaction, alloc_contig, split_free_frozen: clear before use
- buddy merge: clear both pages before merge, then conditionally re-set on merged head if both were zeroed
HPG_zeroed lifecycle (hugetlb pool pages, stored in folio->private):
Sets HPG_zeroed:
- alloc_surplus_hugetlb_folio: after buddy allocation with __GFP_ZERO, mark pool page as known-zero
Clears HPG_zeroed:
- free_huge_folio: page was mapped to userspace, no longer known-zero when it returns to the pool
- alloc_hugetlb_folio: cleared unconditionally on output
- alloc_hugetlb_folio_reserve: cleared after checking
- The optimization is most effective with THP, where entire 2MB pages are allocated directly from reported order-9+ buddy pages. Without THP, only ~21% of order-0 allocations come from reported pages due to low-order fragmentation.
- Persistent hugetlb pool pages are not covered: when freed by userspace they return to the hugetlb free pool, not the buddy allocator, so they are never reported to the host. Surplus hugetlb pages are allocated from buddy and do benefit.
- PG_zeroed is aliased to PG_private. __free_pages_prepare() clears it (preventing filesystem PG_private from leaking as false PG_zeroed). FPI_ZEROED re-sets it after prepare for balloon deflate pages. Is aliasing PG_private acceptable, or should a different bit be used?
- With __GFP_ZERO, the folio is zeroed before mem_cgroup_charge(). If the charge fails (cgroup at limit), the zeroing work is wasted and the folio is freed and retried at a smaller order. Previously, zeroing was done after a successful charge. This is inherent to the __GFP_ZERO approach. Is this acceptable?
- On architectures with aliasing caches, upstream with init_on_alloc double-zeros user pages: once via kernel_init_pages() in post_alloc_hook, and again via clear_user_highpage() at the callsite (because user_alloc_needs_zeroing() returns true). Our patches eliminate this by zeroing once via folio_zero_user() in post_alloc_hook. Not a critical fix (people who set init_on_alloc know they are paying performance) but a nice cleanup anyway.
Test program:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#ifndef MADV_POPULATE_WRITE
#define MADV_POPULATE_WRITE 23
#endif
#ifndef MAP_HUGETLB
#define MAP_HUGETLB 0x40000
#endif
int main(int argc, char **argv)
{
unsigned long size;
int flags = MAP_PRIVATE | MAP_ANONYMOUS;
void *p;
int r; if (argc < 2) {
fprintf(stderr, "usage: %s <size_mb> [huge]\n", argv[0]);
return 1;
}
size = atol(argv[1]) * 1024UL * 1024;
if (argc >= 3 && strcmp(argv[2], "huge") == 0)
flags |= MAP_HUGETLB;
p = mmap(NULL, size, PROT_READ | PROT_WRITE, flags, -1, 0);
if (p == MAP_FAILED) {
perror("mmap");
return 1;
}
r = madvise(p, size, MADV_POPULATE_WRITE);
if (r) {
perror("madvise");
return 1;
}
munmap(p, size);
return 0;}
Test script (bench.sh):
#!/bin/bash
# Usage: bench.sh <size_mb> <iterations> [huge]
# Feature negotiation (DEVICE_INIT_REPORTED/ON_INFLATE) is
# handled by QEMU command line flags, not module parameters.
SZ=${1:-256}; ITER=${2:-10}; HUGE=${3:-}
FLUSH=/sys/module/page_reporting/parameters/flush
CSV=/tmp/perf.csv
rmmod virtio_balloon 2>/dev/null
insmod /mnt/share/virtio_balloon.ko
echo 512 > $FLUSH
[ "$HUGE" = "huge" ] && echo $((SZ/2)) > /proc/sys/vm/nr_overcommit_hugepages
rm -f $CSV
echo "=== sz=${SZ}MB iter=$ITER $HUGE ==="
for i in $(seq 1 $ITER); do
echo 3 > /proc/sys/vm/drop_caches
echo 512 > $FLUSH
perf stat -e task-clock,instructions,cache-misses \
-x, -o $CSV --append -- /mnt/share/alloc_once $SZ $HUGE
done
[ "$HUGE" = "huge" ] && echo 0 > /proc/sys/vm/nr_overcommit_hugepages
rmmod virtio_balloon
awk -F, '/^#/||/^$/{next}{v=$1+0;e=$3;gsub(/ /,"",e);s[e]+=v;ss[e]+=v*v;n[e]++}
END{for(e in s){a=s[e]/n[e];d=sqrt(ss[e]/n[e]-a*a);printf " %-16s %10.0f +- %8.0f (n=%d)\n",e,a,d,n[e]}}' $CSVCompile and run:
gcc -static -O2 -o alloc_once alloc_once.c
bash bench.sh 256 10 # regular pages
bash bench.sh 256 10 huge # hugetlb surplus
Changes since v8 (address Sashiko v8 review findings):
- Fix mempolicy interleave: combine vm_pgoff and VMA offset into a single expression before shifting, fixing carry loss for file-backed VMAs with unaligned vm_pgoff.
- Fix memory-failure: wrap ClearPageHWPoison in retry path with zone->lock (same race as TestSetPageHWPoison).
- Fix stale comment: "folio_zero_user writes" -> "page zeroing" in huge_memory.c __folio_mark_uptodate comment.
- Drop rounddown_pow_of_two for page reporting capacity (no-op for compiler optimization, halves batch size for non-power-of-2).
- Reorder: move "mm: balloon: use put_page_zeroed" before "virtio_balloon: implement DEVICE_INIT_ON_INFLATE" so the ClearPageZeroed handling is in place before any page gets the flag set.
- Various commit log improvements (PowerPC note in aliasing patch, memory-failure note about other HWPoison calls, wording fixes).
Changes since v7 (address Sashiko AI review findings):
- Fix dcache flush on VIPT aliasing architectures: add user_alloc_needs_zeroing() guard in post_alloc_hook to force folio_zero_user for user pages when cache aliasing requires it. Host-zeroed pages excluded (!zeroed). Optimization preserved.
- Fix folio_zero_user stub: replace macro with non-inline function in mm/memory.c to avoid double-evaluation and missing include.
- Fix C89 declaration-after-statement in free_huge_folio.
- Fix CMA __GFP_ZERO: pass through to cma_alloc_frozen_compound so HPG_zeroed accurately reflects whether page was zeroed.
- Fix big-endian bitmap: use test_bit_le() for inflate_bitmap.
- Fix migratepage: clear PageZeroed on old page before deflation.
- Fix page_reporting flush: overflow-safe loop, add -EINTR on signal, add code comment explaining double flush_delayed_work.
- Add atomic ClearPageZeroed (CLEARPAGEFLAG) for balloon migration path where zone->lock is not held.
- Add VM_WARN_ON_ONCE for order>0 without __GFP_COMP in post_alloc_hook (folio_zero_user requires compound metadata).
- Add _noprof pattern for vma_alloc_zeroed_movable_folio to preserve memory allocation profiling attribution.
- Add PageReported propagation in split_large_buddy (was missing from patch 2).
- Add FPI_ZEROED guard: skip PageZeroed when page_poisoning enabled and init_on_free disabled (poison overwrites zeroes).
- Add DMA alignment comment for inflate_bitmap (ACCESS_PLATFORM cleared, so not needed now).
- Restore tell_host comment explaining vq buffer assumption.
- Various code comments documenting design decisions.
- Drop __GFP_ZERO from gather_surplus_pages: avoid shifting zeroing from fault time to reservation time (mmap/fallocate). Pool pages are zeroed at fault time via alloc_hugetlb_folio. Fresh surplus allocations at fault time still benefit from __GFP_ZERO + HPG_zeroed.
- New patch: add alloc_contig_frozen_pages_user API with user_addr for cache-friendly zeroing in the contiguous allocation path.
- New patch: thread user_addr through gigantic hugetlb allocation via alloc_contig_frozen_pages_user.
- New patch: replace user_alloc_needs_zeroing() with aliasing-only checks (cpu_dcache_is_aliasing || cpu_icache_is_aliasing) in the post_alloc_hook guard. Avoids redundant re-zero on non-aliasing.
- New patch: serialize TestSetPageHWPoison with zone->lock in memory_failure to fix pre-existing race with non-atomic buddy flag operations (e.g. page->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP).
- New patch: disable VIRTIO_RING_F_INDIRECT_DESC in balloon to prevent GFP_KERNEL allocation under balloon_lock (OOM deadlock).
- New patch: skip kernel_init_pages for FPI_ZEROED when page poisoning is not enabled (page already zero, skip redundant work).
Also since v7 (address review by Gregory Price):
- Drop from_pool bool in alloc_hugetlb_folio: use folio_test_hugetlb_zeroed directly. HPG_zeroed is set by alloc_surplus_hugetlb_folio for fresh allocations, so the check handles both pool and fresh pages.
- Drop bool *zeroed output parameter from alloc_hugetlb_folio:
sink zeroing inside the function. When __GFP_ZERO is set and !folio_test_hugetlb_zeroed, call folio_zero_user internally.
- Rename addr to user_addr in alloc_hugetlb_folio, align internally with huge_page_mask.
- Add Reviewed-by: Gregory Price tags on reviewed patches.
New patches since v7:
- mm: memory-failure: serialize TestSetPageHWPoison with zone->lock
- mm: add alloc_contig_frozen_pages_user for cache-friendly zeroing
- mm: hugetlb: thread user_addr through gigantic page allocation
- mm: page_alloc: use aliasing checks instead of user_alloc_needs_zeroing
- virtio_balloon: disable indirect descriptors
- mm: page_alloc: skip kernel_init_pages for FPI_ZEROED when safe
Changes since v6 (address review by Gregory Price):
- Rework hugetlb: use gfp_t parameter instead of bool zero / bool *zeroed. Sink zeroing inside alloc_hugetlb_folio(). Pass raw fault address (user_addr) for cache-friendly zeroing on both pool-page and fresh allocation
paths. (Suggested by Gregory Price)
- Reorder compaction_alloc_noprof() to call prep_compound_page before post_alloc_hook for consistency.
(Suggested by Gregory Price)
- Reorder: interleave fix first, PageReported propagation and capacity fix moved to front as dependencies.
- Add USER_ADDR_NONE comments in mmap.c and internal.h explaining why -1 is never a valid userspace address.
- Fix err uninitialized warning in virtballoon_free_page_report().
- Lots of commit log tweaks.
Also in v7:
- Fix hugetlb pool page zeroing to use vmf->real_address (the actual faulting subpage) instead of vmf->address (hugepage-aligned), preserving cache-friendly zeroing locality that upstream had at the callsite.
- Remove dead/broken alloc_hugetlb_folio !CONFIG_HUGETLB_PAGE stub (returned NULL but callers check IS_ERR).
Changes since v5:
- Rebased onto v7.1-rc2.
- Split alloc_anon_folio and alloc_swap_folio raw fault address changes into separate patches.
- In virtio, move PAGE_POISON check for DEVICE_INIT_REPORTED from probe() to validate(), clearing the feature instead of just gating host_zeroes_pages. Same for confidential computing check.
- Fix bisectability: FPI_ZEROED definition and usage now in the same patch.
- Lots of commit log tweaks.
- Reorder: REPORTED before ON_INFLATE.
- Kerneldoc fixes.
Changes since v4:
With virtio spec posted, update to latest spec:
- Add VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED (bit 6) for reporting.
- Add VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE (bit 7) for inflate.
- Per-page virtqueue submission, per-page used_len feedback.
- Balloon migration preserves PageZeroed hint.
- Page_reporting capacity bugfix for small virtqueues.
- PG_zeroed propagation in split_large_buddy.
- Disable both features for confidential computing guests.
- Gate host_zeroes_pages on PAGE_POISON/poison_val: when PAGE_POISON is negotiated with non-zero poison_val, device fills with poison not zeros, so host_zeroes_pages must be false.
- Disable ON_INFLATE when PAGE_POISON with non-zero poison_val.
- Bound inflate bitmap reads by used_len from device.
- Move ON_INFLATE poison_val check to validate() for proper feature negotiation.
- Fix NUMA interleave index for unaligned VMA start (new patch 1).
- Drop vma_alloc_folio_user_addr: with the ilx fix, callers can pass raw fault address to vma_alloc_folio directly.
- Tested with DEBUG_VM, INIT_ON_ALLOC/FREE enabled.
Changes since v3 (address review by Gregory Price and David Hildenbrand):
- Keep user_addr threading internal: public APIs (__alloc_pages, __folio_alloc, folio_alloc_mpol) are unchanged. Only internal functions (__alloc_frozen_pages_noprof, __alloc_pages_mpol) carry user_addr. This eliminates all API churn for external callers.
- Add vma_alloc_folio_user_addr() (2/22) to separate NUMA policy address from the zeroing hint address. Fixes NUMA interleave index corruption when passing unaligned fault address for higher-order allocations.
- Add per-page zeroed_bitmap to page_reporting_dev_info (17/22). The driver's report() callback manages the bitmap. Drain checks it gated by the host_zeroes_pages static key. This matches the proposed virtio balloon extension at
https://lore.kernel.org/all/[email protected]/
- Clear PG_zeroed in __isolate_free_page() to prevent the aliased PG_private flag from leaking to compaction/alloc_contig paths.
- Do not exclude PG_zeroed from PAGE_FLAGS_CHECK_AT_PREP macro. Instead, __free_pages_prepare() clears it (preventing filesystem PG_private leaking as false PG_zeroed), and FPI_ZEROED sets it after prepare. Only buddy merge assertion is relaxed.
- Initialize alloc_context.user_addr in alloc_pages_bulk_noprof.
- Deflate and hugetlb changes are much smaller now. Still, the patchset can be merged gradually, if desired.
Changes since v2 (address review by Gregory Price and David Hildenbrand):
- v2 used pghint_t / vma_alloc_folio_hints API. v3 switches to threading user_addr through the page allocator and using __GFP_ZERO, so post_alloc_hook() can use folio_zero_user() for cache-friendly zeroing when the user fault address is known.
- Use FPI_ZEROED to set PG_zeroed after __free_pages_prepare() instead of runtime masking in __free_one_page (further refined in v4).
- Drop redundant page_poisoning_enabled() check from mm core free path, already guarded at feature negotiation time in virtio_balloon_validate. The balloon driver keeps its own page_poisoning_enabled_static() check as defense in depth.
- Split free_frozen_pages_zeroed and put_page_zeroed into separate patches. David Hildenbrand indicated he intends to rework balloon pages to be frozen (no refcount), at which point put_page_zeroed (21/22) can be dropped and the balloon can call free_frozen_pages_zeroed directly.
- Use HPG_zeroed flag (in hugetlb folio->private) for hugetlb pool pages instead of PG_zeroed, since pool pages are zeroed long after buddy allocation and PG_zeroed is consumed at allocation time.
- syzbot CI found a PF_NO_COMPOUND BUG in the v2 pghint_t approach where __ClearPageZeroed was called on compound hugetlb pages in free_huge_folio. The v3 HPG_zeroed approach avoids this.
- Remove redundant arch vma_alloc_zeroed_movable_folio overrides on x86, s390, m68k, and alpha (12/22). Suggested by David Hildenbrand.
- Updated benchmarking script to compute per-run avg +- stddev via awk on CSV output.
Changes v1->v2:
- Replaced __GFP_PREZEROED with PG_zeroed page flag (aliased PG_private)
- Added pghint_t type and vma_alloc_folio_hints() API
- Track PG_zeroed across buddy merges and splits
- Added post_alloc_hook integration (single consume/clear point)
- Added hugetlb support (pool pages + memfd)
- Added page_reporting flush parameter for deterministic testing
- Added free_frozen_pages_hint/put_page_hint for balloon deflate path
- Added try_to_claim_block PG_zeroed preservation
- Updated perf numbers with per-iteration flush methodology
Written with assistance from Claude (claude-opus-4-6). Reviewed by cursor-agent (GPT-5.4-xhigh).
Everything manually read, patchset split and commit logs edited manually.
Michael S. Tsirkin (37):
mm: mempolicy: fix interleave index calculation
mm: memory-failure: serialize TestSetPageHWPoison with zone->lock
mm: page_alloc: propagate PageReported flag across buddy splits
mm: page_reporting: allow driver to set batch capacity
mm: hugetlb: remove dead alloc_hugetlb_folio stub
mm: move vma_alloc_folio_noprof to page_alloc.c
mm: thread user_addr through page allocator for cache-friendly zeroing
mm: add alloc_contig_frozen_pages_user for cache-friendly zeroing
mm: hugetlb: thread user_addr through gigantic page allocation
mm: add folio_zero_user stub for configs without THP/HUGETLBFS
mm: page_alloc: move prep_compound_page before post_alloc_hook
mm: use folio_zero_user for user pages in post_alloc_hook
mm: use __GFP_ZERO in vma_alloc_zeroed_movable_folio
mm: remove arch vma_alloc_zeroed_movable_folio overrides
mm: alloc_anon_folio: pass raw fault address to vma_alloc_folio
mm: alloc_swap_folio: pass raw fault address to vma_alloc_folio
mm: use __GFP_ZERO in alloc_anon_folio
mm: vma_alloc_anon_folio_pmd: pass raw fault address to
vma_alloc_folio
mm: use __GFP_ZERO in vma_alloc_anon_folio_pmd
mm: hugetlb: add gfp parameter and skip zeroing for zeroed pages
mm: memfd: skip zeroing for zeroed hugetlb pool pages
mm: page_reporting: skip redundant zeroing of host-zeroed reported
pages
mm: page_alloc: use aliasing checks instead of
user_alloc_needs_zeroing
mm: page_reporting: add per-page zeroed bitmap for host feedback
mm: page_alloc: clear PG_zeroed on buddy merge if not both zero
mm: page_alloc: preserve PG_zeroed in page_del_and_expand
virtio_balloon: submit reported pages as individual buffers
virtio_balloon: disable indirect descriptors
mm: page_reporting: add flush parameter with page budget
mm: page_alloc: propagate PG_zeroed in split_large_buddy
virtio_balloon: skip zeroing for host-zeroed reported pages
virtio_balloon: disable reporting zeroed optimization for confidential
guests
mm: add free_frozen_pages_zeroed
mm: page_alloc: skip kernel_init_pages for FPI_ZEROED when safe
mm: add put_page_zeroed and folio_put_zeroed
mm: balloon: use put_page_zeroed for zeroed balloon pages
virtio_balloon: implement VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE
arch/alpha/include/asm/page.h | 3 - arch/m68k/include/asm/page_no.h | 3 - arch/s390/include/asm/page.h | 3 - arch/x86/include/asm/page.h | 3 - drivers/virtio/virtio_balloon.c | 177 +++++++++++++++--- fs/hugetlbfs/inode.c | 3 +- include/linux/cma.h | 3 +- include/linux/gfp.h | 18 +- include/linux/highmem.h | 15 +- include/linux/hugetlb.h | 18 +- include/linux/mm.h | 15 ++ include/linux/page-flags.h | 10 + include/linux/page_reporting.h | 13 ++ include/uapi/linux/virtio_balloon.h | 2 + mm/balloon.c | 10 +- mm/cma.c | 6 +- mm/compaction.c | 9 +- mm/huge_memory.c | 14 +- mm/hugetlb.c | 102 ++++++---- mm/hugetlb_cma.c | 4 +- mm/internal.h | 21 ++- mm/memfd.c | 14 +- mm/memory-failure.c | 8 + mm/memory.c | 26 ++- mm/mempolicy.c | 75 ++++---- mm/mmap.c | 6 + mm/page_alloc.c | 281 ++++++++++++++++++++++++---- mm/page_reporting.c | 99 ++++++++-- mm/page_reporting.h | 12 ++ mm/slub.c | 4 +- mm/swap.c | 20 +- 31 files changed, 748 insertions(+), 249 deletions(-)
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 1/37 mm: mempolicy: fix interleave index calculation
The NUMA interleave index was computed as two separate terms:
*ilx += vma->vm_pgoff >> order;
*ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order);This has two problems:
1. When vm_start is not aligned to the folio size, the subtraction before the shift lets low bits affect the result via borrows.
2. For file-backed VMAs, shifting vm_pgoff and the VMA offset independently loses carries between them, giving wrong chunk indices when vm_pgoff is not aligned to order.
Combine into a single expression that adds vm_pgoff and the page-granularity VMA offset first, then shifts once:
*ilx += (vma->vm_pgoff +
(addr >> PAGE_SHIFT) -
(vma->vm_start >> PAGE_SHIFT)) >> order;For anonymous VMAs, vm_pgoff equals vm_start >> PAGE_SHIFT, so the vm_pgoff and vm_start terms cancel and the result reduces to addr >> (PAGE_SHIFT + order), same as before.
For file-backed VMAs, the sum vm_pgoff + (addr >> PAGE_SHIFT)
- (vm_start >> PAGE_SHIFT) gives the file page offset of addr. Shifting by order gives the correct file chunk index.
--- mm/mempolicy.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/mempolicy.c +3 -2
@@ -2048,8 +2048,9 @@ struct mempolicy *get_vma_policy(struct vm_area_struct *vma, pol =get_task_policy (current);if (pol->mode ==MPOL_INTERLEAVE || pol->mode ==MPOL_WEIGHTED_INTERLEAVE ) { - *ilx += vma->vm_pgoff >> order; - *ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order); + *ilx += (vma->vm_pgoff + + (addr >>PAGE_SHIFT ) - + (vma->vm_start >>PAGE_SHIFT )) >> order; }return pol; }
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 2/37 mm: memory-failure: serialize TestSetPageHWPoison with zone->lock
In reply to Michael S. TsirkinTestSetPageHWPoison() is called without zone->lock, so its atomic update to page->flags can race with non-atomic flag operations that run under zone->lock in the buddy allocator.
In particular, __free_pages_prepare() does:
page->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP;
This non-atomic read-modify-write, while correctly excluding __PG_HWPOISON from the mask, can still lose a concurrent TestSetPageHWPoison if the read happens before the poison bit is set and the write happens after. Follow-up patches in this series add similar non-atomic flag operations as well.
Fix by acquiring zone->lock around TestSetPageHWPoison and around ClearPageHWPoison in the retry path. This serializes with all buddy flag manipulation. The cost is
negligible: one lock/unlock in an extremely rare path (hardware memory errors).
Note: SetPageHWPoison and TestClearPageHWPoison calls elsewhere in this file operate on pages already removed from the buddy allocator or on non-buddy pages (DAX, hugetlb), so they do not need zone->lock protection.
--- mm/memory-failure.c | 8 ++++++++ 1 file changed, 8 insertions(+)
mm/memory-failure.c +8 -0
@@ -2348,6 +2348,8 @@ int memory_failure(unsigned long pfn, int flags)unsigned long page_flags;bool retry = true;int hugetlb =0 ; +struct zone *zone; +unsigned long mf_flags;if (!sysctl_memory_failure_recovery)panic ("Memory failure on page %lx" , pfn);
@@ -2390,7 +2392,10 @@ int memory_failure(unsigned long pfn, int flags)if (hugetlb) goto unlock_mutex; + zone =page_zone (p); +spin_lock_irqsave (&zone->lock, mf_flags);if (TestSetPageHWPoison (p)) { +spin_unlock_irqrestore (&zone->lock, mf_flags); res = -EHWPOISON ;if (flags &MF_ACTION_REQUIRED ) res =kill_accessing_process (current, pfn, flags);
@@ -2399,6 +2404,7 @@ int memory_failure(unsigned long pfn, int flags)action_result (pfn,MF_MSG_ALREADY_POISONED ,MF_FAILED ); goto unlock_mutex; } +spin_unlock_irqrestore (&zone->lock, mf_flags);/* * We need/can do nothing about count=0 pages.
@@ -2420,7 +2426,9 @@ int memory_failure(unsigned long pfn, int flags) }else {/* We lost the race, try again */ if (retry) { +spin_lock_irqsave (&zone->lock, mf_flags);ClearPageHWPoison (p); +spin_unlock_irqrestore (&zone->lock, mf_flags); retry = false; goto try_again; }
Signature · 1 line
MST
Re: [PATCH v9 02/37] mm: memory-failure: serialize TestSetPageHWPoison with zone->lock
Quoted 45 lines · Michael S. Tsirkin
TestSetPageHWPoison()is called withoutzone->lock, so its atomic update topage->flagscan race with non-atomic flag operations that run underzone->lockin the buddy allocator.In particular,
__free_pages_prepare()does:page->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP;This non-atomic read-modify-write, while correctly excluding
__PG_HWPOISONfrom the mask, can still lose a concurrent TestSetPageHWPoison if the read happens before the poison bit is set and the write happens after. Follow-up patches in this series add similar non-atomic flag operations as well.Fix by acquiring
zone->lockaround TestSetPageHWPoison and around ClearPageHWPoison in the retry path. This serializes with all buddy flag manipulation. The cost is
negligible: one lock/unlock in an extremely rare path (hardware memory errors).Note: SetPageHWPoison and TestClearPageHWPoison calls elsewhere in this file operate on pages already removed from the buddy allocator or on non-buddy pages (DAX, hugetlb), so they do not need
zone->lockprotection.Signed-off-by Michael S. Tsirkin <[email protected]>Assisted-by Claude:claude-opus-4-6--- mm/memory-failure.c | 8 ++++++++ 1 file changed, 8 insertions(+)mm/memory-failure.c +2
-0@@ -2348,6 +2348,8 @@ int memory_failure(unsigned long pfn, int flags)unsigned long page_flags;bool retry = true;int hugetlb =0 ; +struct zone *zone; +unsigned long mf_flags;if (!sysctl_memory_failure_recovery)panic ("Memory failure on page %lx" , pfn);
mm/memory-failure.c int memory_failure(unsigned long pfn, int flags) · line 2396
Quoted 5 lines
mm/memory-failure.c +2
-0@@ -2390,7 +2392,10 @@ int memory_failure(unsigned long pfn, int flags)if (hugetlb) goto unlock_mutex; + zone =page_zone (p); +spin_lock_irqsave (&zone->lock, mf_flags);
Would it be better to add a comment here why zone->lock is needed?
Quoted 14 lines
mm/memory-failure.c +2
-0if (TestSetPageHWPoison (p)) { +spin_unlock_irqrestore (&zone->lock, mf_flags); res = -EHWPOISON ;if (flags &MF_ACTION_REQUIRED ) res =kill_accessing_process (current, pfn, flags);@@ -2399,6 +2404,7 @@ int memory_failure(unsigned long pfn, int flags)action_result (pfn,MF_MSG_ALREADY_POISONED ,MF_FAILED ); goto unlock_mutex; } +spin_unlock_irqrestore (&zone->lock, mf_flags);/* * We need/can do nothing about count=0 pages.
mm/memory-failure.c int memory_failure(unsigned long pfn, int flags) · line 2431
Quoted 6 lines
mm/memory-failure.c +2
-0@@ -2420,7 +2426,9 @@ int memory_failure(unsigned long pfn, int flags) }else {/* We lost the race, try again */ if (retry) { +spin_lock_irqsave (&zone->lock, mf_flags);ClearPageHWPoison (p); +spin_unlock_irqrestore (&zone->lock, mf_flags);
Ditto.
Thanks.
.
Michael S. Tsirkin
maintainerPATCH v9 3/37 mm: page_alloc: propagate PageReported flag across buddy splits
In reply to Michael S. TsirkinWhen a reported free page is split via expand() to satisfy a smaller allocation, the sub-pages placed back on the free lists lose the PageReported flag. This means they will be unnecessarily re-reported to the hypervisor in the next reporting cycle, wasting work.
While I was unable to quantify the performance difference, it is an obvious waste, even if small.
Propagate the PageReported flag to sub-pages during expand(), both in page_del_and_expand() and try_to_claim_block(), so that they are recognized as already-reported.
--- mm/page_alloc.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)
mm/page_alloc.c +18 -3
@@ -1510,6 +1510,7 @@ static void split_large_buddy(struct zone *zone, struct page *page, unsigned long pfn, int order, fpi_t fpi) { unsigned long end = pfn + (1 << order); +bool reported =PageReported (page);VM_WARN_ON_ONCE (!IS_ALIGNED (pfn,1 << order));/* Caller removed page from freelist, buddy info cleared! */
@@ -1521,6 +1522,8 @@ static void split_large_buddy(struct zone *zone, struct page *page, do { int mt = get_pfnblock_migratetype(page, pfn); + if (reported) + __SetPageReported(page); __free_one_page(page, pfn, zone, order, mt, fpi); pfn += 1 << order; if (pfn == end)
@@ -1699,7 +1702,7 @@ struct page *__pageblock_pfn_to_page(unsigned long start_pfn,* -- nyc */ static inline unsigned int expand (struct zone *zone,struct page *page,int low, -int high,int migratetype) +int high,int migratetype,bool reported) {unsigned int size =1 << high;unsigned int nr_added =0 ;
@@ -1721,6 +1724,15 @@ static inline unsigned int expand(struct zone *zone, struct page *page, int low,__add_to_free_list (&page[size], zone, high, migratetype, false);set_buddy_order (&page[size], high); nr_added += size; + +/* +* The parent page has been reported to the host. The +* sub-pages are part of the same reported block, so mark +* them reported too. This avoids re-reporting pages that +* the host already knows about. +*/ +if (reported) +__SetPageReported (&page[size]); }return nr_added;
@@ -1731,9 +1743,10 @@ static __always_inline void page_del_and_expand(struct zone *zone, int high, int migratetype) { int nr_pages = 1 << high; +bool was_reported =page_reported (page);__del_page_from_free_list (page, zone, high, migratetype); - nr_pages -=expand (zone, page, low, high, migratetype); + nr_pages -=expand (zone, page, low, high, migratetype, was_reported);account_freepages (zone, -nr_pages, migratetype); }
@@ -2288,10 +2301,12 @@ try_to_claim_block(struct zone *zone, struct page *page,/* Take ownership for orders >= pageblock_order */ if (current_order >= pageblock_order) {unsigned int nr_added; +bool was_reported =page_reported (page);del_page_from_free_list (page, zone, current_order, block_type);change_pageblock_range (page, current_order, start_type); - nr_added =expand (zone, page, order, current_order, start_type); + nr_added =expand (zone, page, order, current_order, start_type, + was_reported);account_freepages (zone, nr_added, start_type);return page; }
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 4/37 mm: page_reporting: allow driver to set batch capacity
In reply to Michael S. TsirkinAdd a capacity field to page_reporting_dev_info so drivers can control the maximum number of pages per report batch. This is useful when the driver needs to reserve virtqueue descriptors for metadata (e.g., a bitmap buffer) alongside the page buffers.
The value is capped at PAGE_REPORTING_CAPACITY and rounded down to a power of 2. If unset (0), defaults to PAGE_REPORTING_CAPACITY.
The virtio_balloon driver sets capacity to the reporting virtqueue size, letting page_reporting adapt to whatever the device provides.
--- drivers/virtio/virtio_balloon.c | 5 +---- include/linux/page_reporting.h | 3 +++ mm/page_reporting.c | 25 ++++++++++++++----------- 3 files changed, 18 insertions(+), 15 deletions(-)
drivers/virtio/virtio_balloon.c +1 -4
@@ -1017,10 +1017,6 @@ static int virtballoon_probe(struct virtio_device *vdev)unsigned int capacity; capacity =virtqueue_get_vring_size (vb->reporting_vq); -if (capacity <PAGE_REPORTING_CAPACITY ) { - err = -ENOSPC ; - goto out_unregister_oom; - } vb->pr_dev_info.order =PAGE_REPORTING_ORDER_UNSPECIFIED ;
@@ -1041,6 +1037,7 @@ static int virtballoon_probe(struct virtio_device *vdev) vb->pr_dev_info.order =5 ;#endif + vb->pr_dev_info.capacity = capacity; err =page_reporting_register (&vb->pr_dev_info);if (err) goto out_unregister_oom;
include/linux/page_reporting.h +3 -0
@@ -22,6 +22,9 @@ struct page_reporting_dev_info {/* Minimal order of page reporting */ unsigned int order; + +/* Max pages per report batch (default PAGE_REPORTING_CAPACITY) */ +unsigned int capacity; };/* Tear-down and bring-up for page reporting devices */
mm/page_reporting.c +14 -11
@@ -174,10 +174,10 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,* list processed. This should result in us reporting all pages on * an idle system in about 30 seconds. * -* The division here should be cheap since PAGE_REPORTING_CAPACITY -* should always be a power of 2. +* The division here should be cheap since capacity should +* always be a power of 2. */ - budget =DIV_ROUND_UP (area->nr_free,PAGE_REPORTING_CAPACITY *16 ); + budget =DIV_ROUND_UP (area->nr_free, prdev->capacity *16 );/* loop through free list adding unreported pages to sg list */ list_for_each_entry_safe (page, next, list, lru) {
@@ -222,10 +222,10 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,spin_unlock_irq (&zone->lock);/* begin processing pages in local list */ - err = prdev->report (prdev, sgl,PAGE_REPORTING_CAPACITY ); + err = prdev->report (prdev, sgl, prdev->capacity);/* reset offset since the full list was reported */ - *offset =PAGE_REPORTING_CAPACITY ; + *offset = prdev->capacity;/* update budget to reflect call to report function */ budget--;
@@ -234,7 +234,7 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone,spin_lock_irq (&zone->lock);/* flush reported pages from the sg list */ -page_reporting_drain (prdev, sgl,PAGE_REPORTING_CAPACITY , !err); +page_reporting_drain (prdev, sgl, prdev->capacity, !err);/* * Reset next to first entry, the old next isn't valid
@@ -260,13 +260,13 @@ static int page_reporting_process_zone(struct page_reporting_dev_info *prdev, struct scatterlist *sgl, struct zone *zone) { - unsigned int order, mt, leftover, offset = PAGE_REPORTING_CAPACITY; + unsigned int order, mt, leftover, offset = prdev->capacity;unsigned long watermark;int err =0 ;/* Generate minimum watermark to be able to guarantee progress */ watermark =low_wmark_pages (zone) + - (PAGE_REPORTING_CAPACITY << page_reporting_order); + (prdev->capacity << page_reporting_order);/* * Cancel request if insufficient free memory or if we failed
@@ -290,7 +290,7 @@ page_reporting_process_zone(struct page_reporting_dev_info *prdev, }/* report the leftover pages before going idle */ - leftover =PAGE_REPORTING_CAPACITY - offset; + leftover = prdev->capacity - offset;if (leftover) { sgl = &sgl[offset]; err = prdev->report (prdev, sgl, leftover);
@@ -322,11 +322,11 @@ static void page_reporting_process(struct work_struct *work)atomic_set (&prdev->state, state);/* allocate scatterlist to store pages being reported on */ - sgl =kmalloc_objs (*sgl,PAGE_REPORTING_CAPACITY ); + sgl =kmalloc_objs (*sgl, prdev->capacity);if (!sgl) goto err_out; -sg_init_table (sgl,PAGE_REPORTING_CAPACITY ); +sg_init_table (sgl, prdev->capacity);for_each_zone (zone) { err =page_reporting_process_zone (prdev, sgl, zone);
@@ -377,6 +377,9 @@ int page_reporting_register(struct page_reporting_dev_info *prdev) page_reporting_order = pageblock_order; } +if (!prdev->capacity || prdev->capacity >PAGE_REPORTING_CAPACITY ) + prdev->capacity =PAGE_REPORTING_CAPACITY ; +/* initialize state and work structures */ atomic_set (&prdev->state,PAGE_REPORTING_IDLE );INIT_DELAYED_WORK (&prdev->work, &page_reporting_process);
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 5/37 mm: hugetlb: remove dead alloc_hugetlb_folio stub
In reply to Michael S. TsirkinRemove the !CONFIG_HUGETLB_PAGE stub for alloc_hugetlb_folio().
The stub is dead code: all callers are in mm/hugetlb.c (CONFIG_HUGETLB_PAGE) or fs/hugetlbfs/inode.c (CONFIG_HUGETLBFS), and CONFIG_HUGETLB_PAGE is def_bool HUGETLBFS with nothing selecting it independently.
The stub is also broken: it returns NULL, but all callers check IS_ERR(folio), so a NULL return would not be caught and would crash on the subsequent folio dereference.
Remove it now since follow-up patches change the signature of alloc_hugetlb_folio and would otherwise need to update the broken stub too.
--- include/linux/hugetlb.h | 7 ------- 1 file changed, 7 deletions(-)
include/linux/hugetlb.h +0 -7
@@ -1129,13 +1129,6 @@ static inline void wait_for_freed_hugetlb_folios(void) { } -static inline struct folio *alloc_hugetlb_folio (struct vm_area_struct *vma, -unsigned long addr, -bool cow_from_owner) -{ -return NULL ; -} -static inline struct folio *alloc_hugetlb_folio_reserve (struct hstate *h,int preferred_nid,nodemask_t *nmask,gfp_t gfp_mask)
Signature · 1 line
MST
Re: [PATCH v9 05/37] mm: hugetlb: remove dead alloc_hugetlb_folio stub
Quoted 11 lines · Michael S. Tsirkin
Remove the !CONFIG_HUGETLB_PAGE stub for
alloc_hugetlb_folio().The stub is dead code: all callers are in
mm/hugetlb.c(CONFIG_HUGETLB_PAGE) orfs/hugetlbfs/inode.c(CONFIG_HUGETLBFS), andCONFIG_HUGETLB_PAGEisdef_boolHUGETLBFS with nothing selecting it independently.The stub is also broken: it returns NULL, but all callers check
IS_ERR(folio), so a NULL return would not be caught and would crash on the subsequent folio dereference.
Quoted 8 lines
Remove it now since follow-up patches change the signature of
alloc_hugetlb_folioand would otherwise need to update the broken stub too.Signed-off-by Michael S. Tsirkin <[email protected]>Reviewed-by Gregory Price <[email protected]>Assisted-by Claude:claude-opus-4-6---
Quoted 21 lines
include/linux/hugetlb.h | 7 ------- 1 file changed, 7 deletions(-)include/linux/hugetlb.h +0
-7@@ -1129,13 +1129,6 @@ static inline void wait_for_freed_hugetlb_folios(void) { } -static inline struct folio *alloc_hugetlb_folio (struct vm_area_struct *vma, -unsigned long addr, -bool cow_from_owner) -{ -return NULL ; -} -static inline struct folio *alloc_hugetlb_folio_reserve (struct hstate *h,int preferred_nid,nodemask_t *nmask,gfp_t gfp_mask)
Michael S. Tsirkin
maintainerPATCH v9 6/37 mm: move vma_alloc_folio_noprof to page_alloc.c
In reply to Michael S. TsirkinMove vma_alloc_folio_noprof() from an inline in gfp.h (for !NUMA) and mempolicy.c (for NUMA) to page_alloc.c.
This prepares for a subsequent patch that will thread user_addr through the allocator: having vma_alloc_folio_noprof in page_alloc.c means user_addr can be passed to the internal allocation path without changing public API signatures or duplicating plumbing in both gfp.h and mempolicy.c.
The !NUMA path gains the VM_DROPPABLE -> __GFP_NOWARN check that the NUMA path already had.
--- include/linux/gfp.h | 9 ++------- mm/mempolicy.c | 32 -------------------------------- mm/page_alloc.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 45 insertions(+), 39 deletions(-)
include/linux/gfp.h +2 -7
@@ -318,13 +318,13 @@ static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask,#define alloc_pages_node (...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__)) +struct folio *vma_alloc_folio_noprof (gfp_t gfp,int order, +struct vm_area_struct *vma,unsigned long addr);#ifdef CONFIG_NUMA struct page *alloc_pages_noprof (gfp_t gfp,unsigned int order);struct folio *folio_alloc_noprof (gfp_t gfp,unsigned int order);struct folio *folio_alloc_mpol_noprof (gfp_t gfp,unsigned int order,struct mempolicy *mpol,pgoff_t ilx,int nid); -struct folio *vma_alloc_folio_noprof (gfp_t gfp,int order,struct vm_area_struct *vma, -unsigned long addr);#else static inline struct page *alloc_pages_noprof (gfp_t gfp_mask,unsigned int order) {
@@ -339,11 +339,6 @@ static inline struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int orde {return folio_alloc_noprof (gfp, order); } -static inline struct folio *vma_alloc_folio_noprof (gfp_t gfp,int order, -struct vm_area_struct *vma,unsigned long addr) -{ -return folio_alloc_noprof (gfp, order); -}#endif #define alloc_pages (...) alloc_hooks(alloc_pages_noprof(__VA_ARGS__))
mm/mempolicy.c +0 -32
@@ -2516,38 +2516,6 @@ struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order,return page_rmappable_folio (page); } -/** -* vma_alloc_folio - Allocate a folio for a VMA. -* @gfp: GFP flags. -* @order: Order of the folio. -* @vma: Pointer to VMA. -* @addr: Virtual address of the allocation. Must be inside @vma. -* -* Allocate a folio for a specific address in @vma, using the appropriate -* NUMA policy. The caller must hold the mmap_lock of the mm_struct of the -* VMA to prevent it from going away. Should be used for all allocations -* for folios that will be mapped into user space, excepting hugetlbfs, and -* excepting where direct use of folio_alloc_mpol() is more appropriate. -* -* Return: The folio on success or NULL if allocation fails. -*/ -struct folio *vma_alloc_folio_noprof (gfp_t gfp,int order,struct vm_area_struct *vma, -unsigned long addr) -{ -struct mempolicy *pol; -pgoff_t ilx; -struct folio *folio; - -if (vma->vm_flags &VM_DROPPABLE ) - gfp |= __GFP_NOWARN; - - pol =get_vma_policy (vma, addr, order, &ilx); - folio =folio_alloc_mpol_noprof (gfp, order, pol, ilx,numa_node_id ()); -mpol_cond_put (pol); -return folio; -} -EXPORT_SYMBOL (vma_alloc_folio_noprof); -struct page *alloc_frozen_pages_noprof (gfp_t gfp,unsigned order) {struct mempolicy *pol = &default_policy;
mm/page_alloc.c +43 -0
49 lines not on this page · raw message
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 7/37 mm: thread user_addr through page allocator for cache-friendly zeroing
In reply to Michael S. TsirkinThread a user virtual address from vma_alloc_folio() down through the page allocator to post_alloc_hook(). This is plumbing preparation for a subsequent patch that will use user_addr to call folio_zero_user() for cache-friendly zeroing of user pages.
The user_addr is stored in struct alloc_context and flows through:
vma_alloc_folio -> folio_alloc_mpol -> __alloc_pages_mpol -> __alloc_frozen_pages -> get_page_from_freelist -> prep_new_page -> post_alloc_hook
USER_ADDR_NONE ((unsigned long)-1) is used for non-user allocations, since address 0 is a valid userspace mapping.
--- include/linux/gfp.h | 2 +- mm/compaction.c | 5 ++--- mm/hugetlb.c | 36 ++++++++++++++++++++---------------- mm/internal.h | 22 +++++++++++++++++++--- mm/mempolicy.c | 44 ++++++++++++++++++++++++++++++++------------ mm/mmap.c | 6 ++++++ mm/page_alloc.c | 44 +++++++++++++++++++++++++++++--------------- mm/slub.c | 4 ++-- 8 files changed, 111 insertions(+), 52 deletions(-)
include/linux/gfp.h +1 -1
@@ -337,7 +337,7 @@ static inline struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order)static inline struct folio *folio_alloc_mpol_noprof (gfp_t gfp,unsigned int order,struct mempolicy *mpol,pgoff_t ilx,int nid) { -return folio_alloc_noprof (gfp, order); +return __folio_alloc_noprof (gfp, order,numa_node_id (),NULL ); }#endif
mm/compaction.c +2 -3
@@ -82,7 +82,7 @@ static inline bool is_via_compact_memory(int order) { return false; }static struct page *mark_allocated_noprof (struct page *page,unsigned int order,gfp_t gfp_flags) { -post_alloc_hook (page, order, __GFP_MOVABLE); +post_alloc_hook (page, order, __GFP_MOVABLE,USER_ADDR_NONE );set_page_refcounted (page);return page; }
@@ -1849,8 +1849,7 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long daset_page_private (&freepage[size], start_order); } dst = (struct folio *)freepage; - -post_alloc_hook (&dst->page, order, __GFP_MOVABLE); +post_alloc_hook (&dst->page, order, __GFP_MOVABLE,USER_ADDR_NONE );set_page_refcounted (&dst->page);if (order)prep_compound_page (&dst->page, order);
mm/hugetlb.c +20 -16
@@ -1806,7 +1806,8 @@ struct address_space *hugetlb_folio_mapping_lock_write(struct folio *folio) }static struct folio *alloc_buddy_frozen_folio (int order,gfp_t gfp_mask, -int nid,nodemask_t *nmask,nodemask_t *node_alloc_noretry) +int nid,nodemask_t *nmask,nodemask_t *node_alloc_noretry, +unsigned long addr) {struct folio *folio;bool alloc_try_hard = true;
@@ -1823,7 +1824,7 @@ static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask,if (alloc_try_hard) gfp_mask |= __GFP_RETRY_MAYFAIL; - folio = (struct folio *)__alloc_frozen_pages (gfp_mask, order, nid, nmask); + folio = (struct folio *)__alloc_frozen_pages (gfp_mask, order, nid, nmask, addr);/* * If we did not specify __GFP_RETRY_MAYFAIL, but still got a
@@ -1852,7 +1853,7 @@ static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask,static struct folio *only_alloc_fresh_hugetlb_folio (struct hstate *h,gfp_t gfp_mask,int nid,nodemask_t *nmask, -nodemask_t *node_alloc_noretry) +nodemask_t *node_alloc_noretry,unsigned long addr) {struct folio *folio;int order =huge_page_order (h);
@@ -1864,7 +1865,7 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, folio =alloc_gigantic_frozen_folio (order, gfp_mask, nid, nmask); else folio =alloc_buddy_frozen_folio (order, gfp_mask, nid, nmask, - node_alloc_noretry); + node_alloc_noretry, addr);if (folio)init_new_hugetlb_folio (folio);return folio;
@@ -1878,11 +1879,12 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h,* pages is zero, and the accounting must be done in the caller. */ static struct folio *alloc_fresh_hugetlb_folio (struct hstate *h, -gfp_t gfp_mask,int nid,nodemask_t *nmask) +gfp_t gfp_mask,int nid,nodemask_t *nmask, +unsigned long addr) {struct folio *folio; - folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask,NULL ); + folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask,NULL , addr);if (folio)hugetlb_vmemmap_optimize_folio (h, folio);return folio;
@@ -1922,7 +1924,7 @@ static struct folio *alloc_pool_huge_folio(struct hstate *h,struct folio *folio; folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, node, - nodes_allowed, node_alloc_noretry); + nodes_allowed, node_alloc_noretry,USER_ADDR_NONE );if (folio)return folio; }
@@ -2091,7 +2093,8 @@ int dissolve_free_hugetlb_folios(unsigned long start_pfn, unsigned long end_pfn)* Allocates a fresh surplus page from the page allocator. */ static struct folio *alloc_surplus_hugetlb_folio (struct hstate *h, -gfp_t gfp_mask,int nid,nodemask_t *nmask) +gfp_t gfp_mask,int nid,nodemask_t *nmask, +unsigned long addr) {struct folio *folio =NULL ;
@@ -2103,7 +2106,7 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h, goto out_unlock;spin_unlock_irq (&hugetlb_lock); - folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask); + folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask, addr);if (!folio)return NULL ;
@@ -2146,7 +2149,7 @@ static struct folio *alloc_migrate_hugetlb_folio(struct hstate *h, gfp_t gfp_masif (hstate_is_gigantic (h))return NULL ; - folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask); + folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask,USER_ADDR_NONE );if (!folio)return NULL ;
@@ -2182,14 +2185,14 @@ struct folio *alloc_buddy_hugetlb_folio_with_mpol(struct hstate *h,if (mpol_is_preferred_many (mpol)) {gfp_t gfp = gfp_mask & ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - folio =alloc_surplus_hugetlb_folio (h, gfp, nid, nodemask); + folio =alloc_surplus_hugetlb_folio (h, gfp, nid, nodemask, addr);/* Fallback to all nodes if page==NULL */ nodemask =NULL ; }if (!folio) - folio =alloc_surplus_hugetlb_folio (h, gfp_mask, nid, nodemask); + folio =alloc_surplus_hugetlb_folio (h, gfp_mask, nid, nodemask, addr);mpol_cond_put (mpol);return folio; }
@@ -2296,7 +2299,8 @@ static int gather_surplus_pages(struct hstate *h, long delta)* down the road to pick the current node if that is the case. */ folio =alloc_surplus_hugetlb_folio (h,htlb_alloc_mask (h), -NUMA_NO_NODE , &alloc_nodemask); +NUMA_NO_NODE , &alloc_nodemask, +USER_ADDR_NONE );if (!folio) { alloc_ok = false;break ;
@@ -2702,7 +2706,7 @@ static int alloc_and_dissolve_hugetlb_folio(struct folio *old_folio,spin_unlock_irq (&hugetlb_lock); gfp_mask =htlb_alloc_mask (h) | __GFP_THISNODE; new_folio =alloc_fresh_hugetlb_folio (h, gfp_mask, - nid,NULL ); + nid,NULL ,USER_ADDR_NONE );if (!new_folio)return -ENOMEM ; goto retry;
@@ -3400,13 +3404,13 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)gfp_t gfp_mask =htlb_alloc_mask (h) | __GFP_THISNODE; folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, - &node_states[N_MEMORY ],NULL ); + &node_states[N_MEMORY ],NULL ,USER_ADDR_NONE );if (!folio && !list_empty (&folio_list) &&hugetlb_vmemmap_optimizable_size (h)) {prep_and_add_allocated_folios (h, &folio_list);INIT_LIST_HEAD (&folio_list); folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, - &node_states[N_MEMORY ],NULL ); + &node_states[N_MEMORY ],NULL ,USER_ADDR_NONE ); }if (!folio)break ;
mm/internal.h +19 -3
@@ -662,6 +662,16 @@ void calculate_min_free_kbytes(void);int __meminit init_per_zone_wmark_min (void );void page_alloc_sysctl_init (void ); +/* +* Sentinel for user_addr: indicates a non-user allocation. +* Cannot use 0 because address 0 is a valid userspace mapping. +* (unsigned long)-1 is safe because: +* 1. vm_end = addr + len <= TASK_SIZE, and vm_end is exclusive, +* so -1 is never inside any VMA. +* 2. It will only be compared to page-aligned addresses. +*/ +#define USER_ADDR_NONE ((unsigned long)-1) +/* * Structure for holding the mostly immutable allocation parameters passed * between functions involved in allocations, including the alloc_pages*
@@ -693,6 +703,7 @@ struct alloc_context {*/ enum zone_type highest_zoneidx;bool spread_dirty_pages; +unsigned long user_addr; };/*
@@ -916,24 +927,29 @@ static inline void init_compound_tail(struct page *tail,prep_compound_tail (tail, head, order); } -void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags); +void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags, +unsigned long user_addr);extern bool free_pages_prepare (struct page *page,unsigned int order);extern int user_min_free_kbytes;struct page *__alloc_frozen_pages_noprof (gfp_t ,unsigned int order,int nid, -nodemask_t *); +nodemask_t *,unsigned long user_addr);#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))void free_frozen_pages (struct page *page,unsigned int order); +void free_frozen_pages_zeroed (struct page *page,unsigned int order);void free_unref_folios (struct folio_batch *fbatch);#ifdef CONFIG_NUMA struct page *alloc_frozen_pages_noprof (gfp_t ,unsigned int order); +struct folio *folio_alloc_mpol_user_noprof (gfp_t gfp,unsigned int order, +struct mempolicy *pol,pgoff_t ilx,int nid, +unsigned long user_addr);#else static inline struct page *alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order) { -return __alloc_frozen_pages_noprof (gfp, order,numa_node_id (),NULL ); +return __alloc_frozen_pages_noprof (gfp, order,numa_node_id (),NULL ,USER_ADDR_NONE ); }#endif
mm/mempolicy.c +32 -12
102 lines not on this page · raw message
mm/mmap.c +6 -0
@@ -855,6 +855,12 @@ __get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,if (IS_ERR_VALUE (addr))return addr; +/* +* The check below ensures vm_end = addr + len <= TASK_SIZE. +* Since (unsigned long)-1 (USER_ADDR_NONE) >= TASK_SIZE and +* vm_end is exclusive, USER_ADDR_NONE is thus never a valid +* userspace address. +*/ if (addr >TASK_SIZE - len)return -ENOMEM ;if (offset_in_page (addr))
mm/page_alloc.c +29 -15
133 lines not on this page · raw message
mm/slub.c +2 -2
@@ -3275,7 +3275,7 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node,else if (node ==NUMA_NO_NODE ) page =alloc_frozen_pages (flags, order);else - page =__alloc_frozen_pages (flags, order, node,NULL ); + page =__alloc_frozen_pages (flags, order, node,NULL ,USER_ADDR_NONE );if (!page)return NULL ;
@@ -5235,7 +5235,7 @@ static void *___kmalloc_large_node(size_t size, gfp_t flags, int node)if (node ==NUMA_NO_NODE ) page =alloc_frozen_pages_noprof (flags, order);else - page =__alloc_frozen_pages_noprof (flags, order, node,NULL ); + page =__alloc_frozen_pages_noprof (flags, order, node,NULL ,USER_ADDR_NONE );if (page) { ptr =page_address (page);
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRe: [PATCH v9 07/37] mm: thread user_addr through page allocator for cache-friendly zeroing
Quoted 245 lines · Michael S. Tsirkin
Thread a user virtual address from
vma_alloc_folio()down through the page allocator topost_alloc_hook(). This is plumbing preparation for a subsequent patch that will useuser_addrto callfolio_zero_user()for cache-friendly zeroing of user pages.The
user_addris stored in structalloc_contextand flows through:
vma_alloc_folio->folio_alloc_mpol->__alloc_pages_mpol->__alloc_frozen_pages->get_page_from_freelist->prep_new_page->post_alloc_hook
USER_ADDR_NONE((unsigned long)-1) is used for non-user allocations, since address 0 is a valid userspace mapping.Signed-off-by Michael S. Tsirkin <[email protected]>Assisted-by Claude:claude-opus-4-6Assisted-by cursor-agent:GPT-5.4-xhigh--- include/linux/gfp.h | 2 +- mm/compaction.c | 5 ++--- mm/hugetlb.c | 36 ++++++++++++++++++++---------------- mm/internal.h | 22 +++++++++++++++++++--- mm/mempolicy.c | 44 ++++++++++++++++++++++++++++++++------------ mm/mmap.c | 6 ++++++ mm/page_alloc.c | 44 +++++++++++++++++++++++++++++--------------- mm/slub.c | 4 ++-- 8 files changed, 111 insertions(+), 52 deletions(-)include/linux/gfp.h +1
-1@@ -337,7 +337,7 @@ static inline struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order)static inline struct folio *folio_alloc_mpol_noprof (gfp_t gfp,unsigned int order,struct mempolicy *mpol,pgoff_t ilx,int nid) { -return folio_alloc_noprof (gfp, order); +return __folio_alloc_noprof (gfp, order,numa_node_id (),NULL ); }#endif mm/compaction.c +2
-3@@ -82,7 +82,7 @@ static inline bool is_via_compact_memory(int order) { return false; }static struct page *mark_allocated_noprof (struct page *page,unsigned int order,gfp_t gfp_flags) { -post_alloc_hook (page, order, __GFP_MOVABLE); +post_alloc_hook (page, order, __GFP_MOVABLE,USER_ADDR_NONE );set_page_refcounted (page);return page; }@@ -1849,8 +1849,7 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long daset_page_private (&freepage[size], start_order); } dst = (struct folio *)freepage; - -post_alloc_hook (&dst->page, order, __GFP_MOVABLE); +post_alloc_hook (&dst->page, order, __GFP_MOVABLE,USER_ADDR_NONE );set_page_refcounted (&dst->page);if (order)prep_compound_page (&dst->page, order);mm/hugetlb.c +20
-16@@ -1806,7 +1806,8 @@ struct address_space *hugetlb_folio_mapping_lock_write(struct folio *folio) }static struct folio *alloc_buddy_frozen_folio (int order,gfp_t gfp_mask, -int nid,nodemask_t *nmask,nodemask_t *node_alloc_noretry) +int nid,nodemask_t *nmask,nodemask_t *node_alloc_noretry, +unsigned long addr) {struct folio *folio;bool alloc_try_hard = true;@@ -1823,7 +1824,7 @@ static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask,if (alloc_try_hard) gfp_mask |= __GFP_RETRY_MAYFAIL; - folio = (struct folio *)__alloc_frozen_pages (gfp_mask, order, nid, nmask); + folio = (struct folio *)__alloc_frozen_pages (gfp_mask, order, nid, nmask, addr);/* * If we did not specify __GFP_RETRY_MAYFAIL, but still got a @@ -1852,7 +1853,7 @@ static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask,static struct folio *only_alloc_fresh_hugetlb_folio (struct hstate *h,gfp_t gfp_mask,int nid,nodemask_t *nmask, -nodemask_t *node_alloc_noretry) +nodemask_t *node_alloc_noretry,unsigned long addr) {struct folio *folio;int order =huge_page_order (h);@@ -1864,7 +1865,7 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, folio =alloc_gigantic_frozen_folio (order, gfp_mask, nid, nmask); else folio =alloc_buddy_frozen_folio (order, gfp_mask, nid, nmask, - node_alloc_noretry); + node_alloc_noretry, addr);if (folio)init_new_hugetlb_folio (folio);return folio;@@ -1878,11 +1879,12 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h,* pages is zero, and the accounting must be done in the caller. */ static struct folio *alloc_fresh_hugetlb_folio (struct hstate *h, -gfp_t gfp_mask,int nid,nodemask_t *nmask) +gfp_t gfp_mask,int nid,nodemask_t *nmask, +unsigned long addr) {struct folio *folio; - folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask,NULL ); + folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask,NULL , addr);if (folio)hugetlb_vmemmap_optimize_folio (h, folio);return folio;@@ -1922,7 +1924,7 @@ static struct folio *alloc_pool_huge_folio(struct hstate *h,struct folio *folio; folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, node, - nodes_allowed, node_alloc_noretry); + nodes_allowed, node_alloc_noretry,USER_ADDR_NONE );if (folio)return folio; }@@ -2091,7 +2093,8 @@ int dissolve_free_hugetlb_folios(unsigned long start_pfn, unsigned long end_pfn)* Allocates a fresh surplus page from the page allocator. */ static struct folio *alloc_surplus_hugetlb_folio (struct hstate *h, -gfp_t gfp_mask,int nid,nodemask_t *nmask) +gfp_t gfp_mask,int nid,nodemask_t *nmask, +unsigned long addr) {struct folio *folio =NULL ;@@ -2103,7 +2106,7 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h, goto out_unlock;spin_unlock_irq (&hugetlb_lock); - folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask); + folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask, addr);if (!folio)return NULL ;@@ -2146,7 +2149,7 @@ static struct folio *alloc_migrate_hugetlb_folio(struct hstate *h, gfp_t gfp_masif (hstate_is_gigantic (h))return NULL ; - folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask); + folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask,USER_ADDR_NONE );if (!folio)return NULL ;@@ -2182,14 +2185,14 @@ struct folio *alloc_buddy_hugetlb_folio_with_mpol(struct hstate *h,if (mpol_is_preferred_many (mpol)) {gfp_t gfp = gfp_mask & ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - folio =alloc_surplus_hugetlb_folio (h, gfp, nid, nodemask); + folio =alloc_surplus_hugetlb_folio (h, gfp, nid, nodemask, addr);/* Fallback to all nodes if page==NULL */ nodemask =NULL ; }if (!folio) - folio =alloc_surplus_hugetlb_folio (h, gfp_mask, nid, nodemask); + folio =alloc_surplus_hugetlb_folio (h, gfp_mask, nid, nodemask, addr);mpol_cond_put (mpol);return folio; }@@ -2296,7 +2299,8 @@ static int gather_surplus_pages(struct hstate *h, long delta)* down the road to pick the current node if that is the case. */ folio =alloc_surplus_hugetlb_folio (h,htlb_alloc_mask (h), -NUMA_NO_NODE , &alloc_nodemask); +NUMA_NO_NODE , &alloc_nodemask, +USER_ADDR_NONE );if (!folio) { alloc_ok = false;break ;@@ -2702,7 +2706,7 @@ static int alloc_and_dissolve_hugetlb_folio(struct folio *old_folio,spin_unlock_irq (&hugetlb_lock); gfp_mask =htlb_alloc_mask (h) | __GFP_THISNODE; new_folio =alloc_fresh_hugetlb_folio (h, gfp_mask, - nid,NULL ); + nid,NULL ,USER_ADDR_NONE );if (!new_folio)return -ENOMEM ; goto retry;@@ -3400,13 +3404,13 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)gfp_t gfp_mask =htlb_alloc_mask (h) | __GFP_THISNODE; folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, - &node_states[N_MEMORY ],NULL ); + &node_states[N_MEMORY ],NULL ,USER_ADDR_NONE );if (!folio && !list_empty (&folio_list) &&hugetlb_vmemmap_optimizable_size (h)) {prep_and_add_allocated_folios (h, &folio_list);INIT_LIST_HEAD (&folio_list); folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, - &node_states[N_MEMORY ],NULL ); + &node_states[N_MEMORY ],NULL ,USER_ADDR_NONE ); }if (!folio)break ;mm/internal.h +13
-1@@ -662,6 +662,16 @@ void calculate_min_free_kbytes(void);int __meminit init_per_zone_wmark_min (void );void page_alloc_sysctl_init (void ); +/* +* Sentinel for user_addr: indicates a non-user allocation. +* Cannot use 0 because address 0 is a valid userspace mapping. +* (unsigned long)-1 is safe because: +* 1. vm_end = addr + len <= TASK_SIZE, and vm_end is exclusive, +* so -1 is never inside any VMA. +* 2. It will only be compared to page-aligned addresses. +*/ +#define USER_ADDR_NONE ((unsigned long)-1) +/* * Structure for holding the mostly immutable allocation parameters passed * between functions involved in allocations, including the alloc_pages* @@ -693,6 +703,7 @@ struct alloc_context {*/ enum zone_type highest_zoneidx;bool spread_dirty_pages; +unsigned long user_addr; };/* @@ -916,24 +927,29 @@ static inline void init_compound_tail(struct page *tail,prep_compound_tail (tail, head, order); } -void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags); +void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags, +unsigned long user_addr);extern bool free_pages_prepare (struct page *page,unsigned int order);
mm/internal.h static inline void init_compound_tail(struct page *tail, · line 941
Quoted 10 lines
mm/internal.h +2
-1extern int user_min_free_kbytes;struct page *__alloc_frozen_pages_noprof (gfp_t ,unsigned int order,int nid, -nodemask_t *); +nodemask_t *,unsigned long user_addr);#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))void free_frozen_pages (struct page *page,unsigned int order); +void free_frozen_pages_zeroed (struct page *page,unsigned int order);
sashiko pointed this one out:
https://sashiko.dev/#/patchset/cover.1780067977.git.mst%40redhat.com?part=7
this landed here by mistake during one of the rebases, harmless but ideally belongs in patch 33. Will move if there's v10.
it's other findings on this patch seem like false positives.
Quoted 320 lines
mm/internal.h +4
-1void free_unref_folios (struct folio_batch *fbatch);#ifdef CONFIG_NUMA struct page *alloc_frozen_pages_noprof (gfp_t ,unsigned int order); +struct folio *folio_alloc_mpol_user_noprof (gfp_t gfp,unsigned int order, +struct mempolicy *pol,pgoff_t ilx,int nid, +unsigned long user_addr);#else static inline struct page *alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order) { -return __alloc_frozen_pages_noprof (gfp, order,numa_node_id (),NULL ); +return __alloc_frozen_pages_noprof (gfp, order,numa_node_id (),NULL ,USER_ADDR_NONE ); }#endif mm/mempolicy.c +32
-12@@ -2413,7 +2413,8 @@ bool mempolicy_in_oom_domain(struct task_struct *tsk, }static struct page *alloc_pages_preferred_many (gfp_t gfp,unsigned int order, -int nid,nodemask_t *nodemask) +int nid,nodemask_t *nodemask, +unsigned long user_addr) {struct page *page;gfp_t preferred_gfp;@@ -2426,25 +2427,29 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order,*/ preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - page =__alloc_frozen_pages_noprof (preferred_gfp, order, nid, nodemask); + page =__alloc_frozen_pages_noprof (preferred_gfp, order, nid, + nodemask, user_addr);if (!page) - page =__alloc_frozen_pages_noprof (gfp, order, nid,NULL ); + page =__alloc_frozen_pages_noprof (gfp, order, nid,NULL , + user_addr);return page; }/** -* alloc_pages_mpol - Allocate pages according to NUMA mempolicy. +* __alloc_pages_mpol - Allocate pages according to NUMA mempolicy. * @gfp: GFP flags. * @order: Order of the page allocation. * @pol: Pointer to the NUMA mempolicy. * @ilx: Index for interleave mempolicy (also distinguishes alloc_pages()). * @nid: Preferred node (usually numa_node_id() but @mpol may override it). +* @user_addr: User fault address for cache-friendly zeroing, or USER_ADDR_NONE. * * Return: The page on success or NULL if allocation fails. */ -static struct page *alloc_pages_mpol (gfp_t gfp,unsigned int order, -struct mempolicy *pol,pgoff_t ilx,int nid) +static struct page *__alloc_pages_mpol (gfp_t gfp,unsigned int order, +struct mempolicy *pol,pgoff_t ilx,int nid, +unsigned long user_addr) {nodemask_t *nodemask;struct page *page;@@ -2452,7 +2457,8 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, nodemask =policy_nodemask (gfp, pol, ilx, &nid);if (pol->mode ==MPOL_PREFERRED_MANY ) -return alloc_pages_preferred_many (gfp, order, nid, nodemask); +return alloc_pages_preferred_many (gfp, order, nid, nodemask, + user_addr); if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&/* filter "hugepage" allocation, unless from alloc_pages() */ @@ -2476,7 +2482,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,*/ page =__alloc_frozen_pages_noprof ( gfp | __GFP_THISNODE | __GFP_NORETRY, order, - nid,NULL ); + nid,NULL , user_addr);if (page || !(gfp & __GFP_DIRECT_RECLAIM))return page;/* @@ -2488,7 +2494,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, } } - page =__alloc_frozen_pages_noprof (gfp, order, nid, nodemask); + page =__alloc_frozen_pages_noprof (gfp, order, nid, nodemask, user_addr);if (unlikely (pol->mode ==MPOL_INTERLEAVE || pol->mode ==MPOL_WEIGHTED_INTERLEAVE ) && page) {@@ -2504,11 +2510,18 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,return page; } -struct folio *folio_alloc_mpol_noprof (gfp_t gfp,unsigned int order, +static struct page *alloc_pages_mpol (gfp_t gfp,unsigned int order,struct mempolicy *pol,pgoff_t ilx,int nid) { -struct page *page =alloc_pages_mpol (gfp | __GFP_COMP, order, pol, - ilx, nid); +return __alloc_pages_mpol (gfp, order, pol, ilx, nid,USER_ADDR_NONE ); +} + +struct folio *folio_alloc_mpol_user_noprof (gfp_t gfp,unsigned int order, +struct mempolicy *pol,pgoff_t ilx,int nid, +unsigned long user_addr) +{ +struct page *page =__alloc_pages_mpol (gfp | __GFP_COMP, order, pol, + ilx, nid, user_addr);if (!page)return NULL ;@@ -2516,6 +2529,13 @@ struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order,return page_rmappable_folio (page); } +struct folio *folio_alloc_mpol_noprof (gfp_t gfp,unsigned int order, +struct mempolicy *pol,pgoff_t ilx,int nid) +{ +return folio_alloc_mpol_user_noprof (gfp, order, pol, ilx, nid, +USER_ADDR_NONE ); +} +struct page *alloc_frozen_pages_noprof (gfp_t gfp,unsigned order) {struct mempolicy *pol = &default_policy;mm/mmap.c +6
-0@@ -855,6 +855,12 @@ __get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,if (IS_ERR_VALUE (addr))return addr; +/* +* The check below ensures vm_end = addr + len <= TASK_SIZE. +* Since (unsigned long)-1 (USER_ADDR_NONE) >= TASK_SIZE and +* vm_end is exclusive, USER_ADDR_NONE is thus never a valid +* userspace address. +*/ if (addr >TASK_SIZE - len)return -ENOMEM ;if (offset_in_page (addr))mm/page_alloc.c +29
-15@@ -1819,7 +1819,7 @@ static inline bool should_skip_init(gfp_t flags) }inline void post_alloc_hook (struct page *page,unsigned int order, -gfp_t gfp_flags) +gfp_t gfp_flags,unsigned long user_addr) {bool init = !want_init_on_free () &&want_init_on_alloc (gfp_flags) && !should_skip_init (gfp_flags);@@ -1874,9 +1874,10 @@ inline void post_alloc_hook(struct page *page, unsigned int order, }static void prep_new_page (struct page *page,unsigned int order,gfp_t gfp_flags, -unsigned int alloc_flags) +unsigned int alloc_flags, +unsigned long user_addr) { -post_alloc_hook (page, order, gfp_flags); +post_alloc_hook (page, order, gfp_flags, user_addr);if (order && (gfp_flags & __GFP_COMP))prep_compound_page (page, order);@@ -3958,7 +3959,8 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, page =rmqueue (zonelist_zone (ac->preferred_zoneref), zone, order, gfp_mask, alloc_flags, ac->migratetype);if (page) { -prep_new_page (page, order, gfp_mask, alloc_flags); +prep_new_page (page, order, gfp_mask, alloc_flags, + ac->user_addr);return page; }else {@@ -4186,7 +4188,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,/* Prep a captured page if available */ if (page) -prep_new_page (page, order, gfp_mask, alloc_flags); +prep_new_page (page, order, gfp_mask, alloc_flags, + ac->user_addr);/* Try get a page from the freelist if available */ if (!page)@@ -5063,7 +5066,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,struct zoneref *z;struct per_cpu_pages *pcp;struct list_head *pcp_list; -struct alloc_context ac; +struct alloc_context ac = { .user_addr =USER_ADDR_NONE };gfp_t alloc_gfp;unsigned int alloc_flags =ALLOC_WMARK_LOW ;int nr_populated =0 , nr_account =0 ;@@ -5178,7 +5181,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, } nr_account++; -prep_new_page (page,0 , gfp,0 ); +prep_new_page (page,0 , gfp,0 ,USER_ADDR_NONE );set_page_refcounted (page); page_array[nr_populated++] = page; }@@ -5203,12 +5206,13 @@ EXPORT_SYMBOL_GPL(alloc_pages_bulk_noprof);* This is the 'heart' of the zoned buddy allocator. */ struct page *__alloc_frozen_pages_noprof (gfp_t gfp,unsigned int order, -int preferred_nid,nodemask_t *nodemask) +int preferred_nid,nodemask_t *nodemask, +unsigned long user_addr) {struct page *page;unsigned int alloc_flags =ALLOC_WMARK_LOW ;gfp_t alloc_gfp;/* The gfp_t that was actually used for allocation */ -struct alloc_context ac = { }; +struct alloc_context ac = { .user_addr = user_addr };/* * There are several places where we assume that the order value is sane @@ -5269,10 +5273,12 @@ EXPORT_SYMBOL(__alloc_frozen_pages_noprof);struct page *__alloc_pages_noprof (gfp_t gfp,unsigned int order,int preferred_nid,nodemask_t *nodemask) + {struct page *page; - page =__alloc_frozen_pages_noprof (gfp, order, preferred_nid, nodemask); + page =__alloc_frozen_pages_noprof (gfp, order, preferred_nid, + nodemask,USER_ADDR_NONE );if (page)set_page_refcounted (page);return page;@@ -5315,7 +5321,8 @@ struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, gfp |= __GFP_NOWARN; pol =get_vma_policy (vma, addr, order, &ilx); - folio =folio_alloc_mpol_noprof (gfp, order, pol, ilx,numa_node_id ()); + folio =folio_alloc_mpol_user_noprof (gfp, order, pol, ilx, +numa_node_id (), addr);mpol_cond_put (pol);return folio; }@@ -5323,10 +5330,17 @@ struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order,struct folio *vma_alloc_folio_noprof (gfp_t gfp,int order,struct vm_area_struct *vma,unsigned long addr) { +struct page *page; +if (vma->vm_flags &VM_DROPPABLE ) gfp |= __GFP_NOWARN; -return folio_alloc_noprof (gfp, order); + page =__alloc_frozen_pages_noprof (gfp | __GFP_COMP, order, +numa_node_id (),NULL , addr); +if (!page) +return NULL ; +set_page_refcounted (page); +return page_rmappable_folio (page); }#endif EXPORT_SYMBOL (vma_alloc_folio_noprof);@@ -6907,7 +6921,7 @@ static void split_free_frozen_pages(struct list_head *list, gfp_t gfp_mask)list_for_each_entry_safe (page, next, &list[order], lru) {int i; -post_alloc_hook (page, order, gfp_mask); +post_alloc_hook (page, order, gfp_mask,USER_ADDR_NONE );if (!order)continue ;@@ -7113,7 +7127,7 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,struct page *head =pfn_to_page (start);check_new_pages (head, order); -prep_new_page (head, order, gfp_mask,0 ); +prep_new_page (head, order, gfp_mask,0 ,USER_ADDR_NONE ); }else { ret = -EINVAL ; WARN(true, "PFN range: requested [%lu, %lu), allocated [%lu, %lu)\n",@@ -7778,7 +7792,7 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsignedgfp_t alloc_gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC | __GFP_COMP | gfp_flags;unsigned int alloc_flags =ALLOC_TRYLOCK ; -struct alloc_context ac = { }; +struct alloc_context ac = { .user_addr =USER_ADDR_NONE };struct page *page;VM_WARN_ON_ONCE (gfp_flags & ~__GFP_ACCOUNT);mm/slub.c +2
-2@@ -3275,7 +3275,7 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node,else if (node ==NUMA_NO_NODE ) page =alloc_frozen_pages (flags, order);else - page =__alloc_frozen_pages (flags, order, node,NULL ); + page =__alloc_frozen_pages (flags, order, node,NULL ,USER_ADDR_NONE );if (!page)return NULL ;@@ -5235,7 +5235,7 @@ static void *___kmalloc_large_node(size_t size, gfp_t flags, int node)if (node ==NUMA_NO_NODE ) page =alloc_frozen_pages_noprof (flags, order);else - page =__alloc_frozen_pages_noprof (flags, order, node,NULL ); + page =__alloc_frozen_pages_noprof (flags, order, node,NULL ,USER_ADDR_NONE );if (page) { ptr =page_address (page);Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 8/37 mm: add alloc_contig_frozen_pages_user for cache-friendly zeroing
In reply to Michael S. TsirkinAdd a _user variant of alloc_contig_frozen_pages that accepts a user_addr parameter for cache-friendly zeroing of contiguous allocations.
No functional change; all existing callers continue to pass USER_ADDR_NONE.
--- include/linux/gfp.h | 6 ++++++ mm/page_alloc.c | 32 +++++++++++++++++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-)
include/linux/gfp.h +6 -0
@@ -453,6 +453,12 @@ struct page *alloc_contig_frozen_pages_noprof(unsigned long nr_pages,#define alloc_contig_frozen_pages (...) \ alloc_hooks(alloc_contig_frozen_pages_noprof(__VA_ARGS__)) +struct page *alloc_contig_frozen_pages_user_noprof (unsigned long nr_pages, +gfp_t gfp_mask,int nid,nodemask_t *nodemask, +unsigned long user_addr); +#define alloc_contig_frozen_pages_user (...) \ + alloc_hooks(alloc_contig_frozen_pages_user_noprof(__VA_ARGS__)) +struct page *alloc_contig_pages_noprof (unsigned long nr_pages,gfp_t gfp_mask,int nid,nodemask_t *nodemask);#define alloc_contig_pages (...) \
mm/page_alloc.c +25 -7
@@ -6999,8 +6999,9 @@ static void __free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages* * Return: zero on success or negative error code. */ -int alloc_contig_frozen_range_noprof (unsigned long start,unsigned long end, -acr_flags_t alloc_flags,gfp_t gfp_mask) +static int __alloc_contig_frozen_range (unsigned long start,unsigned long end, +acr_flags_t alloc_flags,gfp_t gfp_mask, +unsigned long user_addr) {const unsigned int order =ilog2 (end - start);unsigned long outer_start, outer_end;
@@ -7127,7 +7128,7 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,struct page *head =pfn_to_page (start);check_new_pages (head, order); -prep_new_page (head, order, gfp_mask,0 ,USER_ADDR_NONE ); +prep_new_page (head, order, gfp_mask,0 , user_addr); }else { ret = -EINVAL ; WARN(true, "PFN range: requested [%lu, %lu), allocated [%lu, %lu)\n",
@@ -7137,6 +7138,13 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,undo_isolate_page_range (start, end);return ret; } + +int alloc_contig_frozen_range_noprof (unsigned long start,unsigned long end, +acr_flags_t alloc_flags,gfp_t gfp_mask) +{ +return __alloc_contig_frozen_range (start, end, alloc_flags, gfp_mask, +USER_ADDR_NONE ); +}EXPORT_SYMBOL (alloc_contig_frozen_range_noprof);/**
@@ -7255,8 +7263,9 @@ static bool zone_spans_last_pfn(const struct zone *zone,* * Return: pointer to contiguous frozen pages on success, or NULL if not successful. */ -struct page *alloc_contig_frozen_pages_noprof (unsigned long nr_pages, -gfp_t gfp_mask,int nid,nodemask_t *nodemask) +struct page *alloc_contig_frozen_pages_user_noprof (unsigned long nr_pages, +gfp_t gfp_mask,int nid,nodemask_t *nodemask, +unsigned long user_addr) {unsigned long ret, pfn, flags;struct zonelist *zonelist;
@@ -7284,10 +7293,11 @@ struct page *alloc_contig_frozen_pages_noprof(unsigned long nr_pages,* win the race and cause allocation to fail. */ spin_unlock_irqrestore (&zone->lock, flags); - ret =alloc_contig_frozen_range_noprof (pfn, + ret =__alloc_contig_frozen_range (pfn, pfn + nr_pages,ACR_FLAGS_NONE , - gfp_mask); + gfp_mask, + user_addr);if (!ret)return pfn_to_page (pfn);spin_lock_irqsave (&zone->lock, flags);
@@ -7309,6 +7319,14 @@ struct page *alloc_contig_frozen_pages_noprof(unsigned long nr_pages, }return NULL ; } +EXPORT_SYMBOL (alloc_contig_frozen_pages_user_noprof); + +struct page *alloc_contig_frozen_pages_noprof (unsigned long nr_pages, +gfp_t gfp_mask,int nid,nodemask_t *nodemask) +{ +return alloc_contig_frozen_pages_user_noprof (nr_pages, gfp_mask, nid, + nodemask,USER_ADDR_NONE ); +}EXPORT_SYMBOL (alloc_contig_frozen_pages_noprof);/**
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 9/37 mm: hugetlb: thread user_addr through gigantic page allocation
In reply to Michael S. TsirkinThread the user_addr parameter through alloc_gigantic_frozen_folio so that gigantic page allocations can benefit from cache-friendly zeroing.
--- mm/hugetlb.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
mm/hugetlb.c +8 -5
@@ -1375,7 +1375,7 @@ static struct folio *dequeue_hugetlb_folio_vma(struct hstate *h,#if defined(CONFIG_ARCH_HAS_GIGANTIC_PAGE ) && defined(CONFIG_CONTIG_ALLOC )static struct folio *alloc_gigantic_frozen_folio (int order,gfp_t gfp_mask, -int nid,nodemask_t *nodemask) +int nid,nodemask_t *nodemask,unsigned long addr) {struct folio *folio;
@@ -1386,13 +1386,15 @@ static struct folio *alloc_gigantic_frozen_folio(int order, gfp_t gfp_mask,if (hugetlb_cma_exclusive_alloc ())return NULL ; - folio = (struct folio *)alloc_contig_frozen_pages (1 << order, gfp_mask, - nid, nodemask); + folio = (struct folio *)alloc_contig_frozen_pages_user (1 << order, + gfp_mask, + nid, nodemask, + addr); return folio; }#else /* !CONFIG_ARCH_HAS_GIGANTIC_PAGE || !CONFIG_CONTIG_ALLOC */ static struct folio *alloc_gigantic_frozen_folio (int order,gfp_t gfp_mask,int nid, -nodemask_t *nodemask) +nodemask_t *nodemask,unsigned long addr) {return NULL ; }
@@ -1862,7 +1864,8 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, nid =numa_mem_id ();if (order_is_gigantic (order)) - folio =alloc_gigantic_frozen_folio (order, gfp_mask, nid, nmask); + folio =alloc_gigantic_frozen_folio (order, gfp_mask, nid, nmask, + addr);else folio =alloc_buddy_frozen_folio (order, gfp_mask, nid, nmask, node_alloc_noretry, addr);
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 10/37 mm: add folio_zero_user stub for configs without THP/HUGETLBFS
In reply to Michael S. Tsirkinfolio_zero_user() is defined in mm/memory.c under CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS. A subsequent patch will call it from post_alloc_hook() for all user page zeroing, so configs without THP or HUGETLBFS will need a stub.
Add a stub that uses clear_user_highpages() with aligned addr_hint.
Without THP/HUGETLBFS, only order-0 user pages are allocated, so the locality optimization in the real folio_zero_user() (zero near the faulting address last) is not needed.
This also matches what vma_alloc_zeroed_movable_folio currently does.
--- include/linux/mm.h | 2 ++ mm/memory.c | 9 +++++++++ 2 files changed, 11 insertions(+)
include/linux/mm.h +2 -0
@@ -5070,6 +5070,8 @@ long copy_folio_from_user(struct folio *dst_folio,const void __user *usr_src, bool allow_pagefault); +#else /* !CONFIG_TRANSPARENT_HUGEPAGE && !CONFIG_HUGETLBFS */ +void folio_zero_user (struct folio *folio,unsigned long addr_hint);#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS */ #if MAX_NUMNODES >1
mm/memory.c +9 -0
@@ -7546,6 +7546,15 @@ long copy_folio_from_user(struct folio *dst_folio, }#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_HUGETLBFS */ +#if !defined(CONFIG_TRANSPARENT_HUGEPAGE ) && !defined(CONFIG_HUGETLBFS ) +void folio_zero_user (struct folio *folio,unsigned long addr_hint) +{ +unsigned long base =ALIGN_DOWN (addr_hint,folio_size (folio)); + +clear_user_highpages (&folio->page, base,folio_nr_pages (folio)); +} +#endif +#if defined(CONFIG_SPLIT_PTE_PTLOCKS ) &&ALLOC_SPLIT_PTLOCKS static struct kmem_cache *page_ptl_cachep;
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 11/37 mm: page_alloc: move prep_compound_page before post_alloc_hook
In reply to Michael S. TsirkinMove prep_compound_page() before post_alloc_hook() in prep_new_page().
The next patch adds a folio_zero_user() call to post_alloc_hook(), which uses folio_nr_pages() to determine how many pages to zero. Without compound metadata set up first, folio_nr_pages() returns 1 for higher-order allocations, so only the first page would be zeroed.
All other operations in post_alloc_hook() (arch_alloc_page, KASAN, debug, page owner, etc.) use raw page pointers with explicit order counts and are unaffected by this reordering.
Also reorder compaction_alloc_noprof() for consistency. Compaction currently passes USER_ADDR_NONE so folio_zero_user() is not called there, but keeping the same ordering avoids a future tripping hazard.
--- mm/compaction.c | 4 ++-- mm/page_alloc.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
mm/compaction.c +2 -2
@@ -1849,10 +1849,10 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long daset_page_private (&freepage[size], start_order); } dst = (struct folio *)freepage; -post_alloc_hook (&dst->page, order, __GFP_MOVABLE,USER_ADDR_NONE ); -set_page_refcounted (&dst->page);if (order)prep_compound_page (&dst->page, order); +post_alloc_hook (&dst->page, order, __GFP_MOVABLE,USER_ADDR_NONE ); +set_page_refcounted (&dst->page); cc->nr_freepages -=1 << order; cc->nr_migratepages -=1 << order;return page_rmappable_folio (&dst->page);
mm/page_alloc.c +2 -2
@@ -1877,11 +1877,11 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags unsigned int alloc_flags, unsigned long user_addr) { -post_alloc_hook (page ,order ,gfp_flags ,user_addr ); -if (order && (gfp_flags & __GFP_COMP))prep_compound_page (page, order); +post_alloc_hook (page, order, gfp_flags, user_addr); +/* * page is set pfmemalloc when ALLOC_NO_WATERMARKS was necessary to * allocate the page. The expectation is that the caller is taking
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 12/37 mm: use folio_zero_user for user pages in post_alloc_hook
In reply to Michael S. TsirkinWhen post_alloc_hook() needs to zero a page for an explicit __GFP_ZERO allocation for a user page (user_addr is set), use folio_zero_user() instead of kernel_init_pages(). This zeros near the faulting address last, keeping those cachelines hot for the impending
user access.
folio_zero_user() is only used for explicit __GFP_ZERO, not for init_on_alloc. On architectures with virtually-indexed caches (e.g., ARM), clear_user_highpage() performs per-line cache operations; using it for init_on_alloc would add overhead that kernel_init_pages() avoids (the page fault path flushes the cache at PTE installation time regardless).
No functional change yet: current callers do not pass __GFP_ZERO for user pages (they zero at the callsite instead). Subsequent patches will convert them.
--- mm/page_alloc.c | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-)
mm/page_alloc.c +32 -3
@@ -1864,9 +1864,38 @@ inline void post_alloc_hook(struct page *page, unsigned int order,for (i =0 ; i !=1 << order; ++i)page_kasan_tag_reset (page + i); } -/* If memory is still not initialized, initialize it now. */ -if (init) -kernel_init_pages (page,1 << order); +/* +* On architectures with cache aliasing, pages zeroed via the +* kernel direct map (e.g. init_on_free) must be re-zeroed +* through a user-congruent mapping. Host-zeroed pages +* (zeroed flag) don't need this: physical RAM is clean. +*/ +if (!init && (gfp_flags & __GFP_ZERO) && + user_addr !=USER_ADDR_NONE && +user_alloc_needs_zeroing ()) + init = true; +/* +* If memory is still not initialized, initialize it now. +* When __GFP_ZERO was explicitly requested and user_addr is set, +* use folio_zero_user() which zeros near the faulting address +* last, keeping those cachelines hot. For init_on_alloc, use +* kernel_init_pages() to avoid unnecessary cache flush overhead +* on architectures with virtually-indexed caches. +*/ +if (init) { +if ((gfp_flags & __GFP_ZERO) && user_addr !=USER_ADDR_NONE ) { +/* +* folio_zero_user relies on folio_nr_pages which +* requires __GFP_COMP for order > 0. All user folio +* allocations set __GFP_COMP via __folio_alloc. +* user_addr != USER_ADDR_NONE implies sleepable +* context (user page fault). +*/ +VM_WARN_ON_ONCE (order && !(gfp_flags & __GFP_COMP)); +folio_zero_user (page_folio (page), user_addr); + }else +kernel_init_pages (page,1 << order); + }set_page_owner (page, order, gfp_flags);page_table_check_alloc (page, order);
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 13/37 mm: use __GFP_ZERO in vma_alloc_zeroed_movable_folio
In reply to Michael S. TsirkinNow that post_alloc_hook() handles cache-friendly user page zeroing via folio_zero_user(), convert vma_alloc_zeroed_movable_folio() to pass __GFP_ZERO instead of zeroing at the callsite.
--- include/linux/highmem.h | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
include/linux/highmem.h +2 -7
@@ -320,13 +320,8 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); -if (folio &&user_alloc_needs_zeroing ()) -clear_user_highpage (&folio->page, vaddr); - -return folio; +return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO, +0 , vma, vaddr); }#endif
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 14/37 mm: remove arch vma_alloc_zeroed_movable_folio overrides
In reply to Michael S. TsirkinNow that the generic vma_alloc_zeroed_movable_folio() uses __GFP_ZERO, the arch-specific macros on alpha, m68k, s390, and x86 that did the same thing are redundant. Remove them.
arm64 is not affected: it has a real function override that handles MTE tag zeroing, not just __GFP_ZERO.
--- arch/alpha/include/asm/page.h | 3 --- arch/m68k/include/asm/page_no.h | 3 --- arch/s390/include/asm/page.h | 3 --- arch/x86/include/asm/page.h | 3 --- include/linux/highmem.h | 8 +++++--- 5 files changed, 5 insertions(+), 15 deletions(-)
arch/alpha/include/asm/page.h +0 -3
@@ -12,9 +12,6 @@extern void clear_page (void *page); -#define vma_alloc_zeroed_movable_folio (vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr) -extern void copy_page (void * _to,void * _from);#define copy_user_page (to, from, vaddr, pg) copy_page(to, from)
arch/m68k/include/asm/page_no.h +0 -3
@@ -12,9 +12,6 @@ extern unsigned long memory_end;#define copy_user_page (to, from, vaddr, pg) copy_page(to, from) -#define vma_alloc_zeroed_movable_folio (vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr) -#define __pa (vaddr) ((unsigned long)(vaddr))#define __va (paddr) ((void *)((unsigned long)(paddr)))
arch/s390/include/asm/page.h +0 -3
@@ -67,9 +67,6 @@ static inline void copy_page(void *to, void *from)#define copy_user_page (to, from, vaddr, pg) copy_page(to, from) -#define vma_alloc_zeroed_movable_folio (vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr) -#ifdef CONFIG_STRICT_MM_TYPECHECKS #define STRICT_MM_TYPECHECKS #endif
arch/x86/include/asm/page.h +0 -3
@@ -28,9 +28,6 @@ static inline void copy_user_page(void *to, void *from, unsigned long vaddr,copy_page (to, from); } -#define vma_alloc_zeroed_movable_folio (vma, vaddr) \ - vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr) -#ifndef __pa#define __pa (x) __phys_addr((unsigned long)(x))#endif
include/linux/highmem.h +5 -3
@@ -303,7 +303,6 @@ static inline void clear_user_highpages(struct page *page, unsigned long vaddr,#endif } -#ifndef vma_alloc_zeroed_movable_folio/** * vma_alloc_zeroed_movable_folio - Allocate a zeroed page for a VMA. * @vma: The VMA the page is to be allocated for.
@@ -317,12 +316,15 @@ static inline void clear_user_highpages(struct page *page, unsigned long vaddr,* we are out of memory. */ static inline -struct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma, +struct folio *vma_alloc_zeroed_movable_folio_noprof (struct vm_area_struct *vma,unsigned long vaddr) { -return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO, +return vma_alloc_folio_noprof (GFP_HIGHUSER_MOVABLE | __GFP_ZERO,0 , vma, vaddr); } +#ifndef vma_alloc_zeroed_movable_folio +#define vma_alloc_zeroed_movable_folio (...) \ + alloc_hooks(vma_alloc_zeroed_movable_folio_noprof(__VA_ARGS__))#endif static inline void clear_highpage (struct page *page)
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 15/37 mm: alloc_anon_folio: pass raw fault address to vma_alloc_folio
In reply to Michael S. TsirkinPass vmf->address directly instead of ALIGN_DOWN(vmf->address, ...). vma_alloc_folio_noprof now aligns internally for NUMA interleave, and post_alloc_hook will use the raw address for cache-friendly zeroing via folio_zero_user().
--- mm/memory.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
mm/memory.c +1 -2
@@ -5252,8 +5252,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)/* Try allocating the highest of the remaining orders. */ gfp =vma_thp_gfp_mask (vma);while (orders) { - addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << order); - folio =vma_alloc_folio (gfp, order, vma, addr); + folio =vma_alloc_folio (gfp, order, vma, vmf->address);if (folio) {if (mem_cgroup_charge (folio, vma->vm_mm, gfp)) {count_mthp_stat (order,MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE );
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 16/37 mm: alloc_swap_folio: pass raw fault address to vma_alloc_folio
In reply to Michael S. TsirkinSame change as the previous patch but for alloc_swap_folio:
pass vmf->address directly instead of ALIGN_DOWN(vmf->address, ...).
--- mm/memory.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
mm/memory.c +1 -2
@@ -4734,8 +4734,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)/* Try allocating the highest of the remaining orders. */ gfp =vma_thp_gfp_mask (vma);while (orders) { - addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << order); - folio =vma_alloc_folio (gfp, order, vma, addr); + folio =vma_alloc_folio (gfp, order, vma, vmf->address);if (folio) {if (!mem_cgroup_swapin_charge_folio (folio, vma->vm_mm, gfp, entry))
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 17/37 mm: use __GFP_ZERO in alloc_anon_folio
In reply to Michael S. TsirkinConvert alloc_anon_folio() to pass __GFP_ZERO instead of zeroing at the callsite. post_alloc_hook uses the fault address passed through vma_alloc_folio for cache-friendly zeroing.
Note: with __GFP_ZERO, the folio is zeroed before mem_cgroup_charge(). If the charge fails, the zeroing work is wasted. Previously zeroing was done after a successful charge. This is inherent to moving zeroing into the allocator.
--- mm/memory.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-)
mm/memory.c +1 -10
@@ -5249,7 +5249,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto fallback;/* Try allocating the highest of the remaining orders. */ - gfp =vma_thp_gfp_mask (vma); + gfp =vma_thp_gfp_mask (vma) | __GFP_ZERO;while (orders) { folio =vma_alloc_folio (gfp, order, vma, vmf->address);if (folio) {
@@ -5259,15 +5259,6 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto next; }folio_throttle_swaprate (folio, gfp); -/* -* When a folio is not zeroed during allocation -* (__GFP_ZERO not used) or user folios require special -* handling, folio_zero_user() is used to make sure -* that the page corresponding to the faulting address -* will be hot in the cache after zeroing. -*/ -if (user_alloc_needs_zeroing ()) -folio_zero_user (folio, vmf->address);return folio; } next:
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 18/37 mm: vma_alloc_anon_folio_pmd: pass raw fault address to vma_alloc_folio
In reply to Michael S. TsirkinNow that vma_alloc_folio aligns the address internally, drop the redundant HPAGE_PMD_MASK alignment at the callsite.
--- mm/huge_memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/huge_memory.c +1 -1
@@ -1337,7 +1337,7 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma,const int order =HPAGE_PMD_ORDER ;struct folio *folio; - folio =vma_alloc_folio (gfp, order, vma, addr &HPAGE_PMD_MASK ); + folio =vma_alloc_folio (gfp, order, vma, addr);if (unlikely (!folio)) {count_vm_event (THP_FAULT_FALLBACK );
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 19/37 mm: use __GFP_ZERO in vma_alloc_anon_folio_pmd
In reply to Michael S. TsirkinConvert vma_alloc_anon_folio_pmd() to pass __GFP_ZERO instead of zeroing at the callsite. post_alloc_hook uses the fault address passed through vma_alloc_folio for cache-friendly zeroing.
Note: with __GFP_ZERO, the folio is zeroed before mem_cgroup_charge(). If the charge fails, the zeroing work is wasted. Previously zeroing was done after a successful charge. This is inherent to moving zeroing into the allocator.
--- mm/huge_memory.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)
mm/huge_memory.c +2 -10
@@ -1333,7 +1333,7 @@ EXPORT_SYMBOL_GPL(thp_get_unmapped_area);static struct folio *vma_alloc_anon_folio_pmd (struct vm_area_struct *vma,unsigned long addr) { -gfp_t gfp =vma_thp_gfp_mask (vma); +gfp_t gfp =vma_thp_gfp_mask (vma) | __GFP_ZERO;const int order =HPAGE_PMD_ORDER ;struct folio *folio;
@@ -1356,17 +1356,9 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, }folio_throttle_swaprate (folio, gfp); -/* -* When a folio is not zeroed during allocation (__GFP_ZERO not used) -* or user folios require special handling, folio_zero_user() is used to -* make sure that the page corresponding to the faulting address will be -* hot in the cache after zeroing. -*/ -if (user_alloc_needs_zeroing ()) -folio_zero_user (folio, addr);/* * The memory barrier inside __folio_mark_uptodate makes sure that -* folio_zero_user writes become visible before the set_pmd_at() +* page zeroing becomes visible before the set_pmd_at() * write. */ __folio_mark_uptodate (folio);
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 20/37 mm: hugetlb: add gfp parameter and skip zeroing for zeroed pages
In reply to Michael S. TsirkinAdd a gfp_t parameter to alloc_hugetlb_folio(). When __GFP_ZERO is set, the function guarantees the returned folio is zeroed:
- Fresh allocations (buddy or gigantic): zeroed by post_alloc_hook via __GFP_ZERO, HPG_zeroed set by alloc_surplus_hugetlb_folio.
- Pool pages with HPG_zeroed set: already zeroed, skip.
- Pool pages without HPG_zeroed: zeroed via folio_zero_user().
The address parameter is renamed to user_addr; the function aligns it internally for reservation and NUMA policy lookups. For pages that need zeroing, user_addr is passed to folio_zero_user() for cache-friendly zeroing near the faulting subpage. All callers pass a page-aligned address; the hugetlb_no_page caller passes vmf->real_address & PAGE_MASK for consistency.
HPG_zeroed (stored in hugetlb folio->private bits) tracks known-zero pool pages. It is set when alloc_surplus_hugetlb_folio allocates with __GFP_ZERO, and cleared in free_huge_folio when the page returns to the pool after userspace use.
--- fs/hugetlbfs/inode.c | 3 +-- include/linux/hugetlb.h | 5 ++++- mm/hugetlb.c | 42 +++++++++++++++++++++++++++++------------ 3 files changed, 35 insertions(+), 15 deletions(-)
fs/hugetlbfs/inode.c +1 -2
@@ -810,13 +810,12 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,* folios in these areas, we need to consume the reserves * to keep reservation accounting consistent. */ - folio =alloc_hugetlb_folio (&pseudo_vma, addr, false); + folio =alloc_hugetlb_folio (&pseudo_vma, addr, false, __GFP_ZERO);if (IS_ERR (folio)) {mutex_unlock (&hugetlb_fault_mutex_table[hash]); error =PTR_ERR (folio); goto out; } -folio_zero_user (folio, addr);__folio_mark_uptodate (folio); error =hugetlb_add_to_page_cache (folio, mapping, index);if (unlikely (error)) {
include/linux/hugetlb.h +4 -1
@@ -599,6 +599,7 @@ enum hugetlb_page_flags { HPG_vmemmap_optimized, HPG_raw_hwp_unreliable, HPG_cma, + HPG_zeroed, __NR_HPAGEFLAGS, };
@@ -659,6 +660,7 @@ HPAGEFLAG(Freed, freed) HPAGEFLAG(VmemmapOptimized, vmemmap_optimized) HPAGEFLAG(RawHwpUnreliable, raw_hwp_unreliable) HPAGEFLAG(Cma, cma) +HPAGEFLAG(Zeroed, zeroed)#ifdef CONFIG_HUGETLB_PAGE
@@ -706,7 +708,8 @@ int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *list);int replace_free_hugepage_folios (unsigned long start_pfn,unsigned long end_pfn);void wait_for_freed_hugetlb_folios (void );struct folio *alloc_hugetlb_folio (struct vm_area_struct *vma, -unsigned long addr,bool cow_from_owner); +unsigned long user_addr,bool cow_from_owner, +gfp_t gfp);struct folio *alloc_hugetlb_folio_nodemask (struct hstate *h,int preferred_nid,nodemask_t *nmask,gfp_t gfp_mask,bool allow_alloc_fallback);
mm/hugetlb.c +30 -12
130 lines not on this page · raw message
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 21/37 mm: memfd: skip zeroing for zeroed hugetlb pool pages
In reply to Michael S. TsirkinAdd bool *zeroed output to alloc_hugetlb_folio_reserve() so callers can check whether the pool page is known-zero. memfd's memfd_alloc_folio() uses this to skip the explicit folio_zero_user() when the page is already zero.
This avoids redundant zeroing for memfd hugetlb pages that were pre-allocated into the pool and never mapped to userspace.
--- include/linux/cma.h | 3 ++- include/linux/hugetlb.h | 6 ++++-- mm/cma.c | 6 ++++-- mm/hugetlb.c | 11 +++++++++-- mm/hugetlb_cma.c | 4 ++-- mm/memfd.c | 14 ++++++++------ 6 files changed, 29 insertions(+), 15 deletions(-)
include/linux/cma.h +2 -1
@@ -53,7 +53,8 @@ extern bool cma_release(struct cma *cma, const struct page *pages, unsigned longstruct page *cma_alloc_frozen (struct cma *cma,unsigned long count,unsigned int align,bool no_warn); -struct page *cma_alloc_frozen_compound (struct cma *cma,unsigned int order); +struct page *cma_alloc_frozen_compound (struct cma *cma,unsigned int order, +gfp_t caller_gfp);bool cma_release_frozen (struct cma *cma,const struct page *pages,unsigned long count);
include/linux/hugetlb.h +4 -2
@@ -714,7 +714,8 @@ struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, nodemask_t *nmask, gfp_t gfp_mask, bool allow_alloc_fallback);struct folio *alloc_hugetlb_folio_reserve (struct hstate *h,int preferred_nid, -nodemask_t *nmask,gfp_t gfp_mask); +nodemask_t *nmask,gfp_t gfp_mask, +bool *zeroed);int hugetlb_add_to_page_cache (struct folio *folio,struct address_space *mapping,pgoff_t idx);
@@ -1134,7 +1135,8 @@ static inline void wait_for_freed_hugetlb_folios(void)static inline struct folio *alloc_hugetlb_folio_reserve (struct hstate *h,int preferred_nid, -nodemask_t *nmask,gfp_t gfp_mask) +nodemask_t *nmask,gfp_t gfp_mask, +bool *zeroed) {return NULL ; }
mm/cma.c +4 -2
@@ -924,9 +924,11 @@ struct page *cma_alloc_frozen(struct cma *cma, unsigned long count,return __cma_alloc_frozen (cma, count, align, gfp); } -struct page *cma_alloc_frozen_compound (struct cma *cma,unsigned int order) +struct page *cma_alloc_frozen_compound (struct cma *cma,unsigned int order, +gfp_t caller_gfp) { -gfp_t gfp =GFP_KERNEL | __GFP_COMP | __GFP_NOWARN; +gfp_t gfp =GFP_KERNEL | __GFP_COMP | __GFP_NOWARN | + (caller_gfp & __GFP_ZERO);return __cma_alloc_frozen (cma,1 << order, order, gfp); }
mm/hugetlb.c +9 -2
@@ -2208,7 +2208,7 @@ struct folio *alloc_buddy_hugetlb_folio_with_mpol(struct hstate *h, }struct folio *alloc_hugetlb_folio_reserve (struct hstate *h,int preferred_nid, -nodemask_t *nmask,gfp_t gfp_mask) +nodemask_t *nmask,gfp_t gfp_mask,bool *zeroed) {struct folio *folio;
@@ -2224,6 +2224,12 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid, h->resv_huge_pages--;spin_unlock_irq (&hugetlb_lock); + +if (zeroed && folio) { + *zeroed =folio_test_hugetlb_zeroed (folio); +folio_clear_hugetlb_zeroed (folio); + } +return folio; }
@@ -2308,7 +2314,8 @@ static int gather_surplus_pages(struct hstate *h, long delta)* It is okay to use NUMA_NO_NODE because we use numa_mem_id() * down the road to pick the current node if that is the case. */ - folio =alloc_surplus_hugetlb_folio (h,htlb_alloc_mask (h), + folio =alloc_surplus_hugetlb_folio (h, +htlb_alloc_mask (h),NUMA_NO_NODE , &alloc_nodemask,USER_ADDR_NONE );if (!folio) {
mm/hugetlb_cma.c +2 -2
@@ -35,14 +35,14 @@ struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask,return NULL ;if (hugetlb_cma[nid]) - page =cma_alloc_frozen_compound (hugetlb_cma[nid], order); + page =cma_alloc_frozen_compound (hugetlb_cma[nid], order, gfp_mask);if (!page && !(gfp_mask & __GFP_THISNODE)) {for_each_node_mask (node, *nodemask) {if (node == nid || !hugetlb_cma[node])continue ; - page =cma_alloc_frozen_compound (hugetlb_cma[node], order); + page =cma_alloc_frozen_compound (hugetlb_cma[node], order, gfp_mask);if (page)break ; }
mm/memfd.c +8 -6
@@ -69,6 +69,7 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx)#ifdef CONFIG_HUGETLB_PAGE struct folio *folio;gfp_t gfp_mask; +bool zeroed;if (is_file_hugepages (memfd)) {/*
@@ -93,17 +94,18 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx) folio =alloc_hugetlb_folio_reserve (h,numa_node_id (),NULL , - gfp_mask); + gfp_mask, + &zeroed);if (folio) {u32 hash;/* -* Zero the folio to prevent information leaks to userspace. -* Use folio_zero_user() which is optimized for huge/gigantic -* pages. Pass 0 as addr_hint since this is not a faulting path -* and we don't have a user virtual address yet. +* Zero the folio to prevent information leaks to +* userspace. Skip if the pool page is known-zero +* (HPG_zeroed set during pool pre-allocation). */ -folio_zero_user (folio,0 ); +if (!zeroed) +folio_zero_user (folio,0 );/* * Mark the folio uptodate before adding to page cache,
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 22/37 mm: page_reporting: skip redundant zeroing of host-zeroed reported pages
In reply to Michael S. TsirkinWhen a guest reports free pages to the hypervisor via the page reporting framework (used by virtio-balloon and hv_balloon), the host typically zeros those pages when reclaiming their backing memory. However, when those pages are later allocated in the guest, post_alloc_hook() unconditionally zeros them again if __GFP_ZERO is set. This double-zeroing is wasteful, especially for large pages.
Avoid redundant zeroing:
- Add a host_zeroes_pages flag to page_reporting_dev_info, allowing drivers to declare that their host zeros reported pages on reclaim. A static key (page_reporting_host_zeroes) gates the fast path.
- Add PG_zeroed page flag (sharing PG_private bit) to mark pages that have been zeroed by the host. Set it in page_reporting_drain() after the host reports them.
- Thread the zeroed bool through rmqueue -> prep_new_page -> post_alloc_hook, where it skips redundant zeroing for __GFP_ZERO allocations.
Currently the PG_zeroed hint can be lost when pages are split (expand) or merged in the buddy allocator. This is
harmless: losing the hint just means the page gets re-zeroed, which is correct but suboptimal. Follow-up patches propagate PG_zeroed across splits and merges to preserve the hint on common paths.
No driver sets host_zeroes_pages yet; a follow-up patch to virtio_balloon is needed to opt in.
PG_zeroed pages may pass through PCP lists before being freed. This is safe: __free_pages_prepare clears all PAGE_FLAGS_CHECK_AT_PREP flags (including PG_zeroed/PG_private) before the page re-enters the buddy allocator.
--- include/linux/page-flags.h | 9 +++++ include/linux/page_reporting.h | 3 ++ mm/compaction.c | 6 ++- mm/internal.h | 2 +- mm/page_alloc.c | 68 +++++++++++++++++++++++----------- mm/page_reporting.c | 14 ++++++- mm/page_reporting.h | 12 ++++++ 7 files changed, 88 insertions(+), 26 deletions(-)
include/linux/page-flags.h +9 -0
@@ -135,6 +135,8 @@ enum pageflags { PG_swapcache = PG_owner_priv_1,/* Swap page: swp_entry_t in private */ /* Some filesystems */ PG_checked = PG_owner_priv_1, +/* Page contents are known to be zero */ + PG_zeroed = PG_private,/* * Depending on the way an anonymous folio can be mapped into a page
@@ -673,6 +675,13 @@ FOLIO_TEST_CLEAR_FLAG_FALSE(young) FOLIO_FLAG_FALSE(idle)#endif +/* +* PageZeroed() tracks pages known to be zero. The allocator +* uses this to skip redundant zeroing in post_alloc_hook(). +*/ +__PAGEFLAG(Zeroed, zeroed, PF_NO_COMPOUND) +#define __PG_ZEROED (1UL << PG_zeroed) +/* * PageReported() is used to track reported free pages within the Buddy * allocator. We can use the non-atomic version of the test and set
include/linux/page_reporting.h +3 -0
@@ -14,6 +14,9 @@ struct page_reporting_dev_info {int (*report)(struct page_reporting_dev_info *prdev,struct scatterlist *sg,unsigned int nents); +/* If true, host zeros reported pages on reclaim */ +bool host_zeroes_pages; +/* work struct for processing reports */ struct delayed_work work;
mm/compaction.c +4 -2
@@ -82,7 +82,8 @@ static inline bool is_via_compact_memory(int order) { return false; }static struct page *mark_allocated_noprof (struct page *page,unsigned int order,gfp_t gfp_flags) { -post_alloc_hook (page, order, __GFP_MOVABLE,USER_ADDR_NONE ); +__ClearPageZeroed (page); +post_alloc_hook (page, order, __GFP_MOVABLE, false,USER_ADDR_NONE );set_page_refcounted (page);return page; }
@@ -1849,9 +1850,10 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long daset_page_private (&freepage[size], start_order); } dst = (struct folio *)freepage; +__ClearPageZeroed (&dst->page);if (order)prep_compound_page (&dst->page, order); -post_alloc_hook (&dst->page, order, __GFP_MOVABLE,USER_ADDR_NONE ); +post_alloc_hook (&dst->page, order, __GFP_MOVABLE, false,USER_ADDR_NONE );set_page_refcounted (&dst->page); cc->nr_freepages -=1 << order; cc->nr_migratepages -=1 << order;
mm/internal.h +1 -1
@@ -928,7 +928,7 @@ static inline void init_compound_tail(struct page *tail, }void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags, -unsigned long user_addr); +bool zeroed,unsigned long user_addr);extern bool free_pages_prepare (struct page *page,unsigned int order);extern int user_min_free_kbytes;
mm/page_alloc.c +46 -22
214 lines not on this page · raw message
mm/page_reporting.c +13 -1
@@ -50,6 +50,8 @@ EXPORT_SYMBOL_GPL(page_reporting_order);#define PAGE_REPORTING_DELAY (2 * HZ)static struct page_reporting_dev_info __rcu *pr_dev_info__read_mostly ; +DEFINE_STATIC_KEY_FALSE (page_reporting_host_zeroes); +enum {PAGE_REPORTING_IDLE =0 ,PAGE_REPORTING_REQUESTED ,
@@ -129,8 +131,11 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,* report on the new larger page when we make our way * up to that higher order. */ -if (PageBuddy (page) &&buddy_order (page) == order) +if (PageBuddy (page) &&buddy_order (page) == order) {__SetPageReported (page); +if (page_reporting_host_zeroes_pages ()) +__SetPageZeroed (page); + } }while ((sg =sg_next (sg)));/* reinitialize scatterlist now that it is empty */
@@ -390,6 +395,10 @@ int page_reporting_register(struct page_reporting_dev_info *prdev)/* Assign device to allow notifications */ rcu_assign_pointer (pr_dev_info, prdev); +/* enable zeroed page optimization if host zeroes reported pages */ +if (prdev->host_zeroes_pages) +static_branch_enable (&page_reporting_host_zeroes); +/* enable page reporting notification */ if (!static_key_enabled (&page_reporting_enabled)) {static_branch_enable (&page_reporting_enabled);
@@ -414,6 +423,9 @@ void page_reporting_unregister(struct page_reporting_dev_info *prdev)/* Flush any existing work, and lock it out */ cancel_delayed_work_sync (&prdev->work); + +if (prdev->host_zeroes_pages) +static_branch_disable (&page_reporting_host_zeroes); }mutex_unlock (&page_reporting_mutex);
mm/page_reporting.h +12 -0
@@ -15,6 +15,13 @@ DECLARE_STATIC_KEY_FALSE(page_reporting_enabled);extern unsigned int page_reporting_order;void __page_reporting_notify (void ); +DECLARE_STATIC_KEY_FALSE (page_reporting_host_zeroes); + +static inline bool page_reporting_host_zeroes_pages (void ) +{ +return static_branch_unlikely (&page_reporting_host_zeroes); +} +static inline bool page_reported (struct page *page) { return static_branch_unlikely(&page_reporting_enabled) &&
@@ -46,6 +53,11 @@ static inline void page_reporting_notify_free(unsigned int order)#else /* CONFIG_PAGE_REPORTING */ #define page_reported (_page) false +static inline bool page_reporting_host_zeroes_pages (void ) +{ +return false; +} +static inline void page_reporting_notify_free (unsigned int order) { }
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 23/37 mm: page_alloc: use aliasing checks instead of user_alloc_needs_zeroing
In reply to Michael S. TsirkinReplace user_alloc_needs_zeroing() with the direct aliasing checks (cpu_dcache_is_aliasing() || cpu_icache_is_aliasing()) in the post_alloc_hook aliasing guard.
user_alloc_needs_zeroing() includes a !init_on_alloc term that means "allocator didn't zero this page." But in this guard's context (!zeroed && !init && __GFP_ZERO), we already know the page is zero; init incorporates init_on_alloc via want_init_on_alloc(). The only question left is whether the cache architecture needs the data re-zeroed through a congruent mapping, which is purely cpu_dcache_is_aliasing() || cpu_icache_is_aliasing().
On non-aliasing architectures with init_on_free=true and init_on_alloc=false, this avoids a redundant re-zero of an already-zero page.
Note on PowerPC: PowerPC overrides clear_user_page to call flush_dcache_page after clear_page, but on freshly allocated pages PG_dcache_clean is already clear (cleared by __free_pages_prepare), so flush_dcache_page is a no-op. Skipping this here thus has no effect.
--- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c +1 -1
@@ -1883,7 +1883,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order,*/ if (!zeroed && !init && (gfp_flags & __GFP_ZERO) && user_addr !=USER_ADDR_NONE && -user_alloc_needs_zeroing ()) + (cpu_dcache_is_aliasing () ||cpu_icache_is_aliasing ())) init = true;/* * If memory is still not initialized, initialize it now.
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 24/37 mm: page_reporting: add per-page zeroed bitmap for host feedback
In reply to Michael S. TsirkinThe host may skip zeroing some reported pages (e.g., due to alignment constraints or bounce buffer fallback in QEMU). Currently, when host_zeroes_pages is set, all reported pages are unconditionally marked PG_zeroed - even ones the host did not actually zero.
Add a zeroed_bitmap to page_reporting_dev_info that the report() callback can use to indicate which pages were actually zeroed. The driver's report() callback is responsible for managing the
bitmap: zeroing it before sending pages to the host, then setting bits for pages the host actually zeroed.
page_reporting_drain() checks the bitmap per-page in addition to the global host_zeroes_pages flag.
No driver sets host_zeroes_pages yet, so the static key is off and the bitmap is never read. Behavior is unchanged.
--- include/linux/page_reporting.h | 7 +++++++ mm/page_reporting.c | 8 ++++++-- 2 files changed, 13 insertions(+), 2 deletions(-)
include/linux/page_reporting.h +7 -0
@@ -17,6 +17,13 @@ struct page_reporting_dev_info {/* If true, host zeros reported pages on reclaim */ bool host_zeroes_pages; +/* +* Per-page zeroed status, indexed by scatterlist position. +* The driver's report() callback must clear the bitmap, +* then set bits for pages that were actually zeroed. +*/ +DECLARE_BITMAP (zeroed_bitmap,PAGE_REPORTING_CAPACITY ); +/* work struct for processing reports */ struct delayed_work work;
mm/page_reporting.c +6 -2
@@ -108,6 +108,7 @@ page_reporting_drain(struct page_reporting_dev_info *prdev, struct scatterlist *sgl, unsigned int nents, bool reported) { struct scatterlist *sg = sgl; +unsigned int i =0 ;/* * Drain the now reported pages back into their respective
@@ -122,7 +123,7 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,/* If the pages were not reported due to error skip flagging */ if (!reported) -continue ; + goto next;/* * If page was not commingled with another page we can
@@ -133,9 +134,12 @@ page_reporting_drain(struct page_reporting_dev_info *prdev,*/ if (PageBuddy (page) &&buddy_order (page) == order) {__SetPageReported (page); -if (page_reporting_host_zeroes_pages ()) +if (page_reporting_host_zeroes_pages () && +test_bit (i, prdev->zeroed_bitmap))__SetPageZeroed (page); } +next: + i++; }while ((sg =sg_next (sg)));/* reinitialize scatterlist now that it is empty */
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 25/37 mm: page_alloc: clear PG_zeroed on buddy merge if not both zero
In reply to Michael S. TsirkinWhen two buddy pages merge in __free_one_page(), preserve PG_zeroed on the merged page only if both buddies have the flag set. Otherwise clear it.
The merged page would inherit PG_zeroed, and a later __GFP_ZERO allocation would skip zeroing stale data in the non-zero half.
--- include/linux/page-flags.h | 1 + mm/page_alloc.c | 15 ++++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-)
include/linux/page-flags.h +1 -0
@@ -680,6 +680,7 @@ FOLIO_FLAG_FALSE(idle)* uses this to skip redundant zeroing in post_alloc_hook(). */ __PAGEFLAG(Zeroed, zeroed, PF_NO_COMPOUND) +CLEARPAGEFLAG(Zeroed, zeroed, PF_NO_COMPOUND)#define __PG_ZEROED (1UL << PG_zeroed)/*
mm/page_alloc.c +14 -1
@@ -940,10 +940,14 @@ static inline void __free_one_page(struct page *page,unsigned long buddy_pfn =0 ;unsigned long combined_pfn;struct page *buddy; +bool buddy_zeroed; +bool page_zeroed;bool to_tail;VM_BUG_ON (!zone_is_initialized (zone)); -VM_BUG_ON_PAGE (page->flags.f &PAGE_FLAGS_CHECK_AT_PREP , page); +/* PG_zeroed (aliased to PG_private) is valid on free-list pages */ +VM_BUG_ON_PAGE (page->flags.f & + (PAGE_FLAGS_CHECK_AT_PREP & ~__PG_ZEROED), page);VM_BUG_ON (migratetype ==-1 );VM_BUG_ON_PAGE (pfn & ((1 << order) -1 ), page);
@@ -978,6 +982,8 @@ static inline void __free_one_page(struct page *page, goto done_merging; } + buddy_zeroed =PageZeroed (buddy); +/* * Our buddy is free or it is CONFIG_DEBUG_PAGEALLOC guard page, * merge with it and move up one order.
@@ -996,10 +1002,17 @@ static inline void __free_one_page(struct page *page,change_pageblock_range (buddy, order, migratetype); } + page_zeroed =PageZeroed (page); +__ClearPageZeroed (page); +__ClearPageZeroed (buddy); + combined_pfn = buddy_pfn & pfn; page = page + (combined_pfn - pfn); pfn = combined_pfn; order++; + +if (page_zeroed && buddy_zeroed) +__SetPageZeroed (page); } done_merging:
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 26/37 mm: page_alloc: preserve PG_zeroed in page_del_and_expand
In reply to Michael S. TsirkinPropagate PG_zeroed through buddy splits in page_del_and_expand() and try_to_claim_block(). When a zeroed high-order page is split to satisfy a smaller allocation, the sub-pages placed back on the free lists keep PG_zeroed.
--- mm/page_alloc.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
mm/page_alloc.c +9 -3
@@ -1715,7 +1715,8 @@ struct page *__pageblock_pfn_to_page(unsigned long start_pfn,* -- nyc */ static inline unsigned int expand (struct zone *zone,struct page *page,int low, -int high,int migratetype,bool reported) +int high,int migratetype,bool reported, +bool zeroed) {unsigned int size =1 << high;unsigned int nr_added =0 ;
@@ -1746,6 +1747,8 @@ static inline unsigned int expand(struct zone *zone, struct page *page, int low,*/ if (reported)__SetPageReported (&page[size]); +if (zeroed) +__SetPageZeroed (&page[size]); }return nr_added;
@@ -1757,10 +1760,12 @@ static __always_inline void page_del_and_expand(struct zone *zone, {int nr_pages =1 << high;bool was_reported =page_reported (page); +bool was_zeroed =PageZeroed (page);__del_page_from_free_list (page, zone, high, migratetype); - nr_pages -=expand (zone, page, low, high, migratetype, was_reported); + nr_pages -=expand (zone, page, low, high, migratetype, was_reported, + was_zeroed);account_freepages (zone, -nr_pages, migratetype); }
@@ -2356,11 +2361,12 @@ try_to_claim_block(struct zone *zone, struct page *page,if (current_order >= pageblock_order) {unsigned int nr_added;bool was_reported =page_reported (page); +bool was_zeroed =PageZeroed (page);del_page_from_free_list (page, zone, current_order, block_type);change_pageblock_range (page, current_order, start_type); nr_added =expand (zone, page, order, current_order, start_type, - was_reported); + was_reported, was_zeroed);account_freepages (zone, nr_added, start_type);return page; }
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 27/37 virtio_balloon: submit reported pages as individual buffers
In reply to Michael S. TsirkinSubmit each reported page as a separate virtqueue buffer instead of one buffer with an sg list of all pages. This avoids indirect descriptor allocation (kmalloc in the reporting path) and gives per-page used length feedback from the device.
On error, the already-queued pages are kicked and drained before the error is returned. The caller (page_reporting_drain) then marks the batch as unreported, which is conservative but correct.
Note: if the virtqueue is broken, wait_event on virtqueue_get_buf hangs. This is a pre-existing issue:
the old single-buffer code had the same hang. EVENT_IDX is not a concern: callbacks were never disabled, so the virtqueue manages used_event automatically.
--- drivers/virtio/virtio_balloon.c | 40 +++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 14 deletions(-)
drivers/virtio/virtio_balloon.c +26 -14
@@ -187,7 +187,9 @@ static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq)sg_init_one (&sg, vb->pfns,sizeof (vb->pfns[0 ]) * vb->num_pfns); -/* We should always be able to add one buffer to an empty queue. */ +/* We made sure the vq is large enough so we should always +* be able to add one buffer to an empty queue. +*/ virtqueue_add_outbuf (vq, &sg,1 , vb,GFP_KERNEL );virtqueue_kick (vq);
@@ -202,25 +204,35 @@ static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_istruct virtio_balloon *vb = container_of(pr_dev_info, struct virtio_balloon, pr_dev_info);struct virtqueue *vq = vb->reporting_vq; -unsigned int unused, err; +unsigned int i, err =0 ;/* We should always be able to add these buffers to an empty queue. */ - err =virtqueue_add_inbuf (vq, sg, nents, vb,GFP_NOWAIT ); +for (i =0 ; i < nents; i++) { +struct scatterlist one; -/* -* In the extremely unlikely case that something has occurred and we -* are able to trigger an error we will simply display a warning -* and exit without actually processing the pages. -*/ -if (WARN_ON_ONCE (err)) -return err; +sg_init_table (&one,1 ); +sg_set_page (&one,sg_page (&sg[i]), sg[i].length, + sg[i].offset); + err =virtqueue_add_inbuf (vq, &one,1 , &sg[i],GFP_NOWAIT ); +if (WARN_ON_ONCE (err)) { + nents = i; +break ; + } + } -virtqueue_kick (vq); +if (nents) { +virtqueue_kick (vq); -/* When host has read buffer, this completes via balloon_ack */ -wait_event (vb->acked,virtqueue_get_buf (vq, &unused)); +/* When host has read buffer, this completes via balloon_ack */ +for (i =0 ; i < nents; i++) { +unsigned int unused; -return 0 ; +wait_event (vb->acked, +virtqueue_get_buf (vq, &unused)); + } + } + +return err; } static void set_page_pfns(struct virtio_balloon *vb,
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 28/37 virtio_balloon: disable indirect descriptors
In reply to Michael S. TsirkinA follow-up patch (DEVICE_INIT_ON_INFLATE) adds an inbuf (bitmap) alongside the outbuf (pfns), making total_sg=2. With INDIRECT_DESC negotiated, virtqueue_use_indirect() triggers for total_sg > 1. Balloon never needs more than 2 entries and virtqueue sizes are 128+, so indirect descriptors are unnecessary.
Disabling them avoids a GFP_KERNEL allocation inside virtqueue_add_sgs when VIRTIO_RING_F_INDIRECT_DESC is negotiated. This allocation could trigger OOM reclaim while balloon_lock is held, leading to a deadlock: OOM notifier -> leak_balloon -> balloon_lock.
With single-buffer submissions (previous patch) and no indirect descriptors, virtqueue_add never allocates memory.
--- drivers/virtio/virtio_balloon.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/virtio/virtio_balloon.c +8 -0
@@ -7,6 +7,7 @@*/ #include <linux/virtio.h> +#include <uapi/linux/virtio_ring.h> #include <linux/virtio_balloon.h> #include <linux/swap.h> #include <linux/workqueue.h>
@@ -1175,6 +1176,13 @@ static int virtballoon_validate(struct virtio_device *vdev)else if (!virtio_has_feature (vdev,VIRTIO_BALLOON_F_PAGE_POISON ))__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_REPORTING ); +/* +* Balloon submits 1-2 sg entries max per buffer, virtqueue +* sizes are 128+. Disable indirect descriptors to avoid +* GFP_KERNEL allocation in virtqueue_add under balloon_lock, +* which could deadlock via OOM -> oom_notify -> leak_balloon. +*/ +__virtio_clear_bit (vdev,VIRTIO_RING_F_INDIRECT_DESC );__virtio_clear_bit (vdev,VIRTIO_F_ACCESS_PLATFORM );return 0 ; }
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 29/37 mm: page_reporting: add flush parameter with page budget
In reply to Michael S. TsirkinAdd a write-only module parameter 'flush' that triggers immediate page reporting. The value specifies a page budget: at least this many pages (at page_reporting_order) will be reported, or all unreported pages if fewer remain. The actual number reported may exceed the budget since each reporting pass processes a full cycle across all zones.
This is helpful when there is a lot of memory freed quickly, and a single cycle may not process all free pages due to internal budget limits.
echo 512 > /sys/module/page_reporting/parameters/flush
Note: the set callback runs under kernel param_lock, so writing this parameter blocks other built-in parameter writes until the flush loop completes. This is acceptable for a privileged debug/test parameter.
--- mm/page_reporting.c | 54 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+)
mm/page_reporting.c +54 -0
@@ -358,6 +358,60 @@ static void page_reporting_process(struct work_struct *work)static DEFINE_MUTEX (page_reporting_mutex);DEFINE_STATIC_KEY_FALSE (page_reporting_enabled); +static int page_reporting_flush_set (const char *val, +const struct kernel_param *kp) +{ +struct page_reporting_dev_info *prdev; +unsigned int budget; +int err; + + err =kstrtouint (val,0 , &budget); +if (err) +return err; +if (!budget) +return 0 ; + +mutex_lock (&page_reporting_mutex); + prdev =rcu_dereference_protected (pr_dev_info, +lockdep_is_held (&page_reporting_mutex)); +if (prdev) { +unsigned int reported; +bool interrupted = false; + +for (reported =0 ; reported < budget; + reported +=min (prdev->capacity, budget - reported)) { +/* +* First flush completes any previously scheduled +* reporting work. Then request a new reporting +* cycle and flush again to execute it. +*/ +flush_delayed_work (&prdev->work); +__page_reporting_request (prdev); +flush_delayed_work (&prdev->work); +if (atomic_read (&prdev->state) ==PAGE_REPORTING_IDLE ) +break ; +if (signal_pending (current)) { + interrupted = true; +break ; + } + } +if (interrupted) { +mutex_unlock (&page_reporting_mutex); +return -EINTR ; + } + } +mutex_unlock (&page_reporting_mutex); +return 0 ; +} + +static const struct kernel_param_ops flush_ops = { + .set = page_reporting_flush_set, + .get = param_get_uint, +}; +static unsigned int page_reporting_flush; +module_param_cb (flush, &flush_ops, &page_reporting_flush,0200 ); +MODULE_PARM_DESC (flush,"Report at least N pages at page_reporting_order, or until all reported" ); +int page_reporting_register (struct page_reporting_dev_info *prdev) {int err =0 ;
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 30/37 mm: page_alloc: propagate PG_zeroed in split_large_buddy
In reply to Michael S. TsirkinWhen splitting a large buddy page, propagate the PG_zeroed flag to each sub-page before freeing it. __free_pages_prepare clears all flags (including PG_zeroed), so the flag must be re-set on each fragment after the split. This ensures that the buddy merge logic can see PG_zeroed on pages that were part of a larger zeroed block.
--- mm/page_alloc.c | 3 +++ 1 file changed, 3 insertions(+)
mm/page_alloc.c +3 -0
@@ -1524,6 +1524,7 @@ static void split_large_buddy(struct zone *zone, struct page *page, {unsigned long end = pfn + (1 << order);bool reported =PageReported (page); +bool zeroed =PageZeroed (page);VM_WARN_ON_ONCE (!IS_ALIGNED (pfn,1 << order));/* Caller removed page from freelist, buddy info cleared! */
@@ -1537,6 +1538,8 @@ static void split_large_buddy(struct zone *zone, struct page *page,if (reported)__SetPageReported (page); +if (zeroed) +__SetPageZeroed (page);__free_one_page (page, pfn, zone, order, mt, fpi); pfn +=1 << order;if (pfn == end)
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 31/37 virtio_balloon: skip zeroing for host-zeroed reported pages
In reply to Michael S. TsirkinImplement VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED (per virtio spec proposal): when negotiated, the device initializes reported pages (zeros, or poison_val if PAGE_POISON).
Check per-page used length returned by the device to determine which reported pages were zeroed. If used_len matches the page size, the device successfully initialized the page (e.g. via MADV_DONTNEED), and we set the corresponding zeroed_bitmap bit.
Gate host_zeroes_pages on the feature bit and page content:
when PAGE_POISON is negotiated with non-zero poison_val, the device fills with poison not zeros, so pages are not zeroed.
Clear the feature in validate() if REPORTING is not present or if PAGE_POISON is active with non-zero poison_val.
See the virtio spec change:
https://github.com/oasis-tcs/virtio-spec/issues/244
--- drivers/virtio/virtio_balloon.c | 22 ++++++++++++++++++++-- include/uapi/linux/virtio_balloon.h | 1 + 2 files changed, 21 insertions(+), 2 deletions(-)
drivers/virtio/virtio_balloon.c +20 -2
@@ -207,6 +207,8 @@ static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_istruct virtqueue *vq = vb->reporting_vq;unsigned int i, err =0 ; +bitmap_zero (pr_dev_info->zeroed_bitmap, nents); +/* We should always be able to add these buffers to an empty queue. */ for (i =0 ; i < nents; i++) {struct scatterlist one;
@@ -226,10 +228,14 @@ static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_i/* When host has read buffer, this completes via balloon_ack */ for (i =0 ; i < nents; i++) { -unsigned int unused; +struct scatterlist *entry; +unsigned int used_len;wait_event (vb->acked, -virtqueue_get_buf (vq, &unused)); + (entry =virtqueue_get_buf (vq, &used_len))); +if (used_len == entry->length) +set_bit (entry - sg, + pr_dev_info->zeroed_bitmap); } }
@@ -1051,6 +1057,9 @@ static int virtballoon_probe(struct virtio_device *vdev)#endif vb->pr_dev_info.capacity = capacity; + vb->pr_dev_info.host_zeroes_pages = +virtio_has_feature (vdev, +VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED ); err =page_reporting_register (&vb->pr_dev_info);if (err) goto out_unregister_oom;
@@ -1176,6 +1185,14 @@ static int virtballoon_validate(struct virtio_device *vdev)else if (!virtio_has_feature (vdev,VIRTIO_BALLOON_F_PAGE_POISON ))__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_REPORTING ); +if (!virtio_has_feature (vdev,VIRTIO_BALLOON_F_REPORTING )) +__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED ); + +/* Device fills with poison_val, not zeros; disable zeroed hint */ +if (virtio_has_feature (vdev,VIRTIO_BALLOON_F_PAGE_POISON ) && + !want_init_on_free ()) +__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED ); +/* * Balloon submits 1-2 sg entries max per buffer, virtqueue * sizes are 128+. Disable indirect descriptors to avoid
@@ -1194,6 +1211,7 @@ static unsigned int features[] = {VIRTIO_BALLOON_F_FREE_PAGE_HINT ,VIRTIO_BALLOON_F_PAGE_POISON ,VIRTIO_BALLOON_F_REPORTING , + VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED, }; static struct virtio_driver virtio_balloon_driver = {
include/uapi/linux/virtio_balloon.h +1 -0
@@ -37,6 +37,7 @@#define VIRTIO_BALLOON_F_FREE_PAGE_HINT 3/* VQ to report free pages */ #define VIRTIO_BALLOON_F_PAGE_POISON 4/* Guest is using page poisoning */ #define VIRTIO_BALLOON_F_REPORTING 5/* Page reporting virtqueue */ +#define VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED 6/* Device initializes reported pages */ /* Size of a PFN in the balloon interface. */ #define VIRTIO_BALLOON_PFN_SHIFT 12
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 32/37 virtio_balloon: disable reporting zeroed optimization for confidential guests
In reply to Michael S. TsirkinIn confidential computing environments (TDX, SEV-SNP), the host is untrusted and may lie about zeroing reported pages. Clear DEVICE_INIT_REPORTED in validate() so the guest does not skip re-zeroing based on hints from an untrusted device.
Note: currently REPORTING remains enabled and VIRTIO_F_ACCESS_PLATFORM is cleared in CC environments. This is known to leak free page physical addresses to the host. Whether that, or ballooning in general, is a security concern in CC is up to the user. This patch only disables our new zeroed-page hints where the host is untrusted.
--- drivers/virtio/virtio_balloon.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/virtio/virtio_balloon.c +3 -0
@@ -19,6 +19,7 @@#include <linux/wait.h> #include <linux/mm.h> #include <linux/page_reporting.h> +#include <linux/cc_platform.h> /* * Balloon device works in 4K page units. So each page is pointed to by
@@ -1193,6 +1194,8 @@ static int virtballoon_validate(struct virtio_device *vdev) !want_init_on_free())__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED ); +if (cc_platform_has (CC_ATTR_GUEST_MEM_ENCRYPT )) +__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED );/* * Balloon submits 1-2 sg entries max per buffer, virtqueue * sizes are 128+. Disable indirect descriptors to avoid
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 33/37 mm: add free_frozen_pages_zeroed
In reply to Michael S. TsirkinAdd free_frozen_pages_zeroed(page, order) to free a frozen page while marking it as zeroed, so the next allocation can skip redundant zeroing.
An FPI_ZEROED internal flag carries the hint through the free path. PageZeroed is set after __free_pages_prepare() clears all flags, so the hint survives on the free list.
__SetPageZeroed is non-atomic but safe here: the page is frozen (refcount 0) and not yet on any free list.
--- include/linux/gfp.h | 1 + mm/internal.h | 1 - mm/page_alloc.c | 29 ++++++++++++++++++++++++++++- 3 files changed, 29 insertions(+), 2 deletions(-)
include/linux/gfp.h +1 -0
@@ -384,6 +384,7 @@ __meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_masextern void __free_pages (struct page *page,unsigned int order);extern void free_pages_nolock (struct page *page,unsigned int order);extern void free_pages (unsigned long addr,unsigned int order); +void free_frozen_pages_zeroed (struct page *page,unsigned int order);#define __free_page (page) __free_pages((page), 0)#define free_page (addr) free_pages((addr), 0)
mm/internal.h +0 -1
@@ -938,7 +938,6 @@ struct page *__alloc_frozen_pages_noprof(gfp_t, unsigned int order, int nid,#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))void free_frozen_pages (struct page *page,unsigned int order); -void free_frozen_pages_zeroed (struct page *page,unsigned int order);void free_unref_folios (struct folio_batch *fbatch);#ifdef CONFIG_NUMA
mm/page_alloc.c +28 -1
@@ -90,6 +90,13 @@ typedef int __bitwise fpi_t;/* Free the page without taking locks. Rely on trylock only. */ #define FPI_TRYLOCK ((__force fpi_t)BIT(2)) +/* +* The page contents are known to be zero (e.g., the host zeroed them +* during balloon deflate). Set PageZeroed after free so the next +* allocation can skip redundant zeroing. +*/ +#define FPI_ZEROED ((__force fpi_t)BIT(3)) +/* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX (pcp_batch_high_lock);#define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8)
@@ -1599,8 +1606,19 @@ static void __free_pages_ok(struct page *page, unsigned int order,unsigned long pfn =page_to_pfn (page);struct zone *zone =page_zone (page); -if (__free_pages_prepare (page, order, fpi_flags)) +if (__free_pages_prepare (page, order, fpi_flags)) { +/* +* Only mark as zeroed if the page is actually still zero. +* kernel_poison_pages() overwrites with PAGE_POISON (0xAA) +* when CONFIG_PAGE_POISONING is enabled. With init_on_free, +* kernel_init_pages() re-zeroes after poisoning, so the +* page is still zero. +*/ +if ((fpi_flags &FPI_ZEROED ) && + (!page_poisoning_enabled_static () ||want_init_on_free ())) +__SetPageZeroed (page);free_one_page (zone, page, pfn, order, fpi_flags); + } } void __meminit __free_pages_core(struct page *page, unsigned int order,
@@ -3021,6 +3039,9 @@ static void __free_frozen_pages(struct page *page, unsigned int order,if (!__free_pages_prepare (page, order, fpi_flags))return ; +if (fpi_flags &FPI_ZEROED ) +__SetPageZeroed (page); +/* * We only track unmovable, reclaimable and movable on pcp lists. * Place ISOLATE pages on the isolated list because they are being
@@ -3059,6 +3080,12 @@ void free_frozen_pages(struct page *page, unsigned int order)__free_frozen_pages (page, order,FPI_NONE ); } +void free_frozen_pages_zeroed (struct page *page,unsigned int order) +{ +__free_frozen_pages (page, order,FPI_ZEROED ); +} +EXPORT_SYMBOL (free_frozen_pages_zeroed); +void free_frozen_pages_nolock (struct page *page,unsigned int order) {__free_frozen_pages (page, order,FPI_TRYLOCK );
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 34/37 mm: page_alloc: skip kernel_init_pages for FPI_ZEROED when safe
In reply to Michael S. TsirkinIn __free_pages_prepare(), when FPI_ZEROED is set the page is already known to be zero. We can skip kernel_init_pages() if page poisoning is not enabled (because poison would overwrite the zeroes).
This avoids redundant zeroing work when freeing pages that are already known to contain all zeros.
--- mm/page_alloc.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
mm/page_alloc.c +8 -1
@@ -1447,7 +1447,14 @@ __always_inline bool __free_pages_prepare(struct page *page,if (kasan_has_integrated_init ()) init = false; } -if (init) +/* +* Skip redundant zeroing when the page is already known-zero +* (FPI_ZEROED) and page poisoning did not overwrite it. +* When page_poisoning is enabled, kernel_poison_pages above +* wrote PAGE_POISON (0xAA), so we must re-zero. +*/ +if (init && !((fpi_flags &FPI_ZEROED ) && + !page_poisoning_enabled_static ()))kernel_init_pages (page,1 << order);/*
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 35/37 mm: add put_page_zeroed and folio_put_zeroed
In reply to Michael S. TsirkinAdd put_page_zeroed() / folio_put_zeroed() for callers that hold a reference to a page known to be zeroed.
If this drops the last reference, the zeroed hint is propagated to the buddy allocator. If someone else still holds a reference, the hint is simply lost - this is best-effort.
This is useful for balloon drivers during deflation: the host has already zeroed the pages, and the balloon is typically the sole owner. But if the page happens to be shared, silently dropping the hint is safe and avoids the need for callers to check the refcount.
--- include/linux/mm.h | 13 +++++++++++++ mm/swap.c | 20 ++++++++++++++++++-- 2 files changed, 31 insertions(+), 2 deletions(-)
include/linux/mm.h +13 -0
@@ -1913,6 +1913,7 @@ static inline struct folio *virt_to_folio(const void *x) }void __folio_put (struct folio *folio); +void __folio_put_zeroed (struct folio *folio);void split_page (struct page *page,unsigned int order);void folio_copy (struct folio *dst,struct folio *src);
@@ -2090,6 +2091,18 @@ static inline void folio_put(struct folio *folio)__folio_put (folio); } +/* Caller must be sole owner to guarantee page is still zero */ +static inline void folio_put_zeroed (struct folio *folio) +{ +if (folio_put_testzero (folio)) +__folio_put_zeroed (folio); +} + +static inline void put_page_zeroed (struct page *page) +{ +folio_put_zeroed (page_folio (page)); +} +/** * folio_put_refs - Reduce the reference count on a folio. * @folio: The folio.
mm/swap.c +18 -2
@@ -94,13 +94,15 @@ static void page_cache_release(struct folio *folio)lruvec_unlock_irqrestore (lruvec, flags); } -void __folio_put (struct folio *folio) +static void ___folio_put (struct folio *folio,bool zeroed) { +/* zeroed hint ignored for now, no current user */ if (unlikely (folio_is_zone_device (folio))) {free_zone_device_folio (folio);return ; } +/* zeroed hint ignored for now, no current user */ if (folio_test_hugetlb (folio)) {free_huge_folio (folio);return ;
@@ -109,10 +111,24 @@ void __folio_put(struct folio *folio)page_cache_release (folio);folio_unqueue_deferred_split (folio);mem_cgroup_uncharge (folio); -free_frozen_pages (&folio->page,folio_order (folio)); +if (zeroed) +free_frozen_pages_zeroed (&folio->page,folio_order (folio)); +else +free_frozen_pages (&folio->page,folio_order (folio)); +} + +void __folio_put (struct folio *folio) +{ +___folio_put (folio, false); }EXPORT_SYMBOL (__folio_put); +void __folio_put_zeroed (struct folio *folio) +{ +___folio_put (folio, true); +} +EXPORT_SYMBOL (__folio_put_zeroed); +typedef void (*move_fn_t )(struct lruvec *lruvec,struct folio *folio);static void lru_add (struct lruvec *lruvec,struct folio *folio)
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 36/37 mm: balloon: use put_page_zeroed for zeroed balloon pages
In reply to Michael S. TsirkinWhen a balloon page marked PageZeroed is freed during migration, use put_page_zeroed() to propagate the zeroed hint to the buddy allocator. Previously the hint was silently lost via plain put_page().
No page has PageZeroed set yet; the next patch (VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE) will set it on pages the host has zeroed during inflate.
--- mm/balloon.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
mm/balloon.c +9 -1
@@ -324,7 +324,15 @@ static int balloon_page_migrate(struct page *newpage, struct page *page,balloon_page_finalize (page);spin_unlock_irqrestore (&balloon_pages_lock, flags); -put_page (page); +if (PageZeroed (page)) { +/* Atomic to serialize with memory_failure's +* TestSetPageHWPoison; not under zone->lock here. +*/ +ClearPageZeroed (page); +put_page_zeroed (page); + }else { +put_page (page); + }return 0 ; }
Signature · 1 line
MST
Michael S. Tsirkin
maintainerPATCH v9 37/37 virtio_balloon: implement VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE
In reply to Michael S. TsirkinWhen the device offers DEVICE_INIT_ON_INFLATE (bit 7), the device initializes inflated pages and returns a per-page bitmap indicating which pages were successfully initialized.
The driver appends a device-writable bitmap buffer to each inflate descriptor chain via virtqueue_add_sgs. After the host acknowledges, the driver checks bitmap bits (bounded by used_len) and marks pages with SetPageZeroed.
tell_host() returns used_len from virtqueue_get_buf(). Bitmap reads are bounded: fill_balloon() and virtballoon_migratepage() only trust bits within the used_len range.
On deflate, release_pages_balloon checks PageZeroed per page and uses put_page_zeroed for pages the host initialized, propagating the zeroed hint to the buddy allocator.
If inflate_vq has fewer than 2 descriptors, probe fails with -ENOSPC. If PAGE_POISON is negotiated with non-zero poison_val, the feature is cleared in validate().
See the virtio spec change:
https://lore.kernel.org/all/9c69b992c3dd83dfef3db92cd86b2fd8a0730d48.1777731396.git.mst@redhat.com
__SetPageZeroed on inflated pages is non-atomic but safe:
the balloon owns the page exclusively at this point.
--- drivers/virtio/virtio_balloon.c | 107 ++++++++++++++++++++++++---- include/uapi/linux/virtio_balloon.h | 1 + 2 files changed, 96 insertions(+), 12 deletions(-)
drivers/virtio/virtio_balloon.c +95 -12
@@ -20,6 +20,7 @@#include <linux/mm.h> #include <linux/page_reporting.h> #include <linux/cc_platform.h> +#include <asm-generic/bitops/le.h> /* * Balloon device works in 4K page units. So each page is pointed to by
@@ -122,6 +123,13 @@ struct virtio_balloon {struct virtqueue *reporting_vq;struct page_reporting_dev_info pr_dev_info; +/* No DMA alignment needed: ACCESS_PLATFORM is cleared, +* so virtio bypasses the DMA API. If this ever changes, +* add ____dma_from_device_aligned here. +*/ +/* Bitmap returned by host for DEVICE_INIT_ON_INFLATE */ +DECLARE_BITMAP (inflate_bitmap,VIRTIO_BALLOON_ARRAY_PFNS_MAX ); +/* State for keeping the wakeup_source active while adjusting the balloon */ spinlock_t wakeup_lock;bool processing_wakeup_event;
@@ -182,22 +190,33 @@ static void balloon_ack(struct virtqueue *vq)wake_up (&vb->acked); } -static void tell_host (struct virtio_balloon *vb,struct virtqueue *vq) +static unsigned int tell_host (struct virtio_balloon *vb,struct virtqueue *vq) { -struct scatterlist sg; +struct scatterlist sg_out, sg_in; +struct scatterlist *sgs[] = { &sg_out, &sg_in };unsigned int len; -sg_init_one (&sg, vb->pfns,sizeof (vb->pfns[0 ]) * vb->num_pfns); +sg_init_one (&sg_out, vb->pfns,sizeof (vb->pfns[0 ]) * vb->num_pfns);/* We made sure the vq is large enough so we should always * be able to add one buffer to an empty queue. */ -virtqueue_add_outbuf (vq, &sg,1 , vb,GFP_KERNEL ); +if (vq == vb->inflate_vq && +virtio_has_feature (vb->vdev, +VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE )) { +unsigned int bitmap_bytes; + + bitmap_bytes =DIV_ROUND_UP (vb->num_pfns,8 ); +bitmap_zero (vb->inflate_bitmap, vb->num_pfns); +sg_init_one (&sg_in, vb->inflate_bitmap, bitmap_bytes); +virtqueue_add_sgs (vq, sgs,1 ,1 , vb,GFP_KERNEL ); + }else { +virtqueue_add_outbuf (vq, &sg_out,1 , vb,GFP_KERNEL ); + }virtqueue_kick (vq); -/* When host has read buffer, this completes via balloon_ack */ wait_event (vb->acked,virtqueue_get_buf (vq, &len)); - +return len; } static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_info,
@@ -300,8 +319,37 @@ static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num) num_allocated_pages = vb->num_pfns;/* Did we get any? */ -if (vb->num_pfns !=0 ) -tell_host (vb, vb->inflate_vq); +if (vb->num_pfns !=0 ) { +unsigned int used_len =tell_host (vb, vb->inflate_vq); + +if (virtio_has_feature (vb->vdev, +VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE )) { +unsigned int i; +unsigned int valid_bits = used_len *8 ; + +for (i =0 ; i < vb->num_pfns; + i +=VIRTIO_BALLOON_PAGES_PER_PAGE ) { +unsigned int pfn, j; +bool zeroed = true; + +if (i +VIRTIO_BALLOON_PAGES_PER_PAGE > valid_bits) +break ; +for (j =0 ; j <VIRTIO_BALLOON_PAGES_PER_PAGE ; j++) { +if (!test_bit_le (i + j, vb->inflate_bitmap)) { + zeroed = false; +break ; + } + } +if (zeroed) { + pfn =virtio32_to_cpu (vb->vdev, + vb->pfns[i]); +__SetPageZeroed (pfn_to_page (pfn >> + (PAGE_SHIFT - +VIRTIO_BALLOON_PFN_SHIFT ))); + } + } + } + }mutex_unlock (&vb->balloon_lock);return num_allocated_pages;
@@ -314,7 +362,12 @@ static void release_pages_balloon(struct virtio_balloon *vb,list_for_each_entry_safe (page, next, pages, lru) {list_del (&page->lru); -put_page (page);/* balloon reference */ +if (PageZeroed (page)) { +__ClearPageZeroed (page); +put_page_zeroed (page); + }else { +put_page (page); + } } }
@@ -861,8 +914,27 @@ static int virtballoon_migratepage(struct balloon_dev_info *vb_dev_info,/* balloon's page migration 1st step -- inflate "newpage" */ vb->num_pfns =VIRTIO_BALLOON_PAGES_PER_PAGE ;set_page_pfns (vb, vb->pfns, newpage); -tell_host (vb, vb->inflate_vq); + { +unsigned int used_len =tell_host (vb, vb->inflate_vq); +if (virtio_has_feature (vb->vdev, +VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE ) && + used_len >=DIV_ROUND_UP (VIRTIO_BALLOON_PAGES_PER_PAGE ,8 )) { +unsigned int j; +bool zeroed = true; + +for (j =0 ; j <VIRTIO_BALLOON_PAGES_PER_PAGE ; j++) { +if (!test_bit_le (j, vb->inflate_bitmap)) { + zeroed = false; +break ; + } + } +if (zeroed) +__SetPageZeroed (newpage); + } + } + +__ClearPageZeroed (page);/* balloon's page migration 2nd step -- deflate "page" */ vb->num_pfns =VIRTIO_BALLOON_PAGES_PER_PAGE ;set_page_pfns (vb, vb->pfns, page);
@@ -966,6 +1038,12 @@ static int virtballoon_probe(struct virtio_device *vdev)if (err) goto out_free_vb; +if (virtio_has_feature (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE ) && +virtqueue_get_vring_size (vb->inflate_vq) <2 ) { + err = -ENOSPC ; + goto out_del_vqs; + } +if (!virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_DEFLATE_ON_OOM )) vb->vb_dev_info.adjust_managed_page_count = true;#ifdef CONFIG_BALLOON_MIGRATION
@@ -1191,11 +1269,15 @@ static int virtballoon_validate(struct virtio_device *vdev)/* Device fills with poison_val, not zeros; disable zeroed hint */ if (virtio_has_feature (vdev,VIRTIO_BALLOON_F_PAGE_POISON ) && - !want_init_on_free ()) + !want_init_on_free ()) {__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED ); +__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE ); + } -if (cc_platform_has (CC_ATTR_GUEST_MEM_ENCRYPT )) +if (cc_platform_has (CC_ATTR_GUEST_MEM_ENCRYPT )) {__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED ); +__virtio_clear_bit (vdev,VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE ); + }/* * Balloon submits 1-2 sg entries max per buffer, virtqueue * sizes are 128+. Disable indirect descriptors to avoid
@@ -1215,6 +1297,7 @@ static unsigned int features[] = {VIRTIO_BALLOON_F_PAGE_POISON ,VIRTIO_BALLOON_F_REPORTING ,VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED , + VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE, }; static struct virtio_driver virtio_balloon_driver = {
include/uapi/linux/virtio_balloon.h +1 -0
@@ -38,6 +38,7 @@#define VIRTIO_BALLOON_F_PAGE_POISON 4/* Guest is using page poisoning */ #define VIRTIO_BALLOON_F_REPORTING 5/* Page reporting virtqueue */ #define VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED 6/* Device initializes reported pages */ +#define VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE 7/* Device initializes pages on inflate */ /* Size of a PFN in the balloon interface. */ #define VIRTIO_BALLOON_PFN_SHIFT 12
Signature · 1 line
MST
Michael S. Tsirkin
maintainerI went over sashiko review of this:
https://sashiko.dev/#/patchset/cover.1780067977.git.mst%40redhat.com
It's mostly Sashiko apparently not seeing the whole picture (e.g. is says "an optimization is missing" and it is actually
added in follow up patches).
Sometimes it is unhappy while there's an explanation right in the commit log. Weird.
It's a pity there's no way to respond to the review, to make it pay more attention and reconsider.
It did flag a minor rebase artifact (declaration landed in the wrong patch during rebase). Mostly harmless but if there's a v10, I'll fix that.
But it also flagged 4 pre-existing bugs:
- page_reporting_register before DRIVER_OK, theoretical race in virtballoon_probe ordering:
https://sashiko.dev/#/patchset/cover.1780067977.git.mst%40redhat.com?part=4
- spurious OOM on large-folio swapin race in do_swap_page:
https://sashiko.dev/#/patchset/cover.1780067977.git.mst%40redhat.com?part=16
- free_huge_folio called with refcount==1 on mem_cgroup_charge_hugetlb failure:
https://sashiko.dev/#/patchset/cover.1780067977.git.mst%40redhat.com?part=20
- double-decrement of resv_huge_pages in memfd_alloc_folio error path:
https://sashiko.dev/#/patchset/cover.1780067977.git.mst%40redhat.com?part=21
All seem unrelated to the specific feature, so I'd rather put off fixing these until the feature is merged.
Signature · 1 line
MST