mm/virtio: skip redundant zeroing of host-zeroed reported pages
Subsystems (13)
Michael S. Tsirkin
maintainercover mm/virtio: skip redundant zeroing of host-zeroed reported pages
When a guest reports free pages to the hypervisor via virtio-balloon's free page reporting, the host typically zeros those pages when reclaiming their backing memory (e.g., via MADV_DONTNEED on anonymous mappings). When the guest later reallocates those pages, the kernel zeros them again -- redundantly.
This series eliminates that double-zeroing by propagating the "host already zeroed this page" information through the buddy allocator and into the page fault path.
Performance with THP enabled on a 2GB VM, 1 vCPU, allocating 256MB of anonymous pages:
metric baseline optimized delta task-clock 179ms 99ms -45% cache-misses 1.22M 287K -76% instructions 15.1M 13.9M -8%
With hugetlb surplus pages:
metric baseline optimized delta task-clock 322ms 9.9ms -97% cache-misses 659K 88K -87% instructions 18.3M 10.6M -42%
Notes:
- The virtio_balloon patch (9/9) is a testing hack with a module parameter. A proper virtio feature flag is needed before merging.
- Patch 8/9 adds a sysfs flush trigger for deterministic testing (avoids waiting for the 2-second reporting delay).
- The optimization is most effective with THP, where entire 2MB pages are allocated directly from reported order-9+ buddy pages. Without THP, only ~21% of order-0 allocations come from reported pages due to low-order fragmentation.
- Persistent hugetlb pool pages are not covered: when freed by userspace they return to the hugetlb free pool, not the buddy allocator, so they are never reported to the host. Surplus hugetlb pages are allocated from buddy and do benefit.
Test program:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#ifndef MADV_POPULATE_WRITE
#define MADV_POPULATE_WRITE 23
#endif
#ifndef MAP_HUGETLB
#define MAP_HUGETLB 0x40000
#endif
int main(int argc, char **argv)
{
unsigned long size;
int flags = MAP_PRIVATE | MAP_ANONYMOUS;
void *p;
int r; if (argc < 2) {
fprintf(stderr, "usage: %s <size_mb> [huge]\n", argv[0]);
return 1;
}
size = atol(argv[1]) * 1024UL * 1024;
if (argc >= 3 && strcmp(argv[2], "huge") == 0)
flags |= MAP_HUGETLB;
p = mmap(NULL, size, PROT_READ | PROT_WRITE, flags, -1, 0);
if (p == MAP_FAILED) {
perror("mmap");
return 1;
}
r = madvise(p, size, MADV_POPULATE_WRITE);
if (r) {
perror("madvise");
return 1;
}
munmap(p, size);
return 0;}
Test script (bench.sh):
#!/bin/bash
# Usage: bench.sh <size_mb> <mode> <iterations> [huge]
# mode 0 = baseline, mode 1 = skip zeroing
SZ=${1:-256}; MODE=${2:-0}; ITER=${3:-10}; HUGE=${4:-}
FLUSH=/sys/module/page_reporting/parameters/flush
PERF_DATA=/tmp/perf-$MODE.data
rmmod virtio_balloon 2>/dev/null
insmod virtio_balloon.ko host_zeroes_pages=$MODE
echo 1 > $FLUSH
[ "$HUGE" = "huge" ] && echo $((SZ/2)) > /proc/sys/vm/nr_overcommit_hugepages
rm -f $PERF_DATA
echo "=== sz=${SZ}MB mode=$MODE iter=$ITER $HUGE ==="
for i in $(seq 1 $ITER); do
echo 3 > /proc/sys/vm/drop_caches
echo 1 > $FLUSH
perf stat record -e task-clock,instructions,cache-misses \
-o $PERF_DATA --append -- ./alloc_once $SZ $HUGE
done
[ "$HUGE" = "huge" ] && echo 0 > /proc/sys/vm/nr_overcommit_hugepages
rmmod virtio_balloon
perf stat report -i $PERF_DATACompile and run:
gcc -static -O2 -o alloc_once alloc_once.c
bash bench.sh 256 0 10 # baseline (regular pages) bash bench.sh 256 1 10 # optimized (regular pages) bash bench.sh 256 0 10 huge # baseline (hugetlb surplus) bash bench.sh 256 1 10 huge # optimized (hugetlb surplus)
Written with assistance from claude. Everything manually read, patchset split and commit logs edited manually.
Michael S. Tsirkin (9):
mm: page_alloc: propagate PageReported flag across buddy splits
mm: page_reporting: skip redundant zeroing of host-zeroed reported
pages
mm: add __GFP_PREZEROED flag and folio_test_clear_prezeroed()
mm: skip zeroing in vma_alloc_zeroed_movable_folio for pre-zeroed
pages
mm: skip zeroing in alloc_anon_folio for pre-zeroed pages
mm: skip zeroing in vma_alloc_anon_folio_pmd for pre-zeroed pages
mm: hugetlb: skip zeroing of pre-zeroed hugetlb pages
mm: page_reporting: add flush parameter to trigger immediate reporting
virtio_balloon: a hack to enable host-zeroed page optimization
drivers/virtio/virtio_balloon.c | 7 +++++ fs/hugetlbfs/inode.c | 3 ++- include/linux/gfp_types.h | 5 ++++ include/linux/highmem.h | 6 +++-- include/linux/hugetlb.h | 2 +- include/linux/mm.h | 22 ++++++++++++++++ include/linux/page_reporting.h | 3 +++ mm/huge_memory.c | 4 +-- mm/hugetlb.c | 3 ++- mm/memory.c | 5 ++-- mm/page_alloc.c | 46 ++++++++++++++++++++++++++++++--- mm/page_reporting.c | 34 ++++++++++++++++++++++++ mm/page_reporting.h | 2 ++ 13 files changed, 129 insertions(+), 13 deletions(-)
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRFC PATCH 1/9 mm: page_alloc: propagate PageReported flag across buddy splits
When a reported free page is split via expand() to satisfy a smaller allocation, the sub-pages placed back on the free lists lose the PageReported flag. This means they will be unnecessarily re-reported to the hypervisor in the next reporting cycle, wasting work.
Propagate the PageReported flag to sub-pages during expand() so that they are recognized as already-reported.
--- mm/page_alloc.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
mm/page_alloc.c +14 -3
@@ -1730,7 +1730,7 @@ struct page *__pageblock_pfn_to_page(unsigned long start_pfn,* -- nyc */ static inline unsigned int expand (struct zone *zone,struct page *page,int low, -int high,int migratetype) +int high,int migratetype,bool reported) {unsigned int size =1 << high;unsigned int nr_added =0 ;
@@ -1752,6 +1752,15 @@ static inline unsigned int expand(struct zone *zone, struct page *page, int low,__add_to_free_list (&page[size], zone, high, migratetype, false);set_buddy_order (&page[size], high); nr_added += size; + +/* +* The parent page has been reported to the host. The +* sub-pages are part of the same reported block, so mark +* them reported too. This avoids re-reporting pages that +* the host already knows about. +*/ +if (reported) +__SetPageReported (&page[size]); }return nr_added;
@@ -1762,9 +1771,10 @@ static __always_inline void page_del_and_expand(struct zone *zone, int high, int migratetype) { int nr_pages = 1 << high; +bool was_reported =page_reported (page);__del_page_from_free_list (page, zone, high, migratetype); - nr_pages -=expand (zone, page, low, high, migratetype); + nr_pages -=expand (zone, page, low, high, migratetype, was_reported);account_freepages (zone, -nr_pages, migratetype); }
@@ -2322,7 +2332,8 @@ try_to_claim_block(struct zone *zone, struct page *page,del_page_from_free_list (page, zone, current_order, block_type);change_pageblock_range (page, current_order, start_type); - nr_added =expand (zone, page, order, current_order, start_type); + nr_added =expand (zone, page, order, current_order, start_type, + false);account_freepages (zone, nr_added, start_type);return page; }
Signature · 1 line
MST
David Hildenbrand (Arm)
maintainerRe: [PATCH RFC 1/9] mm: page_alloc: propagate PageReported flag across buddy splits
Quoted 29 lines · Michael S. Tsirkin
When a reported free page is split via
expand()to satisfy a smaller allocation, the sub-pages placed back on the free lists lose the PageReported flag. This means they will be unnecessarily re-reported to the hypervisor in the next reporting cycle, wasting work.Propagate the PageReported flag to sub-pages during
expand()so that they are recognized as already-reported.Signed-off-by Michael S. Tsirkin <[email protected]>Assisted-by Claude:claude-opus-4-6--- mm/page_alloc.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)mm/page_alloc.c +1
-1@@ -1730,7 +1730,7 @@ struct page *__pageblock_pfn_to_page(unsigned long start_pfn,* -- nyc */ static inline unsigned int expand (struct zone *zone,struct page *page,int low, -int high,int migratetype) +int high,int migratetype,bool reported) {unsigned int size =1 << high;unsigned int nr_added =0 ;
mm/page_alloc.c static inline unsigned int expand(struct zone *zone, struct page *page, int low, · line 1761
Quoted 10 lines
mm/page_alloc.c +7
-0@@ -1752,6 +1752,15 @@ static inline unsigned int expand(struct zone *zone, struct page *page, int low,__add_to_free_list (&page[size], zone, high, migratetype, false);set_buddy_order (&page[size], high); nr_added += size; + +/* +* The parent page has been reported to the host. The +* sub-pages are part of the same reported block, so mark +* them reported too. This avoids re-reporting pages that +* the host already knows about. +*/
The comment is a bit excessive. I'd say you can drop it completely.
Quoted 18 lines
mm/page_alloc.c +4
-1+if (reported) +__SetPageReported (&page[size]); }return nr_added;@@ -1762,9 +1771,10 @@ static __always_inline void page_del_and_expand(struct zone *zone, int high, int migratetype) { int nr_pages = 1 << high; +bool was_reported =page_reported (page);__del_page_from_free_list (page, zone, high, migratetype); - nr_pages -=expand (zone, page, low, high, migratetype); + nr_pages -=expand (zone, page, low, high, migratetype, was_reported);account_freepages (zone, -nr_pages, migratetype); }
mm/page_alloc.c try_to_claim_block(struct zone *zone, struct page *page, · line 2336
Quoted 6 lines
mm/page_alloc.c +2
-1@@ -2322,7 +2332,8 @@ try_to_claim_block(struct zone *zone, struct page *page,del_page_from_free_list (page, zone, current_order, block_type);change_pageblock_range (page, current_order, start_type); - nr_added =expand (zone, page, order, current_order, start_type); + nr_added =expand (zone, page, order, current_order, start_type, + false);
In MM land we started doing
/* reported= */false
This raises a good question: how does buddy merging handle the reported flag?
Signature · 3 lines
Cheers, David
Michael S. Tsirkin
maintainerRe: [PATCH RFC 1/9] mm: page_alloc: propagate PageReported flag across buddy splits
Quoted 70 lines · David Hildenbrand (Arm)
Quoted 29 lines · Michael S. Tsirkin
When a reported free page is split via
expand()to satisfy a smaller allocation, the sub-pages placed back on the free lists lose the PageReported flag. This means they will be unnecessarily re-reported to the hypervisor in the next reporting cycle, wasting work.Propagate the PageReported flag to sub-pages during
expand()so that they are recognized as already-reported.Signed-off-by Michael S. Tsirkin <[email protected]>Assisted-by Claude:claude-opus-4-6--- mm/page_alloc.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)mm/page_alloc.c +1
-1@@ -1730,7 +1730,7 @@ struct page *__pageblock_pfn_to_page(unsigned long start_pfn,* -- nyc */ static inline unsigned int expand (struct zone *zone,struct page *page,int low, -int high,int migratetype) +int high,int migratetype,bool reported) {unsigned int size =1 << high;unsigned int nr_added =0 ;mm/page_alloc.c static inline unsigned int expand(struct zone *zone, struct page *page, int low, · line 1761
Quoted 10 lines
mm/page_alloc.c +7
-0@@ -1752,6 +1752,15 @@ static inline unsigned int expand(struct zone *zone, struct page *page, int low,__add_to_free_list (&page[size], zone, high, migratetype, false);set_buddy_order (&page[size], high); nr_added += size; + +/* +* The parent page has been reported to the host. The +* sub-pages are part of the same reported block, so mark +* them reported too. This avoids re-reporting pages that +* the host already knows about. +*/ The comment is a bit excessive. I'd say you can drop it completely.
Quoted 18 lines
mm/page_alloc.c +4
-1+if (reported) +__SetPageReported (&page[size]); }return nr_added;@@ -1762,9 +1771,10 @@ static __always_inline void page_del_and_expand(struct zone *zone, int high, int migratetype) { int nr_pages = 1 << high; +bool was_reported =page_reported (page);__del_page_from_free_list (page, zone, high, migratetype); - nr_pages -=expand (zone, page, low, high, migratetype); + nr_pages -=expand (zone, page, low, high, migratetype, was_reported);account_freepages (zone, -nr_pages, migratetype); }mm/page_alloc.c try_to_claim_block(struct zone *zone, struct page *page, · line 2336
Quoted 6 lines
mm/page_alloc.c +2
-1@@ -2322,7 +2332,8 @@ try_to_claim_block(struct zone *zone, struct page *page,del_page_from_free_list (page, zone, current_order, block_type);change_pageblock_range (page, current_order, start_type); - nr_added =expand (zone, page, order, current_order, start_type); + nr_added =expand (zone, page, order, current_order, start_type, + false);In MM land we started doing
Quoted 10 lines
/* reported= */false
This raises a good question: how does buddy merging handle the reported flag?
Signature · 3 lines
Cheers, David
IIUC it doesn't: reported pages are never merged, if a page is merged it has just entered buddy.
Signature · 1 line
MST
David Hildenbrand (Arm)
maintainerRe: [PATCH RFC 1/9] mm: page_alloc: propagate PageReported flag across buddy splits
Quoted 5 lines
Quoted 5 lines
This raises a good question: how does buddy merging handle the reported flag?
Quoted 8 lines
Quoted 4 lines
Signature · 3 lines
Cheers, DavidIIUC it doesn't: reported pages are never merged, if a page is merged it has just entered buddy.
I guess it makes sense: if a page gets freed and we end up merging, the freed page was not reported yet, so we have to report the whole block.
Signature · 3 lines
Cheers, David
Michael S. Tsirkin
maintainerRFC PATCH 2/9 mm: page_reporting: skip redundant zeroing of host-zeroed reported pages
In reply to Michael S. TsirkinWhen a guest reports free pages to the hypervisor via the page reporting framework (used by virtio-balloon and hv_balloon), the host typically zeros those pages when reclaiming their backing memory. However, when those pages are later allocated in the guest, post_alloc_hook() unconditionally zeros them again if __GFP_ZERO is set. This double-zeroing is wasteful, especially for large pages.
Avoid redundant zeroing by propagating the "host already zeroed this" information through the allocation path:
1. Add a host_zeroes_pages flag to page_reporting_dev_info, allowing drivers to declare that their host zeros reported pages on reclaim. A static key (page_reporting_host_zeroes) gates the fast path.
2. In page_del_and_expand(), when the page was reported and the static key is enabled, stash a sentinel value (MAGIC_PAGE_ZEROED) in page->private.
3. In post_alloc_hook(), check page->private for the sentinel. If present and zeroing was requested (but not tag zeroing), skip kernel_init_pages().
In particular, __GFP_ZERO is used by the x86 arch override of vma_alloc_zeroed_movable_folio.
No driver sets host_zeroes_pages yet; a follow-up patch to virtio_balloon is needed to opt in.
--- include/linux/mm.h | 6 ++++++ include/linux/page_reporting.h | 3 +++ mm/page_alloc.c | 21 +++++++++++++++++++++ mm/page_reporting.c | 9 +++++++++ mm/page_reporting.h | 2 ++ 5 files changed, 41 insertions(+)
include/linux/mm.h +6 -0
@@ -4814,6 +4814,12 @@ static inline bool user_alloc_needs_zeroing(void) &init_on_alloc); } +/* +* Sentinel stored in page->private to indicate the page was pre-zeroed +* by the hypervisor (via free page reporting). +*/ +#define MAGIC_PAGE_ZEROED 0x5A45524FU/* ZERO */ +int arch_get_shadow_stack_status (struct task_struct *t,unsigned long __user *status);int arch_set_shadow_stack_status (struct task_struct *t,unsigned long status);int arch_lock_shadow_stack_status (struct task_struct *t,unsigned long status);
include/linux/page_reporting.h +3 -0
@@ -13,6 +13,9 @@ struct page_reporting_dev_info {int (*report)(struct page_reporting_dev_info *prdev,struct scatterlist *sg,unsigned int nents); +/* If true, host zeros reported pages on reclaim */ +bool host_zeroes_pages; +/* work struct for processing reports */ struct delayed_work work;
mm/page_alloc.c +21 -0
@@ -1774,8 +1774,20 @@ static __always_inline void page_del_and_expand(struct zone *zone,bool was_reported =page_reported (page);__del_page_from_free_list (page, zone, high, migratetype); + + was_reported = was_reported && +static_branch_unlikely (&page_reporting_host_zeroes); + nr_pages -=expand (zone, page, low, high, migratetype, was_reported);account_freepages (zone, -nr_pages, migratetype); + +/* +* If the page was reported and the host is known to zero reported +* pages, mark it zeroed via page->private so that +* post_alloc_hook() can skip redundant zeroing. +*/ +if (was_reported) +set_page_private (page,MAGIC_PAGE_ZEROED ); }static void check_new_page_bad (struct page *page)
@@ -1851,11 +1863,20 @@ inline void post_alloc_hook(struct page *page, unsigned int order, {bool init = !want_init_on_free () &&want_init_on_alloc (gfp_flags) && !should_skip_init (gfp_flags); +bool prezeroed =page_private (page) ==MAGIC_PAGE_ZEROED ;bool zero_tags = init && (gfp_flags & __GFP_ZEROTAGS);int i;set_page_private (page,0 ); +/* +* If the page is pre-zeroed, skip memory initialization. +* We still need to handle tag zeroing separately since the host +* does not know about memory tags. +*/ +if (prezeroed && init && !zero_tags) + init = false; +arch_alloc_page (page, order);debug_pagealloc_map_pages (page,1 << order);
mm/page_reporting.c +9 -0
@@ -50,6 +50,8 @@ EXPORT_SYMBOL_GPL(page_reporting_order);#define PAGE_REPORTING_DELAY (2 * HZ)static struct page_reporting_dev_info __rcu *pr_dev_info__read_mostly ; +DEFINE_STATIC_KEY_FALSE (page_reporting_host_zeroes); +enum {PAGE_REPORTING_IDLE =0 ,PAGE_REPORTING_REQUESTED ,
@@ -386,6 +388,10 @@ int page_reporting_register(struct page_reporting_dev_info *prdev)/* Assign device to allow notifications */ rcu_assign_pointer (pr_dev_info, prdev); +/* enable zeroed page optimization if host zeroes reported pages */ +if (prdev->host_zeroes_pages) +static_branch_enable (&page_reporting_host_zeroes); +/* enable page reporting notification */ if (!static_key_enabled (&page_reporting_enabled)) {static_branch_enable (&page_reporting_enabled);
@@ -410,6 +416,9 @@ void page_reporting_unregister(struct page_reporting_dev_info *prdev)/* Flush any existing work, and lock it out */ cancel_delayed_work_sync (&prdev->work); + +if (prdev->host_zeroes_pages) +static_branch_disable (&page_reporting_host_zeroes); }mutex_unlock (&page_reporting_mutex);
mm/page_reporting.h +2 -0
@@ -15,6 +15,8 @@ DECLARE_STATIC_KEY_FALSE(page_reporting_enabled);extern unsigned int page_reporting_order;void __page_reporting_notify (void ); +DECLARE_STATIC_KEY_FALSE (page_reporting_host_zeroes); +static inline bool page_reported (struct page *page) { return static_branch_unlikely(&page_reporting_enabled) &&
Signature · 1 line
MST
David Hildenbrand (Arm)
maintainerRe: [PATCH RFC 2/9] mm: page_reporting: skip redundant zeroing of host-zeroed reported pages
Quoted 43 lines · Michael S. Tsirkin
When a guest reports free pages to the hypervisor via the page reporting framework (used by virtio-balloon and
hv_balloon), the host typically zeros those pages when reclaiming their backing memory. However, when those pages are later allocated in the guest,post_alloc_hook()unconditionally zeros them again if__GFP_ZEROis set. This double-zeroing is wasteful, especially for large pages.Avoid redundant zeroing by propagating the "host already zeroed this" information through the allocation path:
1. Add a
host_zeroes_pagesflag topage_reporting_dev_info, allowing drivers to declare that their host zeros reported pages on reclaim. A static key (page_reporting_host_zeroes) gates the fast path.2. In
page_del_and_expand(), when the page was reported and the static key is enabled, stash a sentinel value (MAGIC_PAGE_ZEROED) inpage->private.3. In
post_alloc_hook(), checkpage->privatefor the sentinel. If present and zeroing was requested (but not tag zeroing), skipkernel_init_pages().In particular,
__GFP_ZEROis used by the x86 arch override ofvma_alloc_zeroed_movable_folio.No driver sets
host_zeroes_pagesyet; a follow-up patch tovirtio_balloonis needed to opt in.Signed-off-by Michael S. Tsirkin <[email protected]>Assisted-by Claude:claude-opus-4-6--- include/linux/mm.h | 6 ++++++ include/linux/page_reporting.h | 3 +++ mm/page_alloc.c | 21 +++++++++++++++++++++ mm/page_reporting.c | 9 +++++++++ mm/page_reporting.h | 2 ++ 5 files changed, 41 insertions(+)include/linux/mm.h +0
-0
include/linux/mm.h static inline bool user_alloc_needs_zeroing(void) · line 4821
Quoted 8 lines
include/linux/mm.h +5
-0@@ -4814,6 +4814,12 @@ static inline bool user_alloc_needs_zeroing(void) &init_on_alloc); } +/* +* Sentinel stored in page->private to indicate the page was pre-zeroed +* by the hypervisor (via free page reporting). +*/ +#define MAGIC_PAGE_ZEROED 0x5A45524FU/* ZERO */
Why are we not using another page flag that is yet unused for buddy pages?
Using page->private for that, and exposing it to buddy users with the __GFP_PREZEROED flag (I hope we can avoid that) does not sound particularly elegant.
Also, if we're going to remember that some pages in the buddy are pre-zeroed, it should better not be free-page-reporting specific.
I'd assume ordinary inflating+deflating of the balloon would also end up with pre-zeroed pages. We'd just need a (mm/balloon.c -specific) interface to tell the buddy that the pages are zeroed.
Signature · 3 lines
Cheers, David
Michael S. Tsirkin
maintainerRe: [PATCH RFC 2/9] mm: page_reporting: skip redundant zeroing of host-zeroed reported pages
Quoted 47 lines · David Hildenbrand (Arm)
Quoted 46 lines · Michael S. Tsirkin
When a guest reports free pages to the hypervisor via the page reporting framework (used by virtio-balloon and
hv_balloon), the host typically zeros those pages when reclaiming their backing memory. However, when those pages are later allocated in the guest,post_alloc_hook()unconditionally zeros them again if__GFP_ZEROis set. This double-zeroing is wasteful, especially for large pages.Avoid redundant zeroing by propagating the "host already zeroed this" information through the allocation path:
1. Add a
host_zeroes_pagesflag topage_reporting_dev_info, allowing drivers to declare that their host zeros reported pages on reclaim. A static key (page_reporting_host_zeroes) gates the fast path.2. In
page_del_and_expand(), when the page was reported and the static key is enabled, stash a sentinel value (MAGIC_PAGE_ZEROED) inpage->private.3. In
post_alloc_hook(), checkpage->privatefor the sentinel. If present and zeroing was requested (but not tag zeroing), skipkernel_init_pages().In particular,
__GFP_ZEROis used by the x86 arch override ofvma_alloc_zeroed_movable_folio.No driver sets
host_zeroes_pagesyet; a follow-up patch tovirtio_balloonis needed to opt in.Signed-off-by Michael S. Tsirkin <[email protected]>Assisted-by Claude:claude-opus-4-6--- include/linux/mm.h | 6 ++++++ include/linux/page_reporting.h | 3 +++ mm/page_alloc.c | 21 +++++++++++++++++++++ mm/page_reporting.c | 9 +++++++++ mm/page_reporting.h | 2 ++ 5 files changed, 41 insertions(+)include/linux/mm.h +0
-0@@ -4814,6 +4814,12 @@ static inline bool user_alloc_needs_zeroing(void) &init_on_alloc); }
Quoted 7 lines
include/linux/mm.h static inline bool user_alloc_needs_zeroing(void) · line 4821
Quoted 5 lines
include/linux/mm.h +5
-0+/* +* Sentinel stored in page->private to indicate the page was pre-zeroed +* by the hypervisor (via free page reporting). +*/ +#define MAGIC_PAGE_ZEROED 0x5A45524FU/* ZERO */ Why are we not using another page flag that is yet unused for buddy pages?
Because we need to report the status *after* it left buddy. And all flags are in use at that point.
Quoted 3 lines
Using
page->privatefor that, and exposing it to buddy users with the__GFP_PREZEROEDflag (I hope we can avoid that) does not sound particularly elegant.
But propagating this all over mm does not sound too palatable, right? There's precedent with MAGIC_HWPOISON already.
Better ideas? Thanks!
Quoted 6 lines
Also, if we're going to remember that some pages in the buddy are pre-zeroed, it should better not be free-page-reporting specific. I'd assume ordinary inflating+deflating of the balloon would also end up with pre-zeroed pages. We'd just need a (
mm/balloon.c-specific) interface to tell the buddy that the pages are zeroed.
Indeed, it's also easily possible - it's a separate optimization, though. Another simple enhancement is including hugetlbfs freelists in page reporting.
Doesn't need to block this patchset though, right?
Quoted 5 lines
Signature · 4 lines
-- Cheers, David
David Hildenbrand (Arm)
maintainerRe: [PATCH RFC 2/9] mm: page_reporting: skip redundant zeroing of host-zeroed reported pages
Quoted 56 lines · Michael S. Tsirkin
Quoted 47 lines · David Hildenbrand (Arm)
Quoted 46 lines · Michael S. Tsirkin
When a guest reports free pages to the hypervisor via the page reporting framework (used by virtio-balloon and
hv_balloon), the host typically zeros those pages when reclaiming their backing memory. However, when those pages are later allocated in the guest,post_alloc_hook()unconditionally zeros them again if__GFP_ZEROis set. This double-zeroing is wasteful, especially for large pages.Avoid redundant zeroing by propagating the "host already zeroed this" information through the allocation path:
1. Add a
host_zeroes_pagesflag topage_reporting_dev_info, allowing drivers to declare that their host zeros reported pages on reclaim. A static key (page_reporting_host_zeroes) gates the fast path.2. In
page_del_and_expand(), when the page was reported and the static key is enabled, stash a sentinel value (MAGIC_PAGE_ZEROED) inpage->private.3. In
post_alloc_hook(), checkpage->privatefor the sentinel. If present and zeroing was requested (but not tag zeroing), skipkernel_init_pages().In particular,
__GFP_ZEROis used by the x86 arch override ofvma_alloc_zeroed_movable_folio.No driver sets
host_zeroes_pagesyet; a follow-up patch tovirtio_balloonis needed to opt in.Signed-off-by Michael S. Tsirkin <[email protected]>Assisted-by Claude:claude-opus-4-6--- include/linux/mm.h | 6 ++++++ include/linux/page_reporting.h | 3 +++ mm/page_alloc.c | 21 +++++++++++++++++++++ mm/page_reporting.c | 9 +++++++++ mm/page_reporting.h | 2 ++ 5 files changed, 41 insertions(+)include/linux/mm.h +0
-0@@ -4814,6 +4814,12 @@ static inline bool user_alloc_needs_zeroing(void) &init_on_alloc); }Quoted 7 lines
include/linux/mm.h static inline bool user_alloc_needs_zeroing(void) · line 4821
Quoted 5 lines
include/linux/mm.h +5
-0+/* +* Sentinel stored in page->private to indicate the page was pre-zeroed +* by the hypervisor (via free page reporting). +*/ +#define MAGIC_PAGE_ZEROED 0x5A45524FU/* ZERO */ Why are we not using another page flag that is yet unused for buddy pages?
Quoted 2 lines
Because we need to report the status *after* it left buddy. And all flags are in use at that point.
I'll comment on that on the other patch, where __GFP_PREZEROED, which I really hate, is added.
Quoted 9 lines
Quoted 5 lines
Using
page->privatefor that, and exposing it to buddy users with the__GFP_PREZEROEDflag (I hope we can avoid that) does not sound particularly elegant.But propagating this all over mm does not sound too palatable, right? There's precedent with
MAGIC_HWPOISONalready.
Better ideas? Thanks!
I'll comment on the __GFP_PREZEROED patch.
Quoted 8 lines
Quoted 7 lines
Also, if we're going to remember that some pages in the buddy are pre-zeroed, it should better not be free-page-reporting specific. I'd assume ordinary inflating+deflating of the balloon would also end up with pre-zeroed pages. We'd just need a (
mm/balloon.c-specific) interface to tell the buddy that the pages are zeroed.
Quoted 4 lines
Indeed, it's also easily possible - it's a separate optimization, though. Another simple enhancement is including hugetlbfs freelists in page reporting.
Doesn't need to block this patchset though, right?
Not blocking, but I don't want something that is too coupled to free-page reporting optimizations in the buddy. The comment above MAGIC_PAGE_ZEROED triggered my reaction.
Signature · 3 lines
Cheers, David
Michael S. Tsirkin
maintainerRe: [PATCH RFC 2/9] mm: page_reporting: skip redundant zeroing of host-zeroed reported pages
Quoted 83 lines · David Hildenbrand (Arm)
Quoted 56 lines · Michael S. Tsirkin
Quoted 47 lines · David Hildenbrand (Arm)
Quoted 46 lines · Michael S. Tsirkin
When a guest reports free pages to the hypervisor via the page reporting framework (used by virtio-balloon and
hv_balloon), the host typically zeros those pages when reclaiming their backing memory. However, when those pages are later allocated in the guest,post_alloc_hook()unconditionally zeros them again if__GFP_ZEROis set. This double-zeroing is wasteful, especially for large pages.Avoid redundant zeroing by propagating the "host already zeroed this" information through the allocation path:
1. Add a
host_zeroes_pagesflag topage_reporting_dev_info, allowing drivers to declare that their host zeros reported pages on reclaim. A static key (page_reporting_host_zeroes) gates the fast path.2. In
page_del_and_expand(), when the page was reported and the static key is enabled, stash a sentinel value (MAGIC_PAGE_ZEROED) inpage->private.3. In
post_alloc_hook(), checkpage->privatefor the sentinel. If present and zeroing was requested (but not tag zeroing), skipkernel_init_pages().In particular,
__GFP_ZEROis used by the x86 arch override ofvma_alloc_zeroed_movable_folio.No driver sets
host_zeroes_pagesyet; a follow-up patch tovirtio_balloonis needed to opt in.Signed-off-by Michael S. Tsirkin <[email protected]>Assisted-by Claude:claude-opus-4-6--- include/linux/mm.h | 6 ++++++ include/linux/page_reporting.h | 3 +++ mm/page_alloc.c | 21 +++++++++++++++++++++ mm/page_reporting.c | 9 +++++++++ mm/page_reporting.h | 2 ++ 5 files changed, 41 insertions(+)include/linux/mm.h +0
-0@@ -4814,6 +4814,12 @@ static inline bool user_alloc_needs_zeroing(void) &init_on_alloc); }Quoted 7 lines
include/linux/mm.h static inline bool user_alloc_needs_zeroing(void) · line 4821
Quoted 5 lines
include/linux/mm.h +5
-0+/* +* Sentinel stored in page->private to indicate the page was pre-zeroed +* by the hypervisor (via free page reporting). +*/ +#define MAGIC_PAGE_ZEROED 0x5A45524FU/* ZERO */ Why are we not using another page flag that is yet unused for buddy pages?
Quoted 2 lines
Because we need to report the status *after* it left buddy. And all flags are in use at that point.
I'll comment on that on the other patch, where
__GFP_PREZEROED, which I really hate, is added.Quoted 9 lines
Quoted 5 lines
Using
page->privatefor that, and exposing it to buddy users with the__GFP_PREZEROEDflag (I hope we can avoid that) does not sound particularly elegant.But propagating this all over mm does not sound too palatable, right? There's precedent with
MAGIC_HWPOISONalready.
Better ideas? Thanks!I'll comment on the
__GFP_PREZEROEDpatch.Quoted 8 lines
Quoted 7 lines
Also, if we're going to remember that some pages in the buddy are pre-zeroed, it should better not be free-page-reporting specific. I'd assume ordinary inflating+deflating of the balloon would also end up with pre-zeroed pages. We'd just need a (
mm/balloon.c-specific) interface to tell the buddy that the pages are zeroed.
Quoted 7 lines
Quoted 4 lines
Indeed, it's also easily possible - it's a separate optimization, though. Another simple enhancement is including hugetlbfs freelists in page reporting.
Doesn't need to block this patchset though, right?Not blocking, but I don't want something that is too coupled to free-page reporting optimizations in the buddy.
I can add that in the next version if you like, sure. The main issue is that it means we need a flag that survives free. And the benefit is much smaller - unlike page reports, deflates are rare.
Quoted 2 lines
The comment above
MAGIC_PAGE_ZEROEDtriggered my reaction.
yea, that's more confusing than helpful.
Quoted 4 lines
Signature · 3 lines
Cheers, David
Michael S. Tsirkin
maintainerRe: [PATCH RFC 2/9] mm: page_reporting: skip redundant zeroing of host-zeroed reported pages
In reply to David Hildenbrand (Arm)Quoted 55 lines · David Hildenbrand (Arm)
Quoted 43 lines · Michael S. Tsirkin
When a guest reports free pages to the hypervisor via the page reporting framework (used by virtio-balloon and
hv_balloon), the host typically zeros those pages when reclaiming their backing memory. However, when those pages are later allocated in the guest,post_alloc_hook()unconditionally zeros them again if__GFP_ZEROis set. This double-zeroing is wasteful, especially for large pages.Avoid redundant zeroing by propagating the "host already zeroed this" information through the allocation path:
1. Add a
host_zeroes_pagesflag topage_reporting_dev_info, allowing drivers to declare that their host zeros reported pages on reclaim. A static key (page_reporting_host_zeroes) gates the fast path.2. In
page_del_and_expand(), when the page was reported and the static key is enabled, stash a sentinel value (MAGIC_PAGE_ZEROED) inpage->private.3. In
post_alloc_hook(), checkpage->privatefor the sentinel. If present and zeroing was requested (but not tag zeroing), skipkernel_init_pages().In particular,
__GFP_ZEROis used by the x86 arch override ofvma_alloc_zeroed_movable_folio.No driver sets
host_zeroes_pagesyet; a follow-up patch tovirtio_balloonis needed to opt in.Signed-off-by Michael S. Tsirkin <[email protected]>Assisted-by Claude:claude-opus-4-6--- include/linux/mm.h | 6 ++++++ include/linux/page_reporting.h | 3 +++ mm/page_alloc.c | 21 +++++++++++++++++++++ mm/page_reporting.c | 9 +++++++++ mm/page_reporting.h | 2 ++ 5 files changed, 41 insertions(+)include/linux/mm.h +0
-0include/linux/mm.h static inline bool user_alloc_needs_zeroing(void) · line 4821
Quoted 8 lines
include/linux/mm.h +5
-0@@ -4814,6 +4814,12 @@ static inline bool user_alloc_needs_zeroing(void) &init_on_alloc); } +/* +* Sentinel stored in page->private to indicate the page was pre-zeroed +* by the hypervisor (via free page reporting). +*/ +#define MAGIC_PAGE_ZEROED 0x5A45524FU/* ZERO */ Why are we not using another page flag that is yet unused for buddy pages?
Quoted 3 lines
Using
page->privatefor that, and exposing it to buddy users with the__GFP_PREZEROEDflag (I hope we can avoid that) does not sound particularly elegant.
So here's an only alternative I see: a page flag for when page is in buddy and a new "prezero" bool that we have to propagate everywhere else. This is a patch on top. More elegant? Please tell me if you prefer that. If yes I will squash it into the appropriate patches.
include/linux/gfp_types.h +1 -1
@@ -294,7 +294,7 @@ enum {#define __GFP_SKIP_ZERO ((__force gfp_t)___GFP_SKIP_ZERO)#define __GFP_SKIP_KASAN ((__force gfp_t)___GFP_SKIP_KASAN) -/* Caller handles pre-zeroed pages; preserve MAGIC_PAGE_ZEROED in private */ +/* Caller handles pre-zeroed pages; preserve PagePrezeroed */ #define __GFP_PREZEROED ((__force gfp_t)___GFP_PREZEROED)/* Disable lockdep for GFP context tracking */
include/linux/mm.h +15 -9
@@ -4814,11 +4814,21 @@ static inline bool user_alloc_needs_zeroing(void) &init_on_alloc); } -/* -* Sentinel stored in page->private to indicate the page was pre-zeroed -* by the hypervisor (via free page reporting). +/** +* __page_test_clear_prezeroed - test and clear the pre-zeroed marker. +* @page: the page to test. +* +* Returns true if the page was pre-zeroed by the host, and clears +* the marker. Caller must have exclusive access to @page. */ -#define MAGIC_PAGE_ZEROED 0x5A45524FU/* ZERO */ +static inline bool __page_test_clear_prezeroed (struct page *page) +{ +if (PagePrezeroed (page)) { +__ClearPagePrezeroed (page); +return true; + } +return false; +}/** * folio_test_clear_prezeroed - test and clear the pre-zeroed marker.
@@ -4829,11 +4839,7 @@ static inline bool user_alloc_needs_zeroing(void)*/ static inline bool folio_test_clear_prezeroed (struct folio *folio) { -if (page_private (&folio->page) ==MAGIC_PAGE_ZEROED ) { -set_page_private (&folio->page,0 ); -return true; - } -return false; +return __page_test_clear_prezeroed (&folio->page); }int arch_get_shadow_stack_status (struct task_struct *t,unsigned long __user *status);
include/linux/page-flags.h +9 -0
@@ -135,6 +135,8 @@ enum pageflags { PG_swapcache = PG_owner_priv_1,/* Swap page: swp_entry_t in private */ /* Some filesystems */ PG_checked = PG_owner_priv_1, +/* Page contents are known to be zero */ + PG_prezeroed = PG_owner_priv_1,/* * Depending on the way an anonymous folio can be mapped into a page
@@ -679,6 +681,13 @@ FOLIO_TEST_CLEAR_FLAG_FALSE(young) FOLIO_FLAG_FALSE(idle)#endif +/* +* PagePrezeroed() tracks pages known to be zero. The +* allocator may preserve this bit for __GFP_PREZEROED callers so they can +* skip redundant zeroing after allocation. +*/ +__PAGEFLAG (Prezeroed, prezeroed,PF_NO_COMPOUND ) +/* * PageReported() is used to track reported free pages within the Buddy * allocator. We can use the non-atomic version of the test and set
mm/compaction.c +2 -2
@@ -82,7 +82,7 @@ static inline bool is_via_compact_memory(int order) { return false; }static struct page *mark_allocated_noprof (struct page *page,unsigned int order,gfp_t gfp_flags) { -post_alloc_hook (page, order, __GFP_MOVABLE); +post_alloc_hook (page, order, __GFP_MOVABLE, false);set_page_refcounted (page);return page; }
@@ -1833,7 +1833,7 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long da } dst = (struct folio *)freepage; -post_alloc_hook (&dst->page, order, __GFP_MOVABLE); +post_alloc_hook (&dst->page, order, __GFP_MOVABLE, false);set_page_refcounted (&dst->page);if (order)prep_compound_page (&dst->page, order);
mm/internal.h +2 -1
@@ -887,7 +887,8 @@ static inline void prep_compound_tail(struct page *head, int tail_idx)set_page_private (p,0 ); } -void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags); +void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags, +bool prezeroed);extern bool free_pages_prepare (struct page *page,unsigned int order);extern int user_min_free_kbytes;
mm/page_alloc.c +41 -27
212 lines not on this page · raw message
David Hildenbrand (Arm)
maintainerRe: [PATCH RFC 2/9] mm: page_reporting: skip redundant zeroing of host-zeroed reported pages
Quoted 61 lines · Michael S. Tsirkin
Quoted 55 lines · David Hildenbrand (Arm)
Quoted 43 lines · Michael S. Tsirkin
When a guest reports free pages to the hypervisor via the page reporting framework (used by virtio-balloon and
hv_balloon), the host typically zeros those pages when reclaiming their backing memory. However, when those pages are later allocated in the guest,post_alloc_hook()unconditionally zeros them again if__GFP_ZEROis set. This double-zeroing is wasteful, especially for large pages.Avoid redundant zeroing by propagating the "host already zeroed this" information through the allocation path:
1. Add a
host_zeroes_pagesflag topage_reporting_dev_info, allowing drivers to declare that their host zeros reported pages on reclaim. A static key (page_reporting_host_zeroes) gates the fast path.2. In
page_del_and_expand(), when the page was reported and the static key is enabled, stash a sentinel value (MAGIC_PAGE_ZEROED) inpage->private.3. In
post_alloc_hook(), checkpage->privatefor the sentinel. If present and zeroing was requested (but not tag zeroing), skipkernel_init_pages().In particular,
__GFP_ZEROis used by the x86 arch override ofvma_alloc_zeroed_movable_folio.No driver sets
host_zeroes_pagesyet; a follow-up patch tovirtio_balloonis needed to opt in.Signed-off-by Michael S. Tsirkin <[email protected]>Assisted-by Claude:claude-opus-4-6--- include/linux/mm.h | 6 ++++++ include/linux/page_reporting.h | 3 +++ mm/page_alloc.c | 21 +++++++++++++++++++++ mm/page_reporting.c | 9 +++++++++ mm/page_reporting.h | 2 ++ 5 files changed, 41 insertions(+)include/linux/mm.h +0
-0include/linux/mm.h static inline bool user_alloc_needs_zeroing(void) · line 4821
Quoted 8 lines
include/linux/mm.h +5
-0@@ -4814,6 +4814,12 @@ static inline bool user_alloc_needs_zeroing(void) &init_on_alloc); } +/* +* Sentinel stored in page->private to indicate the page was pre-zeroed +* by the hypervisor (via free page reporting). +*/ +#define MAGIC_PAGE_ZEROED 0x5A45524FU/* ZERO */ Why are we not using another page flag that is yet unused for buddy pages?
Quoted 3 lines
Using
page->privatefor that, and exposing it to buddy users with the__GFP_PREZEROEDflag (I hope we can avoid that) does not sound particularly elegant.
Quoted 4 lines
So here's an only alternative I see: a page flag for when page is in buddy and a new "prezero" bool that we have to propagate everywhere else. This is a patch on top. More elegant? Please tell me if you prefer that. If yes I will squash it into the appropriate patches.
I'd be interesting to know how this would look without the GFP flag, when we don't have to leak any of this out of the buddy.
Signature · 3 lines
Cheers, David
Michael S. Tsirkin
maintainerRe: [PATCH RFC 2/9] mm: page_reporting: skip redundant zeroing of host-zeroed reported pages
Quoted 62 lines · David Hildenbrand (Arm)
Quoted 61 lines · Michael S. Tsirkin
Quoted 58 lines · David Hildenbrand (Arm)
Quoted 43 lines · Michael S. Tsirkin
When a guest reports free pages to the hypervisor via the page reporting framework (used by virtio-balloon and
hv_balloon), the host typically zeros those pages when reclaiming their backing memory. However, when those pages are later allocated in the guest,post_alloc_hook()unconditionally zeros them again if__GFP_ZEROis set. This double-zeroing is wasteful, especially for large pages.Avoid redundant zeroing by propagating the "host already zeroed this" information through the allocation path:
1. Add a
host_zeroes_pagesflag topage_reporting_dev_info, allowing drivers to declare that their host zeros reported pages on reclaim. A static key (page_reporting_host_zeroes) gates the fast path.2. In
page_del_and_expand(), when the page was reported and the static key is enabled, stash a sentinel value (MAGIC_PAGE_ZEROED) inpage->private.3. In
post_alloc_hook(), checkpage->privatefor the sentinel. If present and zeroing was requested (but not tag zeroing), skipkernel_init_pages().In particular,
__GFP_ZEROis used by the x86 arch override ofvma_alloc_zeroed_movable_folio.No driver sets
host_zeroes_pagesyet; a follow-up patch tovirtio_balloonis needed to opt in.Signed-off-by Michael S. Tsirkin <[email protected]>Assisted-by Claude:claude-opus-4-6--- include/linux/mm.h | 6 ++++++ include/linux/page_reporting.h | 3 +++ mm/page_alloc.c | 21 +++++++++++++++++++++ mm/page_reporting.c | 9 +++++++++ mm/page_reporting.h | 2 ++ 5 files changed, 41 insertions(+)include/linux/mm.h +0
-0include/linux/mm.h static inline bool user_alloc_needs_zeroing(void) · line 4821
Quoted 8 lines
include/linux/mm.h +5
-0@@ -4814,6 +4814,12 @@ static inline bool user_alloc_needs_zeroing(void) &init_on_alloc); } +/* +* Sentinel stored in page->private to indicate the page was pre-zeroed +* by the hypervisor (via free page reporting). +*/ +#define MAGIC_PAGE_ZEROED 0x5A45524FU/* ZERO */ Why are we not using another page flag that is yet unused for buddy pages?
Using
page->privatefor that, and exposing it to buddy users with the__GFP_PREZEROEDflag (I hope we can avoid that) does not sound particularly elegant.
Quoted 12 lines
Quoted 4 lines
So here's an only alternative I see: a page flag for when page is in buddy and a new "prezero" bool that we have to propagate everywhere else. This is a patch on top. More elegant? Please tell me if you prefer that. If yes I will squash it into the appropriate patches.
I'd be interesting to know how this would look without the GFP flag, when we don't have to leak any of this out of the buddy.
Signature · 3 lines
Cheers, David
But the zeroing takes place outside of the buddy now, it's more "reporting" than "leaking".
You mean, moving the zeroing into buddy?
See
https://lore.kernel.org/all/[email protected]/
Signature · 1 line
MST
David Hildenbrand (Arm)
maintainerRe: [PATCH RFC 2/9] mm: page_reporting: skip redundant zeroing of host-zeroed reported pages
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · David Hildenbrand (Arm)
Quoted 6 lines · Michael S. Tsirkin
So here's an only alternative I see: a page flag for when page is in buddy and a new "prezero" bool that we have to propagate everywhere else. This is a patch on top. More elegant? Please tell me if you prefer that. If yes I will squash it into the appropriate patches.
I'd be interesting to know how this would look without the GFP flag, when we don't have to leak any of this out of the buddy.
Quoted 7 lines
Quoted 4 lines
Signature · 3 lines
Cheers, DavidBut the zeroing takes place outside of the buddy now, it's more "reporting" than "leaking".
Exposing some buddy internals using questionable GFP flags, however you want to call that hack ;)
Quoted 2 lines
You mean, moving the zeroing into buddy?
Yes. Such that it doesn't really matter how we maintain this information internally. A page flag is probably still nicer for that purpose.
Signature · 3 lines
Cheers, David
Michael S. Tsirkin
maintainerRe: [PATCH RFC 2/9] mm: page_reporting: skip redundant zeroing of host-zeroed reported pages
Quoted 21 lines · David Hildenbrand (Arm)
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · David Hildenbrand (Arm)
Quoted 6 lines · Michael S. Tsirkin
So here's an only alternative I see: a page flag for when page is in buddy and a new "prezero" bool that we have to propagate everywhere else. This is a patch on top. More elegant? Please tell me if you prefer that. If yes I will squash it into the appropriate patches.
I'd be interesting to know how this would look without the GFP flag, when we don't have to leak any of this out of the buddy.
Quoted 7 lines
Quoted 4 lines
Signature · 3 lines
Cheers, DavidBut the zeroing takes place outside of the buddy now, it's more "reporting" than "leaking".
Quoted 8 lines
Exposing some buddy internals using questionable GFP flags, however you want to call that hack ;)
Quoted 2 lines
You mean, moving the zeroing into buddy?
Yes. Such that it doesn't really matter how we maintain this information internally. A page flag is probably still nicer for that purpose.
OK, sure. Let me just finish testing, and I will post.
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRFC PATCH 3/9 mm: add __GFP_PREZEROED flag and folio_test_clear_prezeroed()
In reply to Michael S. TsirkinThe previous patch skips zeroing in post_alloc_hook() when __GFP_ZERO is used. However, several page allocation paths zero pages via folio_zero_user() or clear_user_highpage() after allocation, not via __GFP_ZERO.
Add __GFP_PREZEROED gfp flag that tells post_alloc_hook() to preserve the MAGIC_PAGE_ZEROED sentinel in page->private so the caller can detect pre-zeroed pages and skip its own zeroing. Add folio_test_clear_prezeroed() helper to check and clear the sentinel.
--- include/linux/gfp_types.h | 5 +++++ include/linux/mm.h | 16 ++++++++++++++++ mm/page_alloc.c | 8 +++++++- 3 files changed, 28 insertions(+), 1 deletion(-)
include/linux/gfp_types.h +5 -0
@@ -56,6 +56,7 @@ enum { ___GFP_NOLOCKDEP_BIT, #endif ___GFP_NO_OBJ_EXT_BIT, + ___GFP_PREZEROED_BIT, ___GFP_LAST_BIT };
@@ -97,6 +98,7 @@ enum {#define ___GFP_NOLOCKDEP 0#endif #define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT) +#define ___GFP_PREZEROED BIT(___GFP_PREZEROED_BIT)/* * Physical address zone modifiers (see linux/mmzone.h - low four bits)
@@ -292,6 +294,9 @@ enum {#define __GFP_SKIP_ZERO ((__force gfp_t)___GFP_SKIP_ZERO)#define __GFP_SKIP_KASAN ((__force gfp_t)___GFP_SKIP_KASAN) +/* Caller handles pre-zeroed pages; preserve MAGIC_PAGE_ZEROED in private */ +#define __GFP_PREZEROED ((__force gfp_t)___GFP_PREZEROED) +/* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)
include/linux/mm.h +16 -0
@@ -4820,6 +4820,22 @@ static inline bool user_alloc_needs_zeroing(void)*/ #define MAGIC_PAGE_ZEROED 0x5A45524FU/* ZERO */ +/** +* folio_test_clear_prezeroed - test and clear the pre-zeroed marker. +* @folio: the folio to test. +* +* Returns true if the folio was pre-zeroed by the host, and clears +* the marker. Callers can skip their own zeroing. +*/ +static inline bool folio_test_clear_prezeroed (struct folio *folio) +{ +if (page_private (&folio->page) ==MAGIC_PAGE_ZEROED ) { +set_page_private (&folio->page,0 ); +return true; + } +return false; +} +int arch_get_shadow_stack_status (struct task_struct *t,unsigned long __user *status);int arch_set_shadow_stack_status (struct task_struct *t,unsigned long status);int arch_lock_shadow_stack_status (struct task_struct *t,unsigned long status);
mm/page_alloc.c +7 -1
@@ -1867,7 +1867,13 @@ inline void post_alloc_hook(struct page *page, unsigned int order,bool zero_tags = init && (gfp_flags & __GFP_ZEROTAGS);int i; -set_page_private (page,0 ); +/* +* If the page is pre-zeroed and the caller opted in via +* __GFP_PREZEROED, preserve the marker so the caller can +* skip its own zeroing. Otherwise always clear private. +*/ +if (!(prezeroed && (gfp_flags & __GFP_PREZEROED))) +set_page_private (page,0 );/* * If the page is pre-zeroed, skip memory initialization.
Signature · 1 line
MST
David Hildenbrand (Arm)
maintainerRe: [PATCH RFC 3/9] mm: add __GFP_PREZEROED flag and folio_test_clear_prezeroed()
Quoted 10 lines · Michael S. Tsirkin
The previous patch skips zeroing in
post_alloc_hook()when__GFP_ZEROis used. However, several page allocation paths zero pages viafolio_zero_user()orclear_user_highpage()after allocation, not via__GFP_ZERO.Add
__GFP_PREZEROEDgfp flag that tellspost_alloc_hook()to preserve theMAGIC_PAGE_ZEROEDsentinel inpage->privateso the caller can detect pre-zeroed pages and skip its own zeroing. Addfolio_test_clear_prezeroed()helper to check and clear the sentinel.
I really don't like __GFP_PREZEROED, and wonder how we can avoid it.
What you want is, allocate a folio (well, actually a page that becomes a folio) and know whether zeroing for that folio (once we establish it from a page) is still required.
Or you just allocate a folio, specify GFP_ZERO, and let the folio allocation code deal with that.
I think we have two options:
(1) Use an indication that can be sticky for callers that do not care.
Assuming we would use a page flag that is only ever used on folios, all we'd have to do is make sure that we clear the flag once we convert the to a folio.
For example, PG_dropbehind is only ever set on folios in the pagecache.
Paths that allocate folios would have to clear the flag. For non-hugetlb folios that happens through page_rmappable_folio().
I'm not super-happy about that, but it would be doable.
(2) Use a dedicated allocation interface for user pages in the buddy.
I hate the whole user_alloc_needs_zeroing()+folio_zero_user() handling.
It shouldn't exist. We should just be passing GFP_ZERO and let the buddy handle all that.
For example, vma_alloc_folio() already gets passed the address in.
Pass the address from vma_alloc_folio_noprof()->folio_alloc_noprof(), and let folio_alloc_noprof() use a buddy interface that can handle it.
Imagine if we had a alloc_user_pages_noprof() that consumes an address. It could just do what folio_zero_user() does, and only if really required.
The whole user_alloc_needs_zeroing() could go away and you could just handle the pre-zeroed optimization internally.
Signature · 3 lines
Cheers, David
Michael S. Tsirkin
maintainerRe: [PATCH RFC 3/9] mm: add __GFP_PREZEROED flag and folio_test_clear_prezeroed()
Quoted 49 lines · David Hildenbrand (Arm)
Quoted 10 lines · Michael S. Tsirkin
The previous patch skips zeroing in
post_alloc_hook()when__GFP_ZEROis used. However, several page allocation paths zero pages viafolio_zero_user()orclear_user_highpage()after allocation, not via__GFP_ZERO.Add
__GFP_PREZEROEDgfp flag that tellspost_alloc_hook()to preserve theMAGIC_PAGE_ZEROEDsentinel inpage->privateso the caller can detect pre-zeroed pages and skip its own zeroing. Addfolio_test_clear_prezeroed()helper to check and clear the sentinel.I really don't like
__GFP_PREZEROED, and wonder how we can avoid it.What you want is, allocate a folio (well, actually a page that becomes a folio) and know whether zeroing for that folio (once we establish it from a page) is still required.
Or you just allocate a folio, specify
GFP_ZERO, and let the folio allocation code deal with that.I think we have two options:
(1) Use an indication that can be sticky for callers that do not care.
Assuming we would use a page flag that is only ever used on folios, all we'd have to do is make sure that we clear the flag once we convert the to a folio.
For example,
PG_dropbehindis only ever set on folios in the pagecache.Paths that allocate folios would have to clear the flag. For non-hugetlb folios that happens through
page_rmappable_folio().I'm not super-happy about that, but it would be doable.
(2) Use a dedicated allocation interface for user pages in the buddy.
I hate the whole user_alloc_needs_zeroing()+folio_zero_user() handling.
It shouldn't exist. We should just be passing
GFP_ZEROand let the buddy handle all that.For example,
vma_alloc_folio()already gets passed the address in.
Quoted 13 lines
Pass the address from vma_alloc_folio_noprof()->folio_alloc_noprof(), and let
folio_alloc_noprof()use a buddy interface that can handle it.Imagine if we had a
alloc_user_pages_noprof()that consumes an address. It could just do whatfolio_zero_user()does, and only if really required.The whole
user_alloc_needs_zeroing()could go away and you could just handle the pre-zeroed optimization internally.Signature · 3 lines
Cheers, David
I admit I only vaguely understand the core mm refactoring you are suggesting.
Signature · 1 line
MST
David Hildenbrand (Arm)
maintainerRe: [PATCH RFC 3/9] mm: add __GFP_PREZEROED flag and folio_test_clear_prezeroed()
Quoted 59 lines · Michael S. Tsirkin
Quoted 58 lines · David Hildenbrand (Arm)
Quoted 10 lines · Michael S. Tsirkin
The previous patch skips zeroing in
post_alloc_hook()when__GFP_ZEROis used. However, several page allocation paths zero pages viafolio_zero_user()orclear_user_highpage()after allocation, not via__GFP_ZERO.Add
__GFP_PREZEROEDgfp flag that tellspost_alloc_hook()to preserve theMAGIC_PAGE_ZEROEDsentinel inpage->privateso the caller can detect pre-zeroed pages and skip its own zeroing. Addfolio_test_clear_prezeroed()helper to check and clear the sentinel.I really don't like
__GFP_PREZEROED, and wonder how we can avoid it.What you want is, allocate a folio (well, actually a page that becomes a folio) and know whether zeroing for that folio (once we establish it from a page) is still required.
Or you just allocate a folio, specify
GFP_ZERO, and let the folio allocation code deal with that.I think we have two options:
(1) Use an indication that can be sticky for callers that do not care.
Assuming we would use a page flag that is only ever used on folios, all we'd have to do is make sure that we clear the flag once we convert the to a folio.
For example,
PG_dropbehindis only ever set on folios in the pagecache.Paths that allocate folios would have to clear the flag. For non-hugetlb folios that happens through
page_rmappable_folio().I'm not super-happy about that, but it would be doable.
(2) Use a dedicated allocation interface for user pages in the buddy.
I hate the whole user_alloc_needs_zeroing()+folio_zero_user() handling.
It shouldn't exist. We should just be passing
GFP_ZEROand let the buddy handle all that.For example,
vma_alloc_folio()already gets passed the address in.Pass the address from vma_alloc_folio_noprof()->folio_alloc_noprof(), and let
folio_alloc_noprof()use a buddy interface that can handle it.Imagine if we had a
alloc_user_pages_noprof()that consumes an address. It could just do whatfolio_zero_user()does, and only if really required.The whole
user_alloc_needs_zeroing()could go away and you could just handle the pre-zeroed optimization internally.
Quoted 7 lines
Quoted 4 lines
Signature · 3 lines
Cheers, DavidI admit I only vaguely understand the core mm refactoring you are suggesting.
Oh, I was hoping claude would figure that out for you.
Essentially, we move the zeroing of folios back into the buddy, by using GFP_ZERO.
user_alloc_needs_zeroing() logic would reside in the buddy and is no longer required in callers.
E.g.,
mm/memory.c +1 -10
@@ -5259,7 +5259,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) gfp =vma_thp_gfp_mask (vma);while (orders) { addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << order); - folio =vma_alloc_folio (gfp, order, vma, addr); + folio =vma_alloc_folio (gfp |GFP_ZERO , order, vma, addr);if (!folio) goto next;if (mem_cgroup_charge (folio, vma->vm_mm, gfp)) {
@@ -5272,15 +5272,6 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto fallback; }folio_throttle_swaprate (folio, gfp); -/* -* When a folio is not zeroed during allocation -* (__GFP_ZERO not used) or user folios require special -* handling, folio_zero_user() is used to make sure -* that the page corresponding to the faulting address -* will be hot in the cache after zeroing. -*/ -if (user_alloc_needs_zeroing ()) -folio_zero_user (folio, vmf->address);return folio; next:count_mthp_stat (order,MTHP_STAT_ANON_FAULT_FALLBACK );
folio_zero_user(), from where we would extract a function that operates on a page+order chunk, requires the address hint.
So we would have to pass that address. For example for the !CONFIG_NUMA case, something like the following could be done.
include/linux/gfp.h +17 -1
@@ -234,6 +234,10 @@ struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_ nodemask_t *nodemask);#define __folio_alloc (...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__)) +struct folio *__folio_alloc_user_noprof (gfp_t gfp,unsigned int order,int preferred_nid, +nodemask_t *nodemask,unsigned long addr); +#define __folio_alloc_user (...) alloc_hooks(__folio_alloc_user_noprof(__VA_ARGS__)) +unsigned long alloc_pages_bulk_noprof (gfp_t gfp,int preferred_nid,nodemask_t *nodemask,int nr_pages,struct page **page_array);
@@ -291,6 +295,18 @@ __alloc_pages_node_noprof(int nid, gfp_t gfp_mask, unsigned int order)#define __alloc_pages_node (...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) +static inline +struct folio *__folio_alloc_user_node_noprof (gfp_t gfp,unsigned int order, +int nid,unsigned long addr) +{ +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); +warn_if_node_offline (nid, gfp); + +return __folio_alloc_user_noprof (gfp, order, nid,NULL , addr); +} + +#define __folio_alloc_user_node (...) alloc_hooks(__folio_alloc_user_node_noprof(__VA_ARGS__)) +static inline struct folio *__folio_alloc_node_noprof (gfp_t gfp,unsigned int order,int nid) {
@@ -342,7 +358,7 @@ static inline struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int ordestatic inline struct folio *vma_alloc_folio_noprof (gfp_t gfp,int order,struct vm_area_struct *vma,unsigned long addr) { -return folio_alloc_noprof (gfp, order); +return __folio_alloc_user_node_noprof (gfp, order,numa_node_id (), addr); }#endif
index ee81f5c67c18..28f448f40b75 100644
mm/page_alloc.c +7 -0
@@ -5260,6 +5260,13 @@ struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_ }EXPORT_SYMBOL (__folio_alloc_noprof); +struct folio *__folio_alloc_user_noprof (gfp_t gfp,unsigned int order,int preferred_nid, +nodemask_t *nodemask,unsigned long addr) +{ +/* TODO */ +} +EXPORT_SYMBOL (__folio_alloc_noprof); +/* * Common helper functions. Never use with __GFP_HIGHMEM because the returned * address cannot represent highmem pages. Use alloc_pages and then kmap if
As alloc_user_pages() resides in the buddy, it can just honor any buddy-internal "pre-zeroed" flag.
Once you are in page_alloc.c you can access internal allocation functions and take care of that without GFP flags.
Signature · 3 lines
Cheers, David
Michael S. Tsirkin
maintainerRe: [PATCH RFC 3/9] mm: add __GFP_PREZEROED flag and folio_test_clear_prezeroed()
Quoted 63 lines · David Hildenbrand (Arm)
Quoted 62 lines · Michael S. Tsirkin
Quoted 61 lines · David Hildenbrand (Arm)
Quoted 10 lines · Michael S. Tsirkin
The previous patch skips zeroing in
post_alloc_hook()when__GFP_ZEROis used. However, several page allocation paths zero pages viafolio_zero_user()orclear_user_highpage()after allocation, not via__GFP_ZERO.Add
__GFP_PREZEROEDgfp flag that tellspost_alloc_hook()to preserve theMAGIC_PAGE_ZEROEDsentinel inpage->privateso the caller can detect pre-zeroed pages and skip its own zeroing. Addfolio_test_clear_prezeroed()helper to check and clear the sentinel.I really don't like
__GFP_PREZEROED, and wonder how we can avoid it.What you want is, allocate a folio (well, actually a page that becomes a folio) and know whether zeroing for that folio (once we establish it from a page) is still required.
Or you just allocate a folio, specify
GFP_ZERO, and let the folio allocation code deal with that.I think we have two options:
(1) Use an indication that can be sticky for callers that do not care.
Assuming we would use a page flag that is only ever used on folios, all we'd have to do is make sure that we clear the flag once we convert the to a folio.
For example,
PG_dropbehindis only ever set on folios in the pagecache.Paths that allocate folios would have to clear the flag. For non-hugetlb folios that happens through
page_rmappable_folio().I'm not super-happy about that, but it would be doable.
(2) Use a dedicated allocation interface for user pages in the buddy.
I hate the whole user_alloc_needs_zeroing()+folio_zero_user() handling.
It shouldn't exist. We should just be passing
GFP_ZEROand let the buddy handle all that.For example,
vma_alloc_folio()already gets passed the address in.Pass the address from vma_alloc_folio_noprof()->folio_alloc_noprof(), and let
folio_alloc_noprof()use a buddy interface that can handle it.Imagine if we had a
alloc_user_pages_noprof()that consumes an address. It could just do whatfolio_zero_user()does, and only if really required.The whole
user_alloc_needs_zeroing()could go away and you could just handle the pre-zeroed optimization internally.Signature · 1 line
Cheers,
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
David
I admit I only vaguely understand the core mm refactoring you are suggesting.
Oh, I was hoping claude would figure that out for you.
We figured it out together)
Quoted 108 lines
Essentially, we move the zeroing of folios back into the buddy, by using
GFP_ZERO.
user_alloc_needs_zeroing()logic would reside in the buddy and is no longer required in callers.E.g.,
mm/memory.c +1
-10@@ -5259,7 +5259,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) gfp =vma_thp_gfp_mask (vma);while (orders) { addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << order); - folio =vma_alloc_folio (gfp, order, vma, addr); + folio =vma_alloc_folio (gfp |GFP_ZERO , order, vma, addr);if (!folio) goto next;if (mem_cgroup_charge (folio, vma->vm_mm, gfp)) {@@ -5272,15 +5272,6 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto fallback; }folio_throttle_swaprate (folio, gfp); -/* -* When a folio is not zeroed during allocation -* (__GFP_ZERO not used) or user folios require special -* handling, folio_zero_user() is used to make sure -* that the page corresponding to the faulting address -* will be hot in the cache after zeroing. -*/ -if (user_alloc_needs_zeroing ()) -folio_zero_user (folio, vmf->address);return folio; next:count_mthp_stat (order,MTHP_STAT_ANON_FAULT_FALLBACK );
folio_zero_user(), from where we would extract a function that operates on a page+order chunk, requires the address hint.So we would have to pass that address. For example for the !CONFIG_NUMA case, something like the following could be done.
include/linux/gfp.h +17
-1@@ -234,6 +234,10 @@ struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_ nodemask_t *nodemask);#define __folio_alloc (...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__)) +struct folio *__folio_alloc_user_noprof (gfp_t gfp,unsigned int order,int preferred_nid, +nodemask_t *nodemask,unsigned long addr); +#define __folio_alloc_user (...) alloc_hooks(__folio_alloc_user_noprof(__VA_ARGS__)) +unsigned long alloc_pages_bulk_noprof (gfp_t gfp,int preferred_nid,nodemask_t *nodemask,int nr_pages,struct page **page_array);@@ -291,6 +295,18 @@ __alloc_pages_node_noprof(int nid, gfp_t gfp_mask, unsigned int order)#define __alloc_pages_node (...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) +static inline +struct folio *__folio_alloc_user_node_noprof (gfp_t gfp,unsigned int order, +int nid,unsigned long addr) +{ +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); +warn_if_node_offline (nid, gfp); + +return __folio_alloc_user_noprof (gfp, order, nid,NULL , addr); +} + +#define __folio_alloc_user_node (...) alloc_hooks(__folio_alloc_user_node_noprof(__VA_ARGS__)) +static inline struct folio *__folio_alloc_node_noprof (gfp_t gfp,unsigned int order,int nid) {@@ -342,7 +358,7 @@ static inline struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int ordestatic inline struct folio *vma_alloc_folio_noprof (gfp_t gfp,int order,struct vm_area_struct *vma,unsigned long addr) { -return folio_alloc_noprof (gfp, order); +return __folio_alloc_user_node_noprof (gfp, order,numa_node_id (), addr); }#endif index
ee81f5c67c18..28f448f40b75100644mm/page_alloc.c +7
-0@@ -5260,6 +5260,13 @@ struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_ }EXPORT_SYMBOL (__folio_alloc_noprof); +struct folio *__folio_alloc_user_noprof (gfp_t gfp,unsigned int order,int preferred_nid, +nodemask_t *nodemask,unsigned long addr) +{ +/* TODO */ +} +EXPORT_SYMBOL (__folio_alloc_noprof); +/* * Common helper functions. Never use with __GFP_HIGHMEM because the returned * address cannot represent highmem pages. Use alloc_pages and then kmap if
Quoted 11 lines
As
alloc_user_pages()resides in the buddy, it can just honor any buddy-internal "pre-zeroed" flag.Once you are in
page_alloc.cyou can access internal allocation functions and take care of that without GFP flags.Signature · 3 lines
Cheers, David
Pretty much what I did, except I felt it is better to change the existing APIs. A bit more churn, but in return there's less of a chance we forget to pass the user address.
Because, if we do, the result works fine on x86 but corrupts memory on other arches.
Signature · 1 line
MST
David Hildenbrand (Arm)
maintainerRe: [PATCH RFC 3/9] mm: add __GFP_PREZEROED flag and folio_test_clear_prezeroed()
Quoted 9 lines · Michael S. Tsirkin
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 3 lines · Michael S. Tsirkin
I admit I only vaguely understand the core mm refactoring you are suggesting.
Oh, I was hoping claude would figure that out for you.
We figured it out together)
:)
[...]
Quoted 8 lines
Pretty much what I did, except I felt it is better to change the existing APIs. A bit more churn, but in return there's less of a chance we forget to pass the user address.
Because, if we do, the result works fine on x86 but corrupts memory on other arches.
Converting everything is probably cleaner long term, But there are many instances where we just don't really have an address, or the address is irrelevant, because we wouldn't care about the difference when zeroing.
vma_alloc_folio() is the API to use when we have a VMA + address.
But yeah, something that is harder to mess up would be nice.
Signature · 3 lines
Cheers, David
Michael S. Tsirkin
maintainerRe: [PATCH RFC 3/9] mm: add __GFP_PREZEROED flag and folio_test_clear_prezeroed()
Quoted 24 lines · David Hildenbrand (Arm)
Quoted 9 lines · Michael S. Tsirkin
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 3 lines · Michael S. Tsirkin
I admit I only vaguely understand the core mm refactoring you are suggesting.
Oh, I was hoping claude would figure that out for you.
We figured it out together)
:)
[...]
Quoted 8 lines
Pretty much what I did, except I felt it is better to change the existing APIs. A bit more churn, but in return there's less of a chance we forget to pass the user address.
Because, if we do, the result works fine on x86 but corrupts memory on other arches.
Quoted 7 lines
Converting everything is probably cleaner long term, But there are many instances where we just don't really have an address, or the address is irrelevant, because we wouldn't care about the difference when zeroing.
vma_alloc_folio()is the API to use when we have a VMA + address.But yeah, something that is harder to mess up would be nice.
Well ... I am making progress on this but it is very intrusive.
Are we sure we do not want simply this, just maybe renaming the flags?
For example:__GFP_PREZERO -> __GFP_ZERO_HINT - request a hint that the allocated page is zero
And an API hiding the fact that the hint is in page->private.
Hmm?
Quoted 4 lines
Signature · 3 lines
Cheers, David
Michael S. Tsirkin
maintainerRe: [PATCH RFC 3/9] mm: add __GFP_PREZEROED flag and folio_test_clear_prezeroed()
In reply to David Hildenbrand (Arm)Quoted 31 lines · David Hildenbrand (Arm)
Quoted 10 lines · Michael S. Tsirkin
The previous patch skips zeroing in
post_alloc_hook()when__GFP_ZEROis used. However, several page allocation paths zero pages viafolio_zero_user()orclear_user_highpage()after allocation, not via__GFP_ZERO.Add
__GFP_PREZEROEDgfp flag that tellspost_alloc_hook()to preserve theMAGIC_PAGE_ZEROEDsentinel inpage->privateso the caller can detect pre-zeroed pages and skip its own zeroing. Addfolio_test_clear_prezeroed()helper to check and clear the sentinel.I really don't like
__GFP_PREZEROED, and wonder how we can avoid it.What you want is, allocate a folio (well, actually a page that becomes a folio) and know whether zeroing for that folio (once we establish it from a page) is still required.
Or you just allocate a folio, specify
GFP_ZERO, and let the folio allocation code deal with that.I think we have two options:
(1) Use an indication that can be sticky for callers that do not care.
Assuming we would use a page flag that is only ever used on folios, all we'd have to do is make sure that we clear the flag once we convert the to a folio.
Quoted 6 lines
For example,
PG_dropbehindis only ever set on folios in the pagecache.Paths that allocate folios would have to clear the flag. For non-hugetlb folios that happens through
page_rmappable_folio().I'm not super-happy about that, but it would be doable.
I suspect PG_dropbehind (or any flag, e.g. PG_owner_priv_1 that the patch that I sent uses) won't work as-is for this. The issue is PAGE_FLAGS_CHECK_AT_PREP:
#define PAGE_FLAGS_CHECK_AT_PREP \
((PAGEFLAGS_MASK & ~__PG_HWPOISON) | ...)This includes all page flags except hwpoison. check_new_pages() verifies that none of these flags are set on an allocated page.
PG_dropbehind is part of PAGEFLAGS_MASK, so if we set it in page_del_and_expand() to mark a page as pre-zeroed, check_new_pages() would reject it as a bad page.
I guess we could exclude it unconditionally, but this looks like a riskier change to me. No?
Quoted 11 lines
(2) Use a dedicated allocation interface for user pages in the buddy.
I hate the whole user_alloc_needs_zeroing()+folio_zero_user() handling.
It shouldn't exist. We should just be passing
GFP_ZEROand let the buddy handle all that.For example,
vma_alloc_folio()already gets passed the address in.
Quoted 8 lines
Pass the address from vma_alloc_folio_noprof()->folio_alloc_noprof(), and let
folio_alloc_noprof()use a buddy interface that can handle it.Imagine if we had a
alloc_user_pages_noprof()that consumes an address. It could just do whatfolio_zero_user()does, and only if really required.The whole
user_alloc_needs_zeroing()could go away and you could just handle the pre-zeroed optimization internally.
It's all rather messy, from what I saw so far there are arch specific hacks actually around this.
Quoted 4 lines
Signature · 3 lines
Cheers, David
Michael S. Tsirkin
maintainerRe: [PATCH RFC 3/9] mm: add __GFP_PREZEROED flag and folio_test_clear_prezeroed()
In reply to David Hildenbrand (Arm)On Mon, Apr 13, 2026 at 11:05:40AM +0200, David Hildenbrand (Arm) wrote:
...
Quoted 10 lines
(2) Use a dedicated allocation interface for user pages in the buddy.
I hate the whole user_alloc_needs_zeroing()+folio_zero_user() handling.
It shouldn't exist. We should just be passing
GFP_ZEROand let the buddy handle all that.For example,
vma_alloc_folio()already gets passed the address in.
Quoted 8 lines
Pass the address from vma_alloc_folio_noprof()->folio_alloc_noprof(), and let
folio_alloc_noprof()use a buddy interface that can handle it.Imagine if we had a
alloc_user_pages_noprof()that consumes an address. It could just do whatfolio_zero_user()does, and only if really required.The whole
user_alloc_needs_zeroing()could go away and you could just handle the pre-zeroed optimization internally.
I looked at this a bit, and I think the issue is that the buddy allocator doesn't do the arch-specific cache handling.
So allocating it is a fundamentally different thing from GFP_ZERO which is "zero a kernel address range".
So I don't get how you want to do it.
Quoted 5 lines
Signature · 4 lines
-- Cheers, David
Michael S. Tsirkin
maintainerRe: [PATCH RFC 3/9] mm: add __GFP_PREZEROED flag and folio_test_clear_prezeroed()
Quoted 23 lines · Michael S. Tsirkin
On Mon, Apr 13, 2026 at 11:05:40AM +0200, David Hildenbrand (Arm) wrote:
...
Quoted 10 lines
(2) Use a dedicated allocation interface for user pages in the buddy.
I hate the whole user_alloc_needs_zeroing()+folio_zero_user() handling.
It shouldn't exist. We should just be passing
GFP_ZEROand let the buddy handle all that.For example,
vma_alloc_folio()already gets passed the address in.Quoted 8 lines
Pass the address from vma_alloc_folio_noprof()->folio_alloc_noprof(), and let
folio_alloc_noprof()use a buddy interface that can handle it.Imagine if we had a
alloc_user_pages_noprof()that consumes an address. It could just do whatfolio_zero_user()does, and only if really required.The whole
user_alloc_needs_zeroing()could go away and you could just handle the pre-zeroed optimization internally.
Quoted 6 lines
I looked at this a bit, and I think the issue is that the buddy allocator doesn't do the arch-specific cache handling.
So allocating it is a fundamentally different thing fromGFP_ZEROwhich is "zero a kernel address range".So I don't get how you want to do it.
Oh, wait, do you mean we thread the userspace address through all the allocation calls? Like the below? This is on top of my patches, on top of mm it will be
a tiny bit smaller. I can rebase no problem.
But: isn't it a bit overkill for something that is, in the end, virtualization specific? It's all mechanical threading through of user_addr, but should we miss
one place, and suddenly you get weird corruption on esoteric arches
since we will get memset instead of folio_zero_user.
Worth it?
Let me know.
fs/hugetlbfs/inode.c +0 -2
@@ -828,8 +828,6 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, error =PTR_ERR (folio); goto out; } -if (!folio_test_clear_prezeroed (folio)) -folio_zero_user (folio, addr);__folio_mark_uptodate (folio); error =hugetlb_add_to_page_cache (folio, mapping, index);if (unlikely (error)) {
include/linux/gfp.h +12 -8
@@ -227,11 +227,11 @@ static inline void arch_alloc_page(struct page *page, int order) { }#endif struct page *__alloc_pages_noprof (gfp_t gfp,unsigned int order,int preferred_nid, -nodemask_t *nodemask); +nodemask_t *nodemask,unsigned long user_addr);#define __alloc_pages (...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__))struct folio *__folio_alloc_noprof (gfp_t gfp,unsigned int order,int preferred_nid, -nodemask_t *nodemask); +nodemask_t *nodemask,unsigned long user_addr);#define __folio_alloc (...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__)) unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
@@ -286,7 +286,7 @@ __alloc_pages_node_noprof(int nid, gfp_t gfp_mask, unsigned int order)VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES );warn_if_node_offline (nid, gfp_mask); -return __alloc_pages_noprof (gfp_mask, order, nid,NULL ); +return __alloc_pages_noprof (gfp_mask, order, nid,NULL ,0 ); }#define __alloc_pages_node (...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__))
@@ -297,7 +297,7 @@ struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES );warn_if_node_offline (nid, gfp); -return __folio_alloc_noprof (gfp, order, nid,NULL ); +return __folio_alloc_noprof (gfp, order, nid,NULL ,0 ); }#define __folio_alloc_node (...) alloc_hooks(__folio_alloc_node_noprof(__VA_ARGS__))
@@ -322,7 +322,8 @@ static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask,struct page *alloc_pages_noprof (gfp_t gfp,unsigned int order);struct folio *folio_alloc_noprof (gfp_t gfp,unsigned int order);struct folio *folio_alloc_mpol_noprof (gfp_t gfp,unsigned int order, -struct mempolicy *mpol,pgoff_t ilx,int nid); +struct mempolicy *mpol,pgoff_t ilx,int nid, +unsigned long user_addr);struct folio *vma_alloc_folio_noprof (gfp_t gfp,int order,struct vm_area_struct *vma,unsigned long addr);#else
@@ -335,14 +336,17 @@ static inline struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order) return __folio_alloc_node_noprof(gfp, order, numa_node_id()); }static inline struct folio *folio_alloc_mpol_noprof (gfp_t gfp,unsigned int order, -struct mempolicy *mpol,pgoff_t ilx,int nid) +struct mempolicy *mpol,pgoff_t ilx,int nid, +unsigned long user_addr) { -return folio_alloc_noprof (gfp, order); +return __folio_alloc_noprof (gfp | __GFP_COMP, order,numa_node_id (), +NULL , user_addr); }static inline struct folio *vma_alloc_folio_noprof (gfp_t gfp,int order,struct vm_area_struct *vma,unsigned long addr) { -return folio_alloc_noprof (gfp, order); +return folio_alloc_mpol_noprof (gfp, order,NULL ,0 ,numa_node_id (), + addr); }#endif
include/linux/gfp_types.h +0 -5
@@ -56,7 +56,6 @@ enum { ___GFP_NOLOCKDEP_BIT, #endif ___GFP_NO_OBJ_EXT_BIT, - ___GFP_PREZEROED_BIT, ___GFP_LAST_BIT };
@@ -98,7 +97,6 @@ enum {#define ___GFP_NOLOCKDEP 0#endif #define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT) -#define ___GFP_PREZEROED BIT(___GFP_PREZEROED_BIT)/* * Physical address zone modifiers (see linux/mmzone.h - low four bits)
@@ -294,9 +292,6 @@ enum {#define __GFP_SKIP_ZERO ((__force gfp_t)___GFP_SKIP_ZERO)#define __GFP_SKIP_KASAN ((__force gfp_t)___GFP_SKIP_KASAN) -/* Caller handles pre-zeroed pages; preserve PagePrezeroed */ -#define __GFP_PREZEROED ((__force gfp_t)___GFP_PREZEROED) -/* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)
include/linux/highmem.h +2 -9
@@ -320,15 +320,8 @@ static inlinestruct folio *vma_alloc_zeroed_movable_folio (struct vm_area_struct *vma,unsigned long vaddr) { -struct folio *folio; - - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_PREZEROED, -0 , vma, vaddr); -if (folio &&user_alloc_needs_zeroing () && - !folio_test_clear_prezeroed (folio)) -clear_user_highpage (&folio->page, vaddr); - -return folio; +return vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_ZERO, +0 , vma, vaddr); }#endif
include/linux/hugetlb.h +1 -1
@@ -937,7 +937,7 @@ static inline bool hugepage_movable_supported(struct hstate *h)/* Movability of hugepages depends on migration support. */ static inline gfp_t htlb_alloc_mask (struct hstate *h) { -gfp_t gfp = __GFP_COMP | __GFP_NOWARN | __GFP_PREZEROED; +gfp_t gfp = __GFP_COMP | __GFP_NOWARN | __GFP_ZERO; gfp |=hugepage_movable_supported (h) ?GFP_HIGHUSER_MOVABLE :GFP_HIGHUSER ;
include/linux/page-flags.h +2 -3
@@ -682,9 +682,8 @@ FOLIO_FLAG_FALSE(idle)#endif /* -* PagePrezeroed() tracks pages known to be zero. The -* allocator may preserve this bit for __GFP_PREZEROED callers so they can -* skip redundant zeroing after allocation. +* PagePrezeroed() tracks pages known to be zero. The allocator +* uses this to skip redundant zeroing in post_alloc_hook(). */ __PAGEFLAG (Prezeroed, prezeroed,PF_NO_COMPOUND )
mm/compaction.c +2 -2
@@ -82,7 +82,7 @@ static inline bool is_via_compact_memory(int order) { return false; }static struct page *mark_allocated_noprof (struct page *page,unsigned int order,gfp_t gfp_flags) { -post_alloc_hook (page, order, __GFP_MOVABLE, false); +post_alloc_hook (page, order, __GFP_MOVABLE, false,0 );set_page_refcounted (page);return page; }
@@ -1833,7 +1833,7 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long da } dst = (struct folio *)freepage; -post_alloc_hook (&dst->page, order, __GFP_MOVABLE, false); +post_alloc_hook (&dst->page, order, __GFP_MOVABLE, false,0 );set_page_refcounted (&dst->page);if (order)prep_compound_page (&dst->page, order);
mm/filemap.c +1 -1
@@ -998,7 +998,7 @@ struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order,if (policy)return folio_alloc_mpol_noprof (gfp, order, policy, -NO_INTERLEAVE_INDEX ,numa_node_id ()); +NO_INTERLEAVE_INDEX ,numa_node_id (),0 );if (cpuset_do_page_mem_spread ()) {unsigned int cpuset_mems_cookie;
mm/huge_memory.c +1 -9
@@ -1256,7 +1256,7 @@ EXPORT_SYMBOL_GPL(thp_get_unmapped_area);static struct folio *vma_alloc_anon_folio_pmd (struct vm_area_struct *vma,unsigned long addr) { -gfp_t gfp =vma_thp_gfp_mask (vma) | __GFP_PREZEROED; +gfp_t gfp =vma_thp_gfp_mask (vma) | __GFP_ZERO;const int order =HPAGE_PMD_ORDER ;struct folio *folio;
@@ -1279,14 +1279,6 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, }folio_throttle_swaprate (folio, gfp); -/* -* When a folio is not zeroed during allocation (__GFP_ZERO not used) -* or user folios require special handling, folio_zero_user() is used to -* make sure that the page corresponding to the faulting address will be -* hot in the cache after zeroing. -*/ -if (user_alloc_needs_zeroing () && !folio_test_clear_prezeroed (folio)) -folio_zero_user (folio, addr);/* * The memory barrier inside __folio_mark_uptodate makes sure that * folio_zero_user writes become visible before the set_pmd_at()
mm/hugetlb.c +19 -18
@@ -1842,7 +1842,8 @@ struct address_space *hugetlb_folio_mapping_lock_write(struct folio *folio) }static struct folio *alloc_buddy_frozen_folio (int order,gfp_t gfp_mask, -int nid,nodemask_t *nmask,nodemask_t *node_alloc_noretry) +int nid,nodemask_t *nmask,nodemask_t *node_alloc_noretry, +unsigned long addr) {struct folio *folio;bool alloc_try_hard = true;
@@ -1859,7 +1860,7 @@ static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask,if (alloc_try_hard) gfp_mask |= __GFP_RETRY_MAYFAIL; - folio = (struct folio *)__alloc_frozen_pages (gfp_mask, order, nid, nmask); + folio = (struct folio *)__alloc_frozen_pages (gfp_mask, order, nid, nmask, addr);/* * If we did not specify __GFP_RETRY_MAYFAIL, but still got a
@@ -1888,7 +1889,7 @@ static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask,static struct folio *only_alloc_fresh_hugetlb_folio (struct hstate *h,gfp_t gfp_mask,int nid,nodemask_t *nmask, -nodemask_t *node_alloc_noretry) +nodemask_t *node_alloc_noretry,unsigned long addr) {struct folio *folio;int order =huge_page_order (h);
@@ -1900,7 +1901,7 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, folio =alloc_gigantic_frozen_folio (order, gfp_mask, nid, nmask); else folio =alloc_buddy_frozen_folio (order, gfp_mask, nid, nmask, - node_alloc_noretry); + node_alloc_noretry, addr);if (folio)init_new_hugetlb_folio (folio);return folio;
@@ -1914,11 +1915,12 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h,* pages is zero, and the accounting must be done in the caller. */ static struct folio *alloc_fresh_hugetlb_folio (struct hstate *h, -gfp_t gfp_mask,int nid,nodemask_t *nmask) +gfp_t gfp_mask,int nid,nodemask_t *nmask, +unsigned long addr) {struct folio *folio; - folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask,NULL ); + folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask,NULL , addr);if (folio)hugetlb_vmemmap_optimize_folio (h, folio);return folio;
@@ -1958,7 +1960,7 @@ static struct folio *alloc_pool_huge_folio(struct hstate *h,struct folio *folio; folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, node, - nodes_allowed, node_alloc_noretry); + nodes_allowed, node_alloc_noretry,0 );if (folio)return folio; }
@@ -2127,7 +2129,8 @@ int dissolve_free_hugetlb_folios(unsigned long start_pfn, unsigned long end_pfn)* Allocates a fresh surplus page from the page allocator. */ static struct folio *alloc_surplus_hugetlb_folio (struct hstate *h, -gfp_t gfp_mask,int nid,nodemask_t *nmask) +gfp_t gfp_mask,int nid,nodemask_t *nmask, +unsigned long addr) {struct folio *folio =NULL ;
@@ -2139,7 +2142,7 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h, goto out_unlock;spin_unlock_irq (&hugetlb_lock); - folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask); + folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask, addr);if (!folio)return NULL ;
@@ -2182,7 +2185,7 @@ static struct folio *alloc_migrate_hugetlb_folio(struct hstate *h, gfp_t gfp_masif (hstate_is_gigantic (h))return NULL ; - folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask); + folio =alloc_fresh_hugetlb_folio (h, gfp_mask, nid, nmask,0 );if (!folio)return NULL ;
@@ -2218,14 +2221,14 @@ struct folio *alloc_buddy_hugetlb_folio_with_mpol(struct hstate *h,if (mpol_is_preferred_many (mpol)) {gfp_t gfp = gfp_mask & ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - folio =alloc_surplus_hugetlb_folio (h, gfp, nid, nodemask); + folio =alloc_surplus_hugetlb_folio (h, gfp, nid, nodemask, addr);/* Fallback to all nodes if page==NULL */ nodemask =NULL ; }if (!folio) - folio =alloc_surplus_hugetlb_folio (h, gfp_mask, nid, nodemask); + folio =alloc_surplus_hugetlb_folio (h, gfp_mask, nid, nodemask, addr);mpol_cond_put (mpol);return folio; }
@@ -2332,7 +2335,7 @@ static int gather_surplus_pages(struct hstate *h, long delta)* down the road to pick the current node if that is the case. */ folio =alloc_surplus_hugetlb_folio (h,htlb_alloc_mask (h), -NUMA_NO_NODE , &alloc_nodemask); +NUMA_NO_NODE , &alloc_nodemask,0 );if (!folio) { alloc_ok = false;break ;
@@ -2738,7 +2741,7 @@ static int alloc_and_dissolve_hugetlb_folio(struct folio *old_folio,spin_unlock_irq (&hugetlb_lock); gfp_mask =htlb_alloc_mask (h) | __GFP_THISNODE; new_folio =alloc_fresh_hugetlb_folio (h, gfp_mask, - nid,NULL ); + nid,NULL ,0 );if (!new_folio)return -ENOMEM ; goto retry;
@@ -3434,13 +3437,13 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)gfp_t gfp_mask =htlb_alloc_mask (h) | __GFP_THISNODE; folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, - &node_states[N_MEMORY ],NULL ); + &node_states[N_MEMORY ],NULL ,0 );if (!folio && !list_empty (&folio_list) &&hugetlb_vmemmap_optimizable_size (h)) {prep_and_add_allocated_folios (h, &folio_list);INIT_LIST_HEAD (&folio_list); folio =only_alloc_fresh_hugetlb_folio (h, gfp_mask, nid, - &node_states[N_MEMORY ],NULL ); + &node_states[N_MEMORY ],NULL ,0 ); }if (!folio)break ;
@@ -5809,8 +5812,6 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, ret =0 ; goto out; } -if (!folio_test_clear_prezeroed (folio)) -folio_zero_user (folio, vmf->real_address);__folio_mark_uptodate (folio); new_folio = true;
mm/internal.h +3 -2
@@ -672,6 +672,7 @@ struct alloc_context {*/ enum zone_type highest_zoneidx;bool spread_dirty_pages; +unsigned long user_addr; };/*
@@ -888,13 +889,13 @@ static inline void prep_compound_tail(struct page *head, int tail_idx) }void post_alloc_hook (struct page *page,unsigned int order,gfp_t gfp_flags, -bool prezeroed); +bool prezeroed,unsigned long user_addr);extern bool free_pages_prepare (struct page *page,unsigned int order);extern int user_min_free_kbytes;struct page *__alloc_frozen_pages_noprof (gfp_t ,unsigned int order,int nid, -nodemask_t *); +nodemask_t *,unsigned long user_addr);#define __alloc_frozen_pages (...) \ alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))void free_frozen_pages (struct page *page,unsigned int order);
mm/khugepaged.c +1 -1
@@ -1055,7 +1055,7 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_struint node =hpage_collapse_find_target_node (cc);struct folio *folio; - folio =__folio_alloc (gfp,HPAGE_PMD_ORDER , node, &cc->alloc_nmask); + folio =__folio_alloc (gfp,HPAGE_PMD_ORDER , node, &cc->alloc_nmask,0 );if (!folio) { *foliop =NULL ;count_vm_event (THP_COLLAPSE_ALLOC_FAILED );
mm/memory.c +1 -11
@@ -5176,7 +5176,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto fallback;/* Try allocating the highest of the remaining orders. */ - gfp =vma_thp_gfp_mask (vma) | __GFP_PREZEROED; + gfp =vma_thp_gfp_mask (vma) | __GFP_ZERO;while (orders) { addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << order); folio =vma_alloc_folio (gfp, order, vma, addr);
@@ -5187,16 +5187,6 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto next; }folio_throttle_swaprate (folio, gfp); -/* -* When a folio is not zeroed during allocation -* (__GFP_ZERO not used) or user folios require special -* handling, folio_zero_user() is used to make sure -* that the page corresponding to the faulting address -* will be hot in the cache after zeroing. -*/ -if (user_alloc_needs_zeroing () && - !folio_test_clear_prezeroed (folio)) -folio_zero_user (folio, vmf->address);return folio; } next:
mm/mempolicy.c +12 -10
@@ -1454,7 +1454,7 @@ static struct folio *alloc_migration_target_by_mpol(struct folio *src, else gfp =GFP_HIGHUSER_MOVABLE | __GFP_RETRY_MAYFAIL | __GFP_COMP; - return folio_alloc_mpol(gfp, order, pol, ilx, nid); + return folio_alloc_mpol(gfp, order, pol, ilx, nid, 0); }#else
@@ -2419,9 +2419,9 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order,*/ preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - page =__alloc_frozen_pages_noprof (preferred_gfp, order, nid, nodemask); + page =__alloc_frozen_pages_noprof (preferred_gfp, order, nid, nodemask,0 );if (!page) - page =__alloc_frozen_pages_noprof (gfp, order, nid,NULL ); + page =__alloc_frozen_pages_noprof (gfp, order, nid,NULL ,0 );return page; }
@@ -2437,7 +2437,8 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order,* Return: The page on success or NULL if allocation fails. */ static struct page *alloc_pages_mpol (gfp_t gfp,unsigned int order, -struct mempolicy *pol,pgoff_t ilx,int nid) +struct mempolicy *pol,pgoff_t ilx,int nid, +unsigned long user_addr) {nodemask_t *nodemask;struct page *page;
@@ -2469,7 +2470,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,*/ page =__alloc_frozen_pages_noprof ( gfp | __GFP_THISNODE | __GFP_NORETRY, order, - nid,NULL ); + nid,NULL , user_addr);if (page || !(gfp & __GFP_DIRECT_RECLAIM))return page;/*
@@ -2481,7 +2482,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, } } - page =__alloc_frozen_pages_noprof (gfp, order, nid, nodemask); + page =__alloc_frozen_pages_noprof (gfp, order, nid, nodemask, user_addr);if (unlikely (pol->mode ==MPOL_INTERLEAVE || pol->mode ==MPOL_WEIGHTED_INTERLEAVE ) && page) {
@@ -2498,10 +2499,11 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, }struct folio *folio_alloc_mpol_noprof (gfp_t gfp,unsigned int order, -struct mempolicy *pol,pgoff_t ilx,int nid) +struct mempolicy *pol,pgoff_t ilx,int nid, +unsigned long user_addr) {struct page *page =alloc_pages_mpol (gfp | __GFP_COMP, order, pol, - ilx, nid); + ilx, nid, user_addr);if (!page)return NULL ;
@@ -2535,7 +2537,7 @@ struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct gfp |= __GFP_NOWARN; pol =get_vma_policy (vma, addr, order, &ilx); - folio =folio_alloc_mpol_noprof (gfp, order, pol, ilx,numa_node_id ()); + folio =folio_alloc_mpol_noprof (gfp, order, pol, ilx,numa_node_id (), addr);mpol_cond_put (pol);return folio; }
@@ -2553,7 +2555,7 @@ struct page *alloc_frozen_pages_noprof(gfp_t gfp, unsigned order) pol =get_task_policy (current);return alloc_pages_mpol (gfp, order, pol,NO_INTERLEAVE_INDEX , -numa_node_id ()); +numa_node_id (),0 ); }/**
mm/migrate.c +1 -1
@@ -2202,7 +2202,7 @@ struct folio *alloc_migration_target(struct folio *src, unsigned long private)if (is_highmem_idx (zidx) || zidx ==ZONE_MOVABLE ) gfp_mask |= __GFP_HIGHMEM; -return __folio_alloc (gfp_mask, order, nid, mtc->nmask); +return __folio_alloc (gfp_mask, order, nid, mtc->nmask,0 ); }#ifdef CONFIG_NUMA
mm/page_alloc.c +30 -27
145 lines not on this page · raw message
mm/page_frag_cache.c +2 -2
@@ -57,10 +57,10 @@ static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, gfp_mask = (gfp_mask & ~__GFP_DIRECT_RECLAIM) | __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC; page =__alloc_pages (gfp_mask,PAGE_FRAG_CACHE_MAX_ORDER , -numa_mem_id (),NULL ); +numa_mem_id (),NULL ,0 );#endif if (unlikely (!page)) { - page =__alloc_pages (gfp,0 ,numa_mem_id (),NULL ); + page =__alloc_pages (gfp,0 ,numa_mem_id (),NULL ,0 ); order =0 ; }
mm/shmem.c +1 -1
@@ -1927,7 +1927,7 @@ static struct folio *shmem_alloc_folio(gfp_t gfp, int order,struct folio *folio; mpol =shmem_get_pgoff_policy (info, index, order, &ilx); - folio =folio_alloc_mpol (gfp, order, mpol, ilx,numa_node_id ()); + folio =folio_alloc_mpol (gfp, order, mpol, ilx,numa_node_id (),0 );mpol_cond_put (mpol);return folio;
mm/slub.c +2 -2
@@ -3266,7 +3266,7 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node,else if (node ==NUMA_NO_NODE ) page =alloc_frozen_pages (flags, order);else - page =__alloc_frozen_pages (flags, order, node,NULL ); + page =__alloc_frozen_pages (flags, order, node,NULL ,0 );if (!page)return NULL ;
@@ -5178,7 +5178,7 @@ static void *___kmalloc_large_node(size_t size, gfp_t flags, int node)if (node ==NUMA_NO_NODE ) page =alloc_frozen_pages_noprof (flags, order);else - page =__alloc_frozen_pages_noprof (flags, order, node,NULL ); + page =__alloc_frozen_pages_noprof (flags, order, node,NULL ,0 );if (page) { ptr =page_address (page);
mm/swap_state.c +1 -1
@@ -568,7 +568,7 @@ struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_mask,return NULL ;/* Allocate a new folio to be added into the swap cache. */ - folio =folio_alloc_mpol (gfp_mask,0 , mpol, ilx,numa_node_id ()); + folio =folio_alloc_mpol (gfp_mask,0 , mpol, ilx,numa_node_id (),0 );if (!folio)return NULL ;/* Try add the new folio, returns existing folio or NULL on failure. */
David Hildenbrand (Arm)
maintainerRe: [PATCH RFC 3/9] mm: add __GFP_PREZEROED flag and folio_test_clear_prezeroed()
Quoted 24 lines · Michael S. Tsirkin
Quoted 23 lines · Michael S. Tsirkin
On Mon, Apr 13, 2026 at 11:05:40AM +0200, David Hildenbrand (Arm) wrote:
...
Quoted 18 lines
(2) Use a dedicated allocation interface for user pages in the buddy.
I hate the whole user_alloc_needs_zeroing()+folio_zero_user() handling.
It shouldn't exist. We should just be passing
GFP_ZEROand let the buddy handle all that.For example,
vma_alloc_folio()already gets passed the address in.Pass the address from vma_alloc_folio_noprof()->folio_alloc_noprof(), and let
folio_alloc_noprof()use a buddy interface that can handle it.Imagine if we had a
alloc_user_pages_noprof()that consumes an address. It could just do whatfolio_zero_user()does, and only if really required.The whole
user_alloc_needs_zeroing()could go away and you could just handle the pre-zeroed optimization internally.
Quoted 7 lines
Quoted 6 lines
I looked at this a bit, and I think the issue is that the buddy allocator doesn't do the arch-specific cache handling.
So allocating it is a fundamentally different thing fromGFP_ZEROwhich is "zero a kernel address range".So I don't get how you want to do it.
Saw your new mails just now.
Quoted 10 lines
Oh, wait, do you mean we thread the userspace address through all the allocation calls? Like the below? This is on top of my patches, on top of mm it will be
a tiny bit smaller. I can rebase no problem.But: isn't it a bit overkill for something that is, in the end, virtualization specific? It's all mechanical threading through of
user_addr, but should we miss
one place, and suddenly you get weird corruption on esoteric arches
since we will get memset instead offolio_zero_user.Worth it?
See my other mail, we'd be forwarding it only for vma_alloc_folio_noprof(), where we call a _user specific interface.
Then we can get rid of the user_alloc_needs_zeroing() hack and just do the right thing in page_alloc.c
Signature · 3 lines
Cheers, David
Michael S. Tsirkin
maintainerRFC PATCH 4/9 mm: skip zeroing in vma_alloc_zeroed_movable_folio for pre-zeroed pages
In reply to Michael S. TsirkinUse __GFP_PREZEROED and folio_test_clear_prezeroed() to skip clear_user_highpage() when the page is already zeroed.
On x86, vma_alloc_zeroed_movable_folio is overridden by a macro that uses __GFP_ZERO directly, so this change has no effect there.
--- include/linux/highmem.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
include/linux/highmem.h +4 -2
@@ -322,8 +322,10 @@ struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, {struct folio *folio; - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , vma, vaddr); -if (folio &&user_alloc_needs_zeroing ()) + folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE | __GFP_PREZEROED, +0 , vma, vaddr); +if (folio &&user_alloc_needs_zeroing () && + !folio_test_clear_prezeroed (folio))clear_user_highpage (&folio->page, vaddr);return folio;
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRFC PATCH 5/9 mm: skip zeroing in alloc_anon_folio for pre-zeroed pages
In reply to Michael S. TsirkinUse __GFP_PREZEROED and folio_test_clear_prezeroed() to skip folio_zero_user() in the mTHP anonymous page allocation path when the page is already zeroed.
--- mm/memory.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/memory.c +3 -2
@@ -5176,7 +5176,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) goto fallback;/* Try allocating the highest of the remaining orders. */ - gfp =vma_thp_gfp_mask (vma); + gfp =vma_thp_gfp_mask (vma) | __GFP_PREZEROED;while (orders) { addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << order); folio =vma_alloc_folio (gfp, order, vma, addr);
@@ -5194,7 +5194,8 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)* that the page corresponding to the faulting address * will be hot in the cache after zeroing. */ -if (user_alloc_needs_zeroing ()) +if (user_alloc_needs_zeroing () && + !folio_test_clear_prezeroed (folio))folio_zero_user (folio, vmf->address);return folio; }
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRFC PATCH 6/9 mm: skip zeroing in vma_alloc_anon_folio_pmd for pre-zeroed pages
In reply to Michael S. TsirkinUse __GFP_PREZEROED and folio_test_clear_prezeroed() to skip folio_zero_user() in the PMD THP anonymous page allocation path when the page is already zeroed.
--- mm/huge_memory.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/huge_memory.c +2 -2
@@ -1256,7 +1256,7 @@ EXPORT_SYMBOL_GPL(thp_get_unmapped_area);static struct folio *vma_alloc_anon_folio_pmd (struct vm_area_struct *vma,unsigned long addr) { -gfp_t gfp =vma_thp_gfp_mask (vma); +gfp_t gfp =vma_thp_gfp_mask (vma) | __GFP_PREZEROED;const int order =HPAGE_PMD_ORDER ;struct folio *folio;
@@ -1285,7 +1285,7 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma,* make sure that the page corresponding to the faulting address will be * hot in the cache after zeroing. */ -if (user_alloc_needs_zeroing ()) +if (user_alloc_needs_zeroing () && !folio_test_clear_prezeroed (folio))folio_zero_user (folio, addr);/* * The memory barrier inside __folio_mark_uptodate makes sure that
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRFC PATCH 7/9 mm: hugetlb: skip zeroing of pre-zeroed hugetlb pages
In reply to Michael S. TsirkinWhen a surplus hugetlb page is allocated from the buddy allocator and the page was previously reported to the host (and zeroed on reclaim), skip the redundant folio_zero_user() in the hugetlb fault path.
This only benefits surplus hugetlb pages that are freshly allocated from the buddy. Pages from the persistent hugetlb pool are not affected since they are not allocated from buddy at fault time.
--- fs/hugetlbfs/inode.c | 3 ++- include/linux/hugetlb.h | 2 +- mm/hugetlb.c | 3 ++- 3 files changed, 5 insertions(+), 3 deletions(-)
fs/hugetlbfs/inode.c +2 -1
@@ -828,7 +828,8 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, error =PTR_ERR (folio); goto out; } -folio_zero_user (folio, addr); +if (!folio_test_clear_prezeroed (folio)) +folio_zero_user (folio, addr);__folio_mark_uptodate (folio); error =hugetlb_add_to_page_cache (folio, mapping, index);if (unlikely (error)) {
include/linux/hugetlb.h +1 -1
@@ -937,7 +937,7 @@ static inline bool hugepage_movable_supported(struct hstate *h)/* Movability of hugepages depends on migration support. */ static inline gfp_t htlb_alloc_mask (struct hstate *h) { -gfp_t gfp = __GFP_COMP | __GFP_NOWARN; +gfp_t gfp = __GFP_COMP | __GFP_NOWARN | __GFP_PREZEROED; gfp |=hugepage_movable_supported (h) ?GFP_HIGHUSER_MOVABLE :GFP_HIGHUSER ;
mm/hugetlb.c +2 -1
@@ -5809,7 +5809,8 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, ret =0 ; goto out; } -folio_zero_user (folio, vmf->real_address); +if (!folio_test_clear_prezeroed (folio)) +folio_zero_user (folio, vmf->real_address);__folio_mark_uptodate (folio); new_folio = true;
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRFC PATCH 8/9 mm: page_reporting: add flush parameter to trigger immediate reporting
In reply to Michael S. TsirkinAdd a write-only module parameter 'flush' that triggers an immediate page reporting cycle. Writing any value flushes pending work and runs one cycle synchronously.
This is useful for testing and benchmarking the pre-zeroed page optimization, where the reporting delay (2 seconds) makes it hard to ensure pages are reported before measuring allocation performance.
echo 1 > /sys/module/page_reporting/parameters/flush
--- mm/page_reporting.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
mm/page_reporting.c +25 -0
@@ -351,6 +351,31 @@ static void page_reporting_process(struct work_struct *work)static DEFINE_MUTEX (page_reporting_mutex);DEFINE_STATIC_KEY_FALSE (page_reporting_enabled); +static int page_reporting_flush_set (const char *val, +const struct kernel_param *kp) +{ +struct page_reporting_dev_info *prdev; + +mutex_lock (&page_reporting_mutex); + prdev =rcu_dereference_protected (pr_dev_info, +lockdep_is_held (&page_reporting_mutex)); +if (prdev) { +flush_delayed_work (&prdev->work); +__page_reporting_request (prdev); +flush_delayed_work (&prdev->work); + } +mutex_unlock (&page_reporting_mutex); +return 0 ; +} + +static const struct kernel_param_ops flush_ops = { + .set = page_reporting_flush_set, + .get = param_get_uint, +}; +static unsigned int page_reporting_flush; +module_param_cb (flush, &flush_ops, &page_reporting_flush,0200 ); +MODULE_PARM_DESC (flush,"Trigger immediate page reporting cycle" ); +int page_reporting_register (struct page_reporting_dev_info *prdev) {int err =0 ;
Signature · 1 line
MST
Michael S. Tsirkin
maintainerRFC PATCH 9/9 virtio_balloon: a hack to enable host-zeroed page optimization
In reply to Michael S. TsirkinAdd a module parameter host_zeroes_pages to opt in to the pre-zeroed page optimization. A proper virtio feature flag is needed before this can be merged.
insmod virtio_balloon.ko host_zeroes_pages=1
--- drivers/virtio/virtio_balloon.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/virtio/virtio_balloon.c +7 -0
@@ -19,6 +19,11 @@#include <linux/mm.h> #include <linux/page_reporting.h> +static bool host_zeroes_pages; +module_param (host_zeroes_pages, bool,0644 ); +MODULE_PARM_DESC (host_zeroes_pages, +"Host zeroes reported pages, skip guest re-zeroing" ); +/* * Balloon device works in 4K page units. So each page is pointed to by * multiple balloon pages. All memory counters in this driver are in balloon
@@ -1039,6 +1044,8 @@ static int virtballoon_probe(struct virtio_device *vdev) vb->pr_dev_info.order =5 ;#endif +/* TODO: needs a virtio feature flag */ + vb->pr_dev_info.host_zeroes_pages = host_zeroes_pages; err =page_reporting_register (&vb->pr_dev_info);if (err) goto out_unregister_oom;
Signature · 1 line
MST