mm/page_alloc: replace kernel_init_pages() with batch page clearing
Subsystems (3)
PATCH v3 1/1 mm/page_alloc: replace kernel_init_pages() with batch page clearing
When init_on_alloc is enabled, kernel_init_pages() clears every page one at a time via clear_highpage_kasan_tagged(), which incurs per-page kmap_local_page()/kunmap_local() overhead and prevents the architecture clearing primitive from operating on contiguous ranges.
Introduce clear_highpages_kasan_tagged() in highmem.h, a batch clearing helper that calls clear_pages() for the full contiguous range on !HIGHMEM systems, bypassing the per-page kmap overhead and allowing a single invocation of the arch clearing primitive across the entire allocation. The HIGHMEM path falls back to per-page clearing since those pages require kmap.
Replace kernel_init_pages() with direct calls to the new helper, as it becomes a trivial wrapper.
Allocating 8192 x 2MB HugeTLB pages (16GB) with init_on_alloc=1:
Before: 0.445s
After: 0.166s (-62.7%, 2.68x faster)
Kernel time (sys) reduction per workload with init_on_alloc=1:
Workload Before After Change Graph500 64C128T 30m 41.8s 15m 14.8s -50.3% Graph500 16C32T 15m 56.7s 9m 43.7s -39.0% Pagerank 32T 1m 58.5s 1m 12.8s -38.5% Pagerank 128T 2m 36.3s 1m 40.4s -35.7%
---
base commit: 2bcc13c29c711381d815c1ba5d5b25737400c71a
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/
Changes since v2:
- Moved kasan_disable_current()/kasan_enable_current() into clear_highpages_kasan_tagged(), per David and Zi Yan's suggestion.
- Removed kernel_init_pages() and replaced its two call sites with direct calls to the helper.
Changes since v1:
- Dropped cond_resched() and PROCESS_PAGES_NON_PREEMPT_BATCH as kernel_init_pages() runs inside the page allocator and can be called from atomic context, making cond_resched() unsafe. The original code never had a cond_resched() here, and the performance gain comes from batching, not rescheduling.
- Moved the !HIGHMEM/HIGHMEM branching into a new clear_highpages_kasan_tagged() helper in highmem.h, per David's suggestion.
include/linux/highmem.h | 15 +++++++++++++++ mm/page_alloc.c | 15 ++------------- 2 files changed, 17 insertions(+), 13 deletions(-)
include/linux/highmem.h +15 -0
@@ -345,6 +345,21 @@ static inline void clear_highpage_kasan_tagged(struct page *page)kunmap_local (kaddr); } +static inline void clear_highpages_kasan_tagged (struct page *page,int numpages) +{ +/* s390's use of memset() could override KASAN redzones. */ +kasan_disable_current (); +if (!IS_ENABLED (CONFIG_HIGHMEM )) { +clear_pages (kasan_reset_tag (page_address (page)), numpages); + }else { +int i; + +for (i =0 ; i < numpages; i++) +clear_highpage_kasan_tagged (page + i); + } +kasan_enable_current (); +} +#ifndef __HAVE_ARCH_TAG_CLEAR_HIGHPAGES/* Return false to let people know we did not initialize the pages */
mm/page_alloc.c +2 -13
@@ -1208,17 +1208,6 @@ static inline bool should_skip_kasan_poison(struct page *page)return page_kasan_tag (page) ==KASAN_TAG_KERNEL ; } -static void kernel_init_pages (struct page *page,int numpages) -{ -int i; - -/* s390's use of memset() could override KASAN redzones. */ -kasan_disable_current (); -for (i =0 ; i < numpages; i++) -clear_highpage_kasan_tagged (page + i); -kasan_enable_current (); -} -#ifdef CONFIG_MEM_ALLOC_PROFILING /* Should be called only if mem_alloc_profiling_enabled() */
@@ -1428,7 +1417,7 @@ __always_inline bool __free_pages_prepare(struct page *page, init = false; }if (init) -kernel_init_pages (page,1 << order); +clear_highpages_kasan_tagged (page,1 << order);/* * arch_free_page() can make the page's contents inaccessible. s390
@@ -1853,7 +1842,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, }/* If memory is still not initialized, initialize it now. */ if (init) -kernel_init_pages (page,1 << order); +clear_highpages_kasan_tagged (page,1 << order);set_page_owner (page, order, gfp_flags);page_table_check_alloc (page, order);
Signature · 1 line
2.43.0
Quoted 20 lines · Hrushikesh Salunke
When
init_on_allocis enabled,kernel_init_pages()clears every page one at a time viaclear_highpage_kasan_tagged(), which incurs per-page kmap_local_page()/kunmap_local() overhead and prevents the architecture clearing primitive from operating on contiguous ranges.Introduce
clear_highpages_kasan_tagged()inhighmem.h, a batch clearing helper that callsclear_pages()for the full contiguous range on !HIGHMEM systems, bypassing the per-page kmap overhead and allowing a single invocation of the arch clearing primitive across the entire allocation. The HIGHMEM path falls back to per-page clearing since those pages require kmap.Replace
kernel_init_pages()with direct calls to the new helper, as it becomes a trivial wrapper.Allocating 8192 x 2MB HugeTLB pages (16GB) with init_on_alloc=1:
Before: 0.445s
After: 0.166s (-62.7%, 2.68x faster)
Quoted 7 lines
Kernel time (sys) reduction per workload with init_on_alloc=1:
Workload Before After Change Graph500 64C128T 30m 41.8s 15m 14.8s -50.3% Graph500 16C32T 15m 56.7s 9m 43.7s -39.0% Pagerank 32T 1m 58.5s 1m 12.8s -38.5% Pagerank 128T 2m 36.3s 1m 40.4s -35.7%
We do have some elaborate handling in clear_contig_highpages() to chunk it up (and to call cond_resched()). But that function can get called with much bigger ranges.
I'm not concerned about the cond_resched() -- we wouldn't do one here before -- but I'm wondering whether we could end up triggering a HW instruction that is uninterruptible and takes a rather long time.
But clear_contig_highpages() breaks it into 32MiB chunks, and only x86 supports it so far. So we won't exceed that with the maximum buddy order of 4MiB on x86.
Signature · 3 lines
Cheers, David
Quoted 42 lines · David Hildenbrand (Arm)
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 20 lines · Hrushikesh Salunke
When
init_on_allocis enabled,kernel_init_pages()clears every page one at a time viaclear_highpage_kasan_tagged(), which incurs per-page kmap_local_page()/kunmap_local() overhead and prevents the architecture clearing primitive from operating on contiguous ranges.Introduce
clear_highpages_kasan_tagged()inhighmem.h, a batch clearing helper that callsclear_pages()for the full contiguous range on !HIGHMEM systems, bypassing the per-page kmap overhead and allowing a single invocation of the arch clearing primitive across the entire allocation. The HIGHMEM path falls back to per-page clearing since those pages require kmap.Replace
kernel_init_pages()with direct calls to the new helper, as it becomes a trivial wrapper.Allocating 8192 x 2MB HugeTLB pages (16GB) with init_on_alloc=1:
Before: 0.445s
After: 0.166s (-62.7%, 2.68x faster)Quoted 7 lines
Kernel time (sys) reduction per workload with init_on_alloc=1:
Workload Before After Change Graph500 64C128T 30m 41.8s 15m 14.8s -50.3% Graph500 16C32T 15m 56.7s 9m 43.7s -39.0% Pagerank 32T 1m 58.5s 1m 12.8s -38.5% Pagerank 128T 2m 36.3s 1m 40.4s -35.7%We do have some elaborate handling in
clear_contig_highpages()to chunk it up (and to callcond_resched()). But that function can get called with much bigger ranges.I'm not concerned about the
cond_resched()-- we wouldn't do one here before -- but I'm wondering whether we could end up triggering a HW instruction that is uninterruptible and takes a rather long time.But
clear_contig_highpages()breaks it into 32MiB chunks, and only x86 supports it so far. So we won't exceed that with the maximum buddy order of 4MiB on x86.
Quoted 6 lines
--
Cheers,David
Right, on x86 the max buddy order keeps it well within safe limits.
Also, rep stosb/stosq on x86, currently used for clearing, is interruptible, the CPU can take interrupts between iterations and resume where it left off. So even for larger ranges it wouldn't be a single uninterruptible operation. Other architectures use a per-page loop for clearing, so the same applies there.
Thanks for the Ack!
Regards,
Hrushikesh
David Hildenbrand (Arm)
maintainerQuoted 49 lines · Hrushikesh
Quoted 47 lines · David Hildenbrand (Arm)
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 20 lines · Hrushikesh Salunke
When
init_on_allocis enabled,kernel_init_pages()clears every page one at a time viaclear_highpage_kasan_tagged(), which incurs per-page kmap_local_page()/kunmap_local() overhead and prevents the architecture clearing primitive from operating on contiguous ranges.Introduce
clear_highpages_kasan_tagged()inhighmem.h, a batch clearing helper that callsclear_pages()for the full contiguous range on !HIGHMEM systems, bypassing the per-page kmap overhead and allowing a single invocation of the arch clearing primitive across the entire allocation. The HIGHMEM path falls back to per-page clearing since those pages require kmap.Replace
kernel_init_pages()with direct calls to the new helper, as it becomes a trivial wrapper.Allocating 8192 x 2MB HugeTLB pages (16GB) with init_on_alloc=1:
Before: 0.445s
After: 0.166s (-62.7%, 2.68x faster)Quoted 7 lines
Kernel time (sys) reduction per workload with init_on_alloc=1:
Workload Before After Change Graph500 64C128T 30m 41.8s 15m 14.8s -50.3% Graph500 16C32T 15m 56.7s 9m 43.7s -39.0% Pagerank 32T 1m 58.5s 1m 12.8s -38.5% Pagerank 128T 2m 36.3s 1m 40.4s -35.7%We do have some elaborate handling in
clear_contig_highpages()to chunk it up (and to callcond_resched()). But that function can get called with much bigger ranges.I'm not concerned about the
cond_resched()-- we wouldn't do one here before -- but I'm wondering whether we could end up triggering a HW instruction that is uninterruptible and takes a rather long time.But
clear_contig_highpages()breaks it into 32MiB chunks, and only x86 supports it so far. So we won't exceed that with the maximum buddy order of 4MiB on x86.--
Cheers,
Quoted 7 lines
Quoted 1 line
David
Right, on x86 the max buddy order keeps it well within safe limits.
Also, rep stosb/stosq on x86, currently used for clearing, is interruptible, the CPU can take interrupts between iterations and resume where it left off.
Ah yes, indeed!
Signature · 3 lines
Cheers, David
Andrew Morton
maintainerQuoted 12 lines · On Wed, 22 Apr 2026 10:26:58 +0000 Hrushikesh Salunke <[email protected]> wrote:
When
init_on_allocis enabled,kernel_init_pages()clears every page one at a time viaclear_highpage_kasan_tagged(), which incurs per-page kmap_local_page()/kunmap_local() overhead and prevents the architecture clearing primitive from operating on contiguous ranges.Introduce
clear_highpages_kasan_tagged()inhighmem.h, a batch clearing helper that callsclear_pages()for the full contiguous range on !HIGHMEM systems, bypassing the per-page kmap overhead and allowing a single invocation of the arch clearing primitive across the entire allocation. The HIGHMEM path falls back to per-page clearing since those pages require kmap.
Quoted 7 lines
Replace
kernel_init_pages()with direct calls to the new helper, as it becomes a trivial wrapper.Allocating 8192 x 2MB HugeTLB pages (16GB) with init_on_alloc=1:
Before: 0.445s
After: 0.166s (-62.7%, 2.68x faster)
Nice.
Quoted 20 lines
Kernel time (sys) reduction per workload with init_on_alloc=1:
Workload Before After Change Graph500 64C128T 30m 41.8s 15m 14.8s -50.3% Graph500 16C32T 15m 56.7s 9m 43.7s -39.0% Pagerank 32T 1m 58.5s 1m 12.8s -38.5% Pagerank 128T 2m 36.3s 1m 40.4s -35.7%...
include/linux/highmem.h +4
-0@@ -345,6 +345,21 @@ static inline void clear_highpage_kasan_tagged(struct page *page)kunmap_local (kaddr); } +static inline void clear_highpages_kasan_tagged (struct page *page,int numpages) +{ +/* s390's use of memset() could override KASAN redzones. */ +kasan_disable_current ();
include/linux/highmem.h static inline void clear_highpage_kasan_tagged(struct page *page) · line 361
Quoted 10 lines
include/linux/highmem.h +10
-0+if (!IS_ENABLED (CONFIG_HIGHMEM )) { +clear_pages (kasan_reset_tag (page_address (page)), numpages); + }else { +int i; + +for (i =0 ; i < numpages; i++) +clear_highpage_kasan_tagged (page + i); + } +kasan_enable_current (); +}
Why was it globally published and inlined? Is there any expectation that this will be used outside of page_alloc.c?
Both of the callsites are themselves inlined. The patch adds 330 bytes to my arm allmodcnfig page_alloc.o - did we gain anything from that?
Quoted 59 lines · Andrew Morton
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 12 lines · On Wed, 22 Apr 2026 10:26:58 +0000 Hrushikesh Salunke <[email protected]> wrote:
When
init_on_allocis enabled,kernel_init_pages()clears every page one at a time viaclear_highpage_kasan_tagged(), which incurs per-page kmap_local_page()/kunmap_local() overhead and prevents the architecture clearing primitive from operating on contiguous ranges.Introduce
clear_highpages_kasan_tagged()inhighmem.h, a batch clearing helper that callsclear_pages()for the full contiguous range on !HIGHMEM systems, bypassing the per-page kmap overhead and allowing a single invocation of the arch clearing primitive across the entire allocation. The HIGHMEM path falls back to per-page clearing since those pages require kmap.Quoted 7 lines
Replace
kernel_init_pages()with direct calls to the new helper, as it becomes a trivial wrapper.Allocating 8192 x 2MB HugeTLB pages (16GB) with init_on_alloc=1:
Before: 0.445s
After: 0.166s (-62.7%, 2.68x faster)Nice.
Quoted 20 lines
Kernel time (sys) reduction per workload with init_on_alloc=1:
Workload Before After Change Graph500 64C128T 30m 41.8s 15m 14.8s -50.3% Graph500 16C32T 15m 56.7s 9m 43.7s -39.0% Pagerank 32T 1m 58.5s 1m 12.8s -38.5% Pagerank 128T 2m 36.3s 1m 40.4s -35.7%...
include/linux/highmem.h +4
-0@@ -345,6 +345,21 @@ static inline void clear_highpage_kasan_tagged(struct page *page)kunmap_local (kaddr); } +static inline void clear_highpages_kasan_tagged (struct page *page,int numpages) +{ +/* s390's use of memset() could override KASAN redzones. */ +kasan_disable_current ();include/linux/highmem.h static inline void clear_highpage_kasan_tagged(struct page *page) · line 361
Quoted 10 lines
include/linux/highmem.h +10
-0+if (!IS_ENABLED (CONFIG_HIGHMEM )) { +clear_pages (kasan_reset_tag (page_address (page)), numpages); + }else { +int i; + +for (i =0 ; i < numpages; i++) +clear_highpage_kasan_tagged (page + i); + } +kasan_enable_current (); +}Why was it globally published and inlined? Is there any expectation that this will be used outside of
page_alloc.c?
Quoted 3 lines
Both of the callsites are themselves inlined. The patch adds 330 bytes to my arm allmodcnfig page_alloc.o - did we gain anything from that?
Hi Andrew,
The idea was to keep it alongside clear_highpage_kasan_tagged() as its batch counterpart, but currently it is only used by page_alloc.c.
Your concern about the code size increase is valid. Would you prefer if I move it to page_alloc.c as a static function and drop the inline in v4? If an external user comes along later it can always be moved back to the header.
regards,
Hrushikesh
David Hildenbrand (Arm)
maintainerQuoted 61 lines · Hrushikesh
Quoted 59 lines · Andrew Morton
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 12 lines · On Wed, 22 Apr 2026 10:26:58 +0000 Hrushikesh Salunke <[email protected]> wrote:
When
init_on_allocis enabled,kernel_init_pages()clears every page one at a time viaclear_highpage_kasan_tagged(), which incurs per-page kmap_local_page()/kunmap_local() overhead and prevents the architecture clearing primitive from operating on contiguous ranges.Introduce
clear_highpages_kasan_tagged()inhighmem.h, a batch clearing helper that callsclear_pages()for the full contiguous range on !HIGHMEM systems, bypassing the per-page kmap overhead and allowing a single invocation of the arch clearing primitive across the entire allocation. The HIGHMEM path falls back to per-page clearing since those pages require kmap.Quoted 7 lines
Replace
kernel_init_pages()with direct calls to the new helper, as it becomes a trivial wrapper.Allocating 8192 x 2MB HugeTLB pages (16GB) with init_on_alloc=1:
Before: 0.445s
After: 0.166s (-62.7%, 2.68x faster)Nice.
Quoted 20 lines
Kernel time (sys) reduction per workload with init_on_alloc=1:
Workload Before After Change Graph500 64C128T 30m 41.8s 15m 14.8s -50.3% Graph500 16C32T 15m 56.7s 9m 43.7s -39.0% Pagerank 32T 1m 58.5s 1m 12.8s -38.5% Pagerank 128T 2m 36.3s 1m 40.4s -35.7%...
include/linux/highmem.h +4
-0@@ -345,6 +345,21 @@ static inline void clear_highpage_kasan_tagged(struct page *page)kunmap_local (kaddr); } +static inline void clear_highpages_kasan_tagged (struct page *page,int numpages) +{ +/* s390's use of memset() could override KASAN redzones. */ +kasan_disable_current ();include/linux/highmem.h static inline void clear_highpage_kasan_tagged(struct page *page) · line 361
Quoted 10 lines
include/linux/highmem.h +10
-0+if (!IS_ENABLED (CONFIG_HIGHMEM )) { +clear_pages (kasan_reset_tag (page_address (page)), numpages); + }else { +int i; + +for (i =0 ; i < numpages; i++) +clear_highpage_kasan_tagged (page + i); + } +kasan_enable_current (); +}Why was it globally published and inlined? Is there any expectation that this will be used outside of
page_alloc.c?
Quoted 7 lines
Quoted 3 lines
Both of the callsites are themselves inlined. The patch adds 330 bytes to my arm allmodcnfig page_alloc.o - did we gain anything from that?
Hi Andrew,
The idea was to keep it alongside
clear_highpage_kasan_tagged()as its batch counterpart, but currently it is only used bypage_alloc.c.
Right.
Looking at init_vmalloc_pages(), I wonder if it could also benefit from batching if we find that pages are actually contiguous.
That would require looking up multiple pages at once. vmalloc_to_pages() or sth like that. Surely, doing such an optimized page table walk could be beneficial by itself.
Quoted 5 lines
Your concern about the code size increase is valid. Would you prefer if I move it to
page_alloc.cas a static function and drop the inline in v4? If an external user comes along later it can always be moved back to the header.
What is exactly is responsible for the code increase? Two calls in clear_highpages_kasan_tagged()?
Surely the compiler would just inline kernel_init_pages() already?
So my best guess that the 330 bytes are just clear_pages() overhead or some code layout changes?
Signature · 3 lines
Cheers, David
Quoted 71 lines · David Hildenbrand (Arm)
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 60 lines · Hrushikesh
Quoted 59 lines · Andrew Morton
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 12 lines · On Wed, 22 Apr 2026 10:26:58 +0000 Hrushikesh Salunke <[email protected]> wrote:
When
init_on_allocis enabled,kernel_init_pages()clears every page one at a time viaclear_highpage_kasan_tagged(), which incurs per-page kmap_local_page()/kunmap_local() overhead and prevents the architecture clearing primitive from operating on contiguous ranges.Introduce
clear_highpages_kasan_tagged()inhighmem.h, a batch clearing helper that callsclear_pages()for the full contiguous range on !HIGHMEM systems, bypassing the per-page kmap overhead and allowing a single invocation of the arch clearing primitive across the entire allocation. The HIGHMEM path falls back to per-page clearing since those pages require kmap.Quoted 7 lines
Replace
kernel_init_pages()with direct calls to the new helper, as it becomes a trivial wrapper.Allocating 8192 x 2MB HugeTLB pages (16GB) with init_on_alloc=1:
Before: 0.445s
After: 0.166s (-62.7%, 2.68x faster)Nice.
Quoted 20 lines
Kernel time (sys) reduction per workload with init_on_alloc=1:
Workload Before After Change Graph500 64C128T 30m 41.8s 15m 14.8s -50.3% Graph500 16C32T 15m 56.7s 9m 43.7s -39.0% Pagerank 32T 1m 58.5s 1m 12.8s -38.5% Pagerank 128T 2m 36.3s 1m 40.4s -35.7%...
include/linux/highmem.h +4
-0@@ -345,6 +345,21 @@ static inline void clear_highpage_kasan_tagged(struct page *page)kunmap_local (kaddr); } +static inline void clear_highpages_kasan_tagged (struct page *page,int numpages) +{ +/* s390's use of memset() could override KASAN redzones. */ +kasan_disable_current ();include/linux/highmem.h static inline void clear_highpage_kasan_tagged(struct page *page) · line 361
Quoted 10 lines
include/linux/highmem.h +10
-0+if (!IS_ENABLED (CONFIG_HIGHMEM )) { +clear_pages (kasan_reset_tag (page_address (page)), numpages); + }else { +int i; + +for (i =0 ; i < numpages; i++) +clear_highpage_kasan_tagged (page + i); + } +kasan_enable_current (); +}Why was it globally published and inlined? Is there any expectation that this will be used outside of
page_alloc.c?Quoted 7 lines
Quoted 3 lines
Both of the callsites are themselves inlined. The patch adds 330 bytes to my arm allmodcnfig page_alloc.o - did we gain anything from that?
Hi Andrew,
The idea was to keep it alongside
clear_highpage_kasan_tagged()as its batch counterpart, but currently it is only used bypage_alloc.c.
Quoted 8 lines
Right.
Looking at
init_vmalloc_pages(), I wonder if it could also benefit from batching if we find that pages are actually contiguous.That would require looking up multiple pages at once.
vmalloc_to_pages()or sth like that. Surely, doing such an optimized page table walk could be beneficial by itself.
Interesting idea. For the general case where we only have struct page pointers, we'd need physical contiguity detection and a batched page table walk as you described. But looking at init_vmalloc_pages() specifically, it already has the vmalloc virtual address which is contiguous, so can we just do following and potentially skip the vmalloc_to_page() walk entirely:
clear_pages(kasan_reset_tag((void *)start), size >> PAGE_SHIFT);
What do you think? would this simpler approach work , or am I missing something?
Quoted 5 lines
Quoted 5 lines
Your concern about the code size increase is valid. Would you prefer if I move it to
page_alloc.cas a static function and drop the inline in v4? If an external user comes along later it can always be moved back to the header.
Quoted 7 lines
What is exactly is responsible for the code increase? Two calls in
clear_highpages_kasan_tagged()?Surely the compiler would just inline
kernel_init_pages()already?So my best guess that the 330 bytes are just
clear_pages()overhead or some code layout changes?
You're right, it's essentially the clear_pages() overhead being duplicated at each call site. The compiler was actually not inlining kernel_init_pages(), it was a standalone function. But it was inlining post_alloc_hook() and free_pages_prepare() into their callers, so with the patch each of those inlined copies now carries the full clear_highpages_kasan_tagged() code instead of a small call instruction.
I ran bloat-o-meter on arm allmodconfig and confirmed this. I also tested moving clear_highpages_kasan_tagged() into page_alloc.c as a static (non-inline) function, and the bloat disappears entirely. As currently there are no other users of this function so I will move it in page_alloc.c. I will make this change in v4.
Regards,
Hrushikesh.
David Hildenbrand (Arm)
maintainerQuoted 26 lines · Hrushikesh
Quoted 8 lines · David Hildenbrand (Arm)
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 4 lines · Hrushikesh
Hi Andrew,
The idea was to keep it alongside
clear_highpage_kasan_tagged()as its batch counterpart, but currently it is only used bypage_alloc.c.Quoted 8 lines
Right.
Looking at
init_vmalloc_pages(), I wonder if it could also benefit from batching if we find that pages are actually contiguous.That would require looking up multiple pages at once.
vmalloc_to_pages()or sth like that. Surely, doing such an optimized page table walk could be beneficial by itself.Interesting idea. For the general case where we only have struct page pointers, we'd need physical contiguity detection and a batched page table walk as you described. But looking at
init_vmalloc_pages()specifically, it already has the vmalloc virtual address which is contiguous, so can we just do following and potentially skip thevmalloc_to_page()walk entirely:
Quoted 4 lines
clear_pages(kasan_reset_tag((void *)start), size >> PAGE_SHIFT);What do you think? would this simpler approach work , or am I missing something?
Good question. :)
That way you'd be operating on the vmalloc address range, not on the direct map.
Is the vmalloc address range guaranteed to be writable at that point?
Signature · 3 lines
Cheers, David
Quoted 27 lines · David Hildenbrand (Arm)
Quoted 26 lines · Hrushikesh
Quoted 8 lines · David Hildenbrand (Arm)
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 4 lines · Hrushikesh
Hi Andrew,
The idea was to keep it alongside
clear_highpage_kasan_tagged()as its batch counterpart, but currently it is only used bypage_alloc.c.Quoted 8 lines
Right.
Looking at
init_vmalloc_pages(), I wonder if it could also benefit from batching if we find that pages are actually contiguous.That would require looking up multiple pages at once.
vmalloc_to_pages()or sth like that. Surely, doing such an optimized page table walk could be beneficial by itself.Interesting idea. For the general case where we only have struct page pointers, we'd need physical contiguity detection and a batched page table walk as you described. But looking at
init_vmalloc_pages()specifically, it already has the vmalloc virtual address which is contiguous, so can we just do following and potentially skip thevmalloc_to_page()walk entirely:
Quoted 8 lines
Quoted 4 lines
clear_pages(kasan_reset_tag((void *)start), size >> PAGE_SHIFT);What do you think? would this simpler approach work , or am I missing something?
Good question. :)
That way you'd be operating on the vmalloc address range, not on the direct map.
From my testing (including when using userspace VA) most of the speedup was from CPU prefetch. Which we should get when working with the vmalloc address range. Assuming it is writable and stable.
Quoted 1 line
Is the vmalloc address range guaranteed to be writable at that point?
What happens if we get preempted and migrated while clearing? Seems like the vmalloc lazy syncing should be able to handle that?
--
ankur
Quoted 29 lines · Ankur Arora
Quoted 27 lines · David Hildenbrand (Arm)
Quoted 26 lines · Hrushikesh
Quoted 8 lines · David Hildenbrand (Arm)
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Quoted 4 lines · Hrushikesh
Hi Andrew,
The idea was to keep it alongside
clear_highpage_kasan_tagged()as its batch counterpart, but currently it is only used bypage_alloc.c.Quoted 8 lines
Right.
Looking at
init_vmalloc_pages(), I wonder if it could also benefit from batching if we find that pages are actually contiguous.That would require looking up multiple pages at once.
vmalloc_to_pages()or sth like that. Surely, doing such an optimized page table walk could be beneficial by itself.Interesting idea. For the general case where we only have struct page pointers, we'd need physical contiguity detection and a batched page table walk as you described. But looking at
init_vmalloc_pages()specifically, it already has the vmalloc virtual address which is contiguous, so can we just do following and potentially skip thevmalloc_to_page()walk entirely:
clear_pages(kasan_reset_tag((void *)start), size >> PAGE_SHIFT);
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines
What do you think? would this simpler approach work , or am I missing something?
Good question. :)
That way you'd be operating on the vmalloc address range, not on the direct map.
From my testing (including when using userspace VA) most of the speedup was from CPU prefetch. Which we should get when working with the vmalloc address range. Assuming it is writable and stable.
Quoted 1 line
Is the vmalloc address range guaranteed to be writable at that point?
Hi David, Ankur
Yes. init_vmalloc_pages() is only reachable from __kasan_unpoison_vmalloc() in hw_tags.c, and the only caller that passes KASAN_VMALLOC_INIT is __vmalloc_node_range_noprof() in vmalloc.c. The pages are mapped by the time init_vmalloc_pages() runs, so the vmalloc range is writable at that point.
Quoted 2 lines
What happens if we get preempted and migrated while clearing? Seems like the vmalloc lazy syncing should be able to handle that?
Good point. I haven't looked into the vmalloc lazy syncing behavior in this context yet. I'll need to look into that to make sure there are no surprises with pre-emption and migration during clearing. Will follow up on this.
Regards,
Hrushikesh.