Skip to contentSkip to conversation

mm/hugetlb: optionally pre-zero hugetlb pages

PATCH In review Frank van der Linden 7 participants 17 messages Dec 2, 2024 – Dec 4, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm/hugetlb: optionally pre-zero hugetlb pages

Fresh hugetlb pages are zeroed out when they are faulted in, just like with all other page types. This can take up a good amount of time for larger page sizes (e.g. around 40 milliseconds for a 1G page on a recent AMD-based system).

This normally isn't a problem, since hugetlb pages are typically mapped by the application for a long time, and the initial delay when touching them isn't much of an issue.

However, there are some use cases where a large number of hugetlb pages are touched when an application (such as a VM backed by these pages) starts. For 256 1G pages and 40ms per page, this would take 10 seconds, a noticeable delay.

To make the above scenario faster, optionally pre-zero hugetlb pages. This is done by a kthread for each hstate/node combination that is eligible for pre-zeroing.

If there are pages to be pre-zeroed, the kthread starts at the tail of the per-node free list, and zeroes out pages, walking the list backwards until done. To make this logic work, new allocations are always taken from the head, and freed pages are added to the tail. After a page has been pre-zeroed, it is moved to the head of the list.

The kthread runs on init, and is also woken up when a page is put back on the free list.

A thread that allocates a hugetlb page can examine the prezero flag to see whether it's been zeroed out already. If this is not set, it must do so itself, as before.

For the rare corner where the kthread is busy zeroing out the only page available on the freelist, the caller must wait, which is done via a per-hstate wait queue.

Prezeroing can be switched on or off per hugetlb page size (hstate) by a sysfs value, /sys/kernel/mm/hugepages/hugepage-XXX/prezero_enabled If this is set to 0 (default), no pre-zeroing is done. If it's set to 1, all hugetlb pages of that size are pre-zeroed.

A new sysfs value, free_huge_pages_zero, tracks the number of free prezeroed pages per hstate.

Some test results: a simple test with a loop that maps new hugetlb pages and touches them, run on an AMD Milan system, touching 512 1G pages. It records the time in microseconds that a new touch (fault) takes, and the number of hits, misses and waits.

Without prezeroing With prezeroing

total time 20253322us total time 760us

average    39557us		average    1us
min        38199us		min        0us
max        41347us		max        10us

prezero hits 0 prezero hits 512
prezero misses 512 prezero misses 0
prezero waits 0 prezero waits 0

Repeating the loop 16 times, meaning the background zeroing is not yet complete when the loop restarts (so the worst case scenario for prezeroing with these parameters):

Without prezeroing With prezeroing

total time 321927564us total time 111980668us

average    39297us		average    13669us
min        38174us		min        0us
max        41935us		max        41876us

prezero hits 0 prezero hits 5299
prezero misses 8192 prezero misses 2877
prezero waits 0 prezero waits 16

So, even in this scenario, the average fault time is still quite a bit faster, and the maximum fault time is never bigger than without prezeroing.

---
 fs/hugetlbfs/inode.c    |   3 +-
 include/linux/hugetlb.h |  44 +++++
 mm/hugetlb.c            | 374 +++++++++++++++++++++++++++++++++++++++-
 mm/memory_hotplug.c     |   2 +
 4 files changed, 413 insertions(+), 10 deletions(-)
fs/hugetlbfs/inode.c +1 -2
index 5cf327337e22..7a6749959372 100644
@@ -893,8 +893,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,
 			error = PTR_ERR(folio);
 			goto out;
 		}
-		folio_zero_user(folio, ALIGN_DOWN(addr, hpage_size));
-		__folio_mark_uptodate(folio);
+		clear_hugetlb_folio(folio, addr);
 		error = hugetlb_add_to_page_cache(folio, mapping, index);
 		if (unlikely(error)) {
 			restore_reserve_on_error(h, &pseudo_vma, addr, folio);
include/linux/hugetlb.h +44 -0

86 lines not on this page · raw message

mm/hugetlb.c +366 -8

509 lines not on this page · raw message

mm/memory_hotplug.c +2 -0
index 621ae1015106..8398e4d0130d 100644
@@ -1217,6 +1217,7 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 
 	kswapd_run(nid);
 	kcompactd_run(nid);
+	khzerod_run(nid);
 
 	writeback_set_ratelimit();
 
@@ -2092,6 +2093,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 	if (arg.status_change_nid >= 0) {
 		kcompactd_stop(node);
 		kswapd_stop(node);
+		khzerod_stop(node);
 	}
 
 	writeback_set_ratelimit();
Signature · 1 line
2.47.0.338.g60cca15819-goog
#1 lore raw
Quoted 5 lines · Frank van der Linden

Fresh hugetlb pages are zeroed out when they are faulted in, just like with all other page types. This can take up a good amount of time for larger page sizes (e.g. around 40 milliseconds for a 1G page on a recent AMD-based system).

Quoted 8 lines

This normally isn't a problem, since hugetlb pages are typically mapped by the application for a long time, and the initial delay when touching them isn't much of an issue.

However, there are some use cases where a large number of hugetlb pages are touched when an application (such as a VM backed by these pages) starts. For 256 1G pages and 40ms per page, this would take 10 seconds, a noticeable delay.

The current huge page zeroing code is not that great to begin with.

There was a patchset posted some time ago to remedy at least some of it:
https://lore.kernel.org/all/[email protected]/

but it apparently fell through the cracks.

Any games with "background zeroing" are notoriously crappy and I would argue one should exhaust other avenues before going there -- at the end of the day the cost of zeroing will have to get paid.

To that end I would suggest picking up the patchset and experimenting with more variants of the zeroing code (for example for 1G it may be it is faster to employ SIMD usage in the routine).

If this is really such a problem I wonder if this could start as a series of 2MB pages instead faulted as needed, eventually promoted to 1G after passing some threshold?

#2 lore raw
Quoted 16 lines · Mateusz Guzik
Quoted 5 lines · Frank van der Linden

Fresh hugetlb pages are zeroed out when they are faulted in, just like with all other page types. This can take up a good amount of time for larger page sizes (e.g. around 40 milliseconds for a 1G page on a recent AMD-based system).

Quoted 8 lines

This normally isn't a problem, since hugetlb pages are typically mapped by the application for a long time, and the initial delay when touching them isn't much of an issue.

However, there are some use cases where a large number of hugetlb pages are touched when an application (such as a VM backed by these pages) starts. For 256 1G pages and 40ms per page, this would take 10 seconds, a noticeable delay.

Quoted 6 lines

The current huge page zeroing code is not that great to begin with.

There was a patchset posted some time ago to remedy at least some of it:
https://lore.kernel.org/all/[email protected]/

but it apparently fell through the cracks.

Hi Mateusz, thanks for your reply.

I am aware of that patch set, yes. The discussion around it evolved in to one about kernel preemption and the evilness of cond_resched().

You can certainly improve the time it takes to zero out a 1G page by optimizing the code that does it. See also, for example,
https://lore.kernel.org/all/[email protected]/

However, while, say, a 50% improvement in zero out time, at the max, is nice, this still leaves the faulting process spending considerable time doing it. Like you say, that's cost that needs to be paid - but it would be good to avoid paying it inline. This patch avoids doing that altogether, leading to a basically 100% improvement under reasonably good circumstances.

Quoted 4 lines

Any games with "background zeroing" are notoriously crappy and I would argue one should exhaust other avenues before going there -- at the end of the day the cost of zeroing will have to get paid.

I understand that the concept of background prezeroing has been, and will be, met with some resistance. But, do you have any specific concerns with the patch I posted? It's pretty well isolated from the rest of the code, and optional.

Quoted 4 lines

To that end I would suggest picking up the patchset and experimenting with more variants of the zeroing code (for example for 1G it may be it is faster to employ SIMD usage in the routine).

See above - happy to pick up older patch(es) as a separate effort, but they won't fully solve the issue for the scenario I'm describing.

Quoted 4 lines

If this is really such a problem I wonder if this could start as a series of 2MB pages instead faulted as needed, eventually promoted to 1G after passing some threshold?

This idea sounds similar to HGM (high granularity mapping), an idea which was originally posted for the purpose of live migration of VMs (but never made it in). It's not trivial, and seems like overkill. Again, my patch is non-invasive and optional, so I think it's better in that regard.

- Frank

#3 lore raw
Quoted 9 lines · Frank van der Linden
Quoted 3 lines · Mateusz Guzik

Any games with "background zeroing" are notoriously crappy and I would argue one should exhaust other avenues before going there -- at the end of the day the cost of zeroing will have to get paid.

I understand that the concept of background prezeroing has been, and will be, met with some resistance. But, do you have any specific concerns with the patch I posted? It's pretty well isolated from the rest of the code, and optional.

The biggest concern I have is that the overhead is payed by everybody on the system - it is considered to be a system overhead regardless only part of the workload benefits from hugetlb pages. In other words the workload using those pages is not accounted for the use completely.

If the startup latency is a real problem is there a way to workaround that in the userspace by preallocating hugetlb pages ahead of time before those VMs are launched and hand over already pre-allocated pages?

Signature · 2 lines
Michal Hocko
SUSE Labs
#4 lore raw
Quoted 11 lines · Michal Hocko
Quoted 9 lines · Frank van der Linden
Quoted 3 lines · Mateusz Guzik

Any games with "background zeroing" are notoriously crappy and I would argue one should exhaust other avenues before going there -- at the end of the day the cost of zeroing will have to get paid.

I understand that the concept of background prezeroing has been, and will be, met with some resistance. But, do you have any specific concerns with the patch I posted? It's pretty well isolated from the rest of the code, and optional.

Quoted 8 lines

The biggest concern I have is that the overhead is payed by everybody on the system - it is considered to be a system overhead regardless only part of the workload benefits from hugetlb pages. In other words the workload using those pages is not accounted for the use completely.

If the startup latency is a real problem is there a way to workaround that in the userspace by preallocating hugetlb pages ahead of time before those VMs are launched and hand over already pre-allocated pages?

In QEMU we support parallel preallocation, and even NUMA-aware preallocation, to reduce VM startup times with hugetlb.

We discussed improvements as referred to by Mateusz to further speed up preallocation, but for now parallel+NUMA-aware preallocation seems to do the trick.

Signature · 3 lines
Cheers,

David / dhildenb
#5 lore raw
In reply to Michal Hocko
Quoted 11 lines · Michal Hocko
Quoted 9 lines · Frank van der Linden
Quoted 3 lines · Mateusz Guzik

Any games with "background zeroing" are notoriously crappy and I would argue one should exhaust other avenues before going there -- at the end of the day the cost of zeroing will have to get paid.

I understand that the concept of background prezeroing has been, and will be, met with some resistance. But, do you have any specific concerns with the patch I posted? It's pretty well isolated from the rest of the code, and optional.

Quoted 8 lines

The biggest concern I have is that the overhead is payed by everybody on the system - it is considered to be a system overhead regardless only part of the workload benefits from hugetlb pages. In other words the workload using those pages is not accounted for the use completely.

If the startup latency is a real problem is there a way to workaround that in the userspace by preallocating hugetlb pages ahead of time before those VMs are launched and hand over already pre-allocated pages?

It should be relatively simple to actually do this. Me and Mike had experimented ourselves a couple years back but we never had the chance to send it over. IIRC if we:

- add the PageZeroed tracking bit when a page is zeroed
- clear it in the write (fixup/non-fixup) fault-path

[somewhat similar to this series I suspect]

Then what's left is to change the lookup of free hugetlb pages (dequeue_hugetlb_folio_node_exact() I think) to search first for non-zeroed pages. Provided we don't track its 'cleared' state, there's no UAPI change in behaviour. A daemon can just allocate/mmap+touch/etc them with read-only and free them back 'as zeroed' to implement a userspace scrubber. And in principle existing apps should see no difference. The amount of changes is consequently significantly smaller (or it looked as such in a quick PoC years back).

Something extra on the top would perhaps be the ability so select a lookup heuristic such that we can pick the search method of non-zero-first/only-nonzero/zeroed pages behind ioctl() (or a better generic UAPI) to allow a scrubber to easily coexist with hugepage user (e.g. a VMM, etc) without too much of a dance.

Joao

#6 lore raw
Quoted 23 lines · Joao Martins
Quoted 3 lines · Michal Hocko

If the startup latency is a real problem is there a way to workaround that in the userspace by preallocating hugetlb pages ahead of time before those VMs are launched and hand over already pre-allocated pages?

It should be relatively simple to actually do this. Me and Mike had experimented ourselves a couple years back but we never had the chance to send it over. IIRC if we:

- add the PageZeroed tracking bit when a page is zeroed
- clear it in the write (fixup/non-fixup) fault-path

[somewhat similar to this series I suspect]

Then what's left is to change the lookup of free hugetlb pages (dequeue_hugetlb_folio_node_exact() I think) to search first for non-zeroed pages. Provided we don't track its 'cleared' state, there's no UAPI change in behaviour. A daemon can just allocate/mmap+touch/etc them with read-only and free them back 'as zeroed' to implement a userspace scrubber. And in principle existing apps should see no difference. The amount of changes is consequently significantly smaller (or it looked as such in a quick PoC years back).

Quoted 6 lines

Something extra on the top would perhaps be the ability so select a lookup heuristic such that we can pick the search method of non-zero-first/only-nonzero/zeroed pages behind ioctl() (or a better generic UAPI) to allow a scrubber to easily coexist with hugepage user (e.g. a VMM, etc) without too much of a dance.

Ye after the qemu prefaulting got pointed out I started thinking about a userlevel daemon which would do the work proposed here.

Except I got stuck at a good way to do it. The mmap + load from the area + munmap triple does work but also entails more overhead than necessary, but I only have some handwaving how to not do it. :)

Suppose a daemon of the sort exists and there is a machine with 4 or more NUMA domains to deal with. Further suppose it spawns at least one thread per such domain and tasksets them accordingly.

Then perhaps an ioctl somewhere on hugetlbfs(?) could take a parameter indicating how many pages to zero out (or even just accept one page). This would avoid crap on munmap.

This would still need majority of the patch, but all the zeroing policy would be taken out. Key point being that whatever specific behavior one sees fit, they can implement it in userspace, preventing future kernel patches to add more tweaks.

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#7 lore raw
Quoted 43 lines · Mateusz Guzik
Quoted 23 lines · Joao Martins
Quoted 3 lines · Michal Hocko

If the startup latency is a real problem is there a way to workaround that in the userspace by preallocating hugetlb pages ahead of time before those VMs are launched and hand over already pre-allocated pages?

It should be relatively simple to actually do this. Me and Mike had experimented ourselves a couple years back but we never had the chance to send it over. IIRC if we:

- add the PageZeroed tracking bit when a page is zeroed
- clear it in the write (fixup/non-fixup) fault-path

[somewhat similar to this series I suspect]

Then what's left is to change the lookup of free hugetlb pages (dequeue_hugetlb_folio_node_exact() I think) to search first for non-zeroed pages. Provided we don't track its 'cleared' state, there's no UAPI change in behaviour. A daemon can just allocate/mmap+touch/etc them with read-only and free them back 'as zeroed' to implement a userspace scrubber. And in principle existing apps should see no difference. The amount of changes is consequently significantly smaller (or it looked as such in a quick PoC years back).

Quoted 6 lines

Something extra on the top would perhaps be the ability so select a lookup heuristic such that we can pick the search method of non-zero-first/only-nonzero/zeroed pages behind ioctl() (or a better generic UAPI) to allow a scrubber to easily coexist with hugepage user (e.g. a VMM, etc) without too much of a dance.

Ye after the qemu prefaulting got pointed out I started thinking about a userlevel daemon which would do the work proposed here.

Except I got stuck at a good way to do it. The mmap + load from the area + munmap triple does work but also entails more overhead than necessary, but I only have some handwaving how to not do it. :)

Suppose a daemon of the sort exists and there is a machine with 4 or more NUMA domains to deal with. Further suppose it spawns at least one thread per such domain and tasksets them accordingly.

Quoted 8 lines

Then perhaps an ioctl somewhere on hugetlbfs(?) could take a parameter indicating how many pages to zero out (or even just accept one page). This would avoid crap on munmap.

This would still need majority of the patch, but all the zeroing policy would be taken out. Key point being that whatever specific behavior one sees fit, they can implement it in userspace, preventing future kernel patches to add more tweaks.

How about this for a rough sketch (which I have 0 intention of implementing myself):

/dev/hugepagectl or whatever is created with a bunch of ioctls, notably:
- something to query hugepage stats
- an event generated for epoll if count in any domain goes below a threshold
- something to zero a page of given size from the free list

Perhaps make it so that fds require an upfront ioctl to set a numa domain of interest before poll works -- for example if there is one thread per domain, each of them sleeps on its own relevant fd. Or maybe someone still wants the main thread to get the full view so they poll on all of them.

then a google internal tool can react however it sees fit without waking up in a periodic fashion. (replace google with any other company which may want to mess with this).

optional:
- allocating and zeroing (but not mmaping!) a page

then a party which shares the file descriptor could obtain it by passing the fd to mmap. munmap would just free it as it does now. this would allow qemu et al to avoid the mmap/munmap dance just to zero, but I don't know how useful it is for them

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#8 lore raw
In reply to Mateusz Guzik
Quoted 31 lines · Mateusz Guzik
Quoted 23 lines · Joao Martins
Quoted 3 lines · Michal Hocko

If the startup latency is a real problem is there a way to workaround that in the userspace by preallocating hugetlb pages ahead of time before those VMs are launched and hand over already pre-allocated pages?

It should be relatively simple to actually do this. Me and Mike had experimented ourselves a couple years back but we never had the chance to send it over. IIRC if we:

- add the PageZeroed tracking bit when a page is zeroed
- clear it in the write (fixup/non-fixup) fault-path

[somewhat similar to this series I suspect]

Then what's left is to change the lookup of free hugetlb pages (dequeue_hugetlb_folio_node_exact() I think) to search first for non-zeroed pages. Provided we don't track its 'cleared' state, there's no UAPI change in behaviour. A daemon can just allocate/mmap+touch/etc them with read-only and free them back 'as zeroed' to implement a userspace scrubber. And in principle existing apps should see no difference. The amount of changes is consequently significantly smaller (or it looked as such in a quick PoC years back).

Quoted 6 lines

Something extra on the top would perhaps be the ability so select a lookup heuristic such that we can pick the search method of non-zero-first/only-nonzero/zeroed pages behind ioctl() (or a better generic UAPI) to allow a scrubber to easily coexist with hugepage user (e.g. a VMM, etc) without too much of a dance.

Quoted 7 lines

Ye after the qemu prefaulting got pointed out I started thinking about a userlevel daemon which would do the work proposed here.

Except I got stuck at a good way to do it. The mmap + load from the area + munmap triple does work but also entails more overhead than necessary, but I only have some handwaving how to not do it. :)

What I was trying to suggest above is that it would be no different that how you use hugetlb. I am not enterily sure I follow the triple work part on unmap.

Quoted 9 lines

Suppose a daemon of the sort exists and there is a machine with 4 or more NUMA domains to deal with. Further suppose it spawns at least one thread per such domain and tasksets them accordingly.

Then perhaps an ioctl somewhere on hugetlbfs(?) could take a parameter indicating how many pages to zero out (or even just accept one page). This would avoid crap on munmap.

This would still need majority of the patch, but all the zeroing> policy would

be taken out. Key point being that whatever specific

Quoted 2 lines

behavior one sees fit, they can implement it in userspace, preventing future kernel patches to add more tweaks.

Kernel should still ensure it tracks if it's cleared or not -- so what I said above was just letting the allocation zero out the page or not (if it's not zeroed already) and just tweak the dirtyness of pages it picks before installing PTEs. A scrubber would pick only dirty pages (and maybe fail if there aren't any), and a VMM would pick clean pages (taking advantage of the scrubber work). An explicit zero sounds a somewhat limiting ... but hmm

What throws all this away (in primary MM) is the prefaulting with write as we would clear the PageCleared bit all the time (I think that's what you mean 'crap on unmap'?).

But there could be hope for systems with a secondary pagetables (with paging), where the secondary faulting is the one in control of the cleared status. That is because reads inside the VM ultimately trigger secondary-VM read-faults and get fixed up later with write on writes.

Well, at least it would work given we don't prefault secondary page tables yet...

#9 lore raw
In reply to Joao Martins
Quoted 31 lines · Joao Martins
Quoted 11 lines · Michal Hocko
Quoted 9 lines · Frank van der Linden
Quoted 3 lines · Mateusz Guzik

Any games with "background zeroing" are notoriously crappy and I would argue one should exhaust other avenues before going there -- at the end of the day the cost of zeroing will have to get paid.

I understand that the concept of background prezeroing has been, and will be, met with some resistance. But, do you have any specific concerns with the patch I posted? It's pretty well isolated from the rest of the code, and optional.

Quoted 8 lines

The biggest concern I have is that the overhead is payed by everybody on the system - it is considered to be a system overhead regardless only part of the workload benefits from hugetlb pages. In other words the workload using those pages is not accounted for the use completely.

If the startup latency is a real problem is there a way to workaround that in the userspace by preallocating hugetlb pages ahead of time before those VMs are launched and hand over already pre-allocated pages?

It should be relatively simple to actually do this. Me and Mike had experimented ourselves a couple years back but we never had the chance to send it over. IIRC if we:

- add the PageZeroed tracking bit when a page is zeroed
- clear it in the write (fixup/non-fixup) fault-path

[somewhat similar to this series I suspect]

Quoted 7 lines

Then what's left is to change the lookup of free hugetlb pages (dequeue_hugetlb_folio_node_exact() I think) to search first for non-zeroed pages. Provided we don't track its 'cleared' state, there's no UAPI change in behaviour. A daemon can just allocate/mmap+touch/etc them with read-only and free them back 'as zeroed' to implement a userspace scrubber. And in principle existing apps should see no difference. The amount of changes is consequently significantly smaller (or it looked as such in a quick PoC years back).

This would work, and is easy to do, but:
* You now have a userspace daemon that depends on kernel-internal behavior.
* It has no way to track how much work is left to do or what needs
to be done (unless it is part of an application that is the only user of hugetlbfs on the system).

Quoted 6 lines

Something extra on the top would perhaps be the ability so select a lookup heuristic such that we can pick the search method of non-zero-first/only-nonzero/zeroed pages behind ioctl() (or a better generic UAPI) to allow a scrubber to easily coexist with hugepage user (e.g. a VMM, etc) without too much of a dance.

Again, that would probably work, but if you take a step back: you now have a kernel behavior that can be guided in certain directions, but no guarantees and no stats to see if things are working out. And an explicit allocation method option (basically: take from the head or the tail of the freelist). The picture is getting murkier. At least with the patch I sent you have a clearly defined, optional, behavior that can be switched on or off, and stats to see if it's working.

I do understand the argument against having pre-zeroing not being accounted to the current thread. I would counter that benefiting from work by kernel threads is not unheard of in the kernel today already. Also, the other proposals so far also have another thread doing the zeroing - it just is explicitly started by userspace. So, the cost is still not paid by the user of the pages. You just end up with explicitly controlling who does pay the cost. Which I suppose is better, but it's still not trivial to get it completely right (you perhaps could do it at the container level with some trickery).

What we have done so far is to bind the khzerod threads introduced in this patch to CPUs in such a way that it doesn't interfere with the rest of the system. Which you would also have to do with any userspace solution.

Again, this is optional - if you are a system manager who prefers to have the resources used by zeroing hugetlb pages to be explicitly accounted to the actual user, you can not enable this behavior (it's off by default).

I guess I can summarize my thoughts like this: while I understand the argument against doing this outside of the context of the actual user of the pages, this is 1) optional, and 2) so far the other solutions introduce interfaces that I don't think are that great, or would require maintaining a hugetlb 'shadow pool' in userspace through hugetlbfs files.

- Frank

#10 lore raw
Quoted 82 lines · Frank van der Linden
Quoted 31 lines · Joao Martins
Quoted 11 lines · Michal Hocko
Quoted 9 lines · Frank van der Linden
Quoted 3 lines · Mateusz Guzik

Any games with "background zeroing" are notoriously crappy and I would argue one should exhaust other avenues before going there -- at the end of the day the cost of zeroing will have to get paid.

I understand that the concept of background prezeroing has been, and will be, met with some resistance. But, do you have any specific concerns with the patch I posted? It's pretty well isolated from the rest of the code, and optional.

Quoted 8 lines

The biggest concern I have is that the overhead is payed by everybody on the system - it is considered to be a system overhead regardless only part of the workload benefits from hugetlb pages. In other words the workload using those pages is not accounted for the use completely.

If the startup latency is a real problem is there a way to workaround that in the userspace by preallocating hugetlb pages ahead of time before those VMs are launched and hand over already pre-allocated pages?

It should be relatively simple to actually do this. Me and Mike had experimented ourselves a couple years back but we never had the chance to send it over. IIRC if we:

- add the PageZeroed tracking bit when a page is zeroed
- clear it in the write (fixup/non-fixup) fault-path

[somewhat similar to this series I suspect]

Quoted 7 lines

Then what's left is to change the lookup of free hugetlb pages (dequeue_hugetlb_folio_node_exact() I think) to search first for non-zeroed pages. Provided we don't track its 'cleared' state, there's no UAPI change in behaviour. A daemon can just allocate/mmap+touch/etc them with read-only and free them back 'as zeroed' to implement a userspace scrubber. And in principle existing apps should see no difference. The amount of changes is consequently significantly smaller (or it looked as such in a quick PoC years back).

This would work, and is easy to do, but:
* You now have a userspace daemon that depends on kernel-internal behavior.
* It has no way to track how much work is left to do or what needs
to be done (unless it is part of an application that is the only user of hugetlbfs on the system).

Quoted 6 lines

Something extra on the top would perhaps be the ability so select a lookup heuristic such that we can pick the search method of non-zero-first/only-nonzero/zeroed pages behind ioctl() (or a better generic UAPI) to allow a scrubber to easily coexist with hugepage user (e.g. a VMM, etc) without too much of a dance.

Again, that would probably work, but if you take a step back: you now have a kernel behavior that can be guided in certain directions, but no guarantees and no stats to see if things are working out. And an explicit allocation method option (basically: take from the head or the tail of the freelist). The picture is getting murkier. At least with the patch I sent you have a clearly defined, optional, behavior that can be switched on or off, and stats to see if it's working.

I do understand the argument against having pre-zeroing not being accounted to the current thread. I would counter that benefiting from work by kernel threads is not unheard of in the kernel today already. Also, the other proposals so far also have another thread doing the zeroing - it just is explicitly started by userspace. So, the cost is still not paid by the user of the pages. You just end up with explicitly controlling who does pay the cost. Which I suppose is better, but it's still not trivial to get it completely right (you perhaps could do it at the container level with some trickery).

What we have done so far is to bind the khzerod threads introduced in this patch to CPUs in such a way that it doesn't interfere with the rest of the system. Which you would also have to do with any userspace solution.

Again, this is optional - if you are a system manager who prefers to have the resources used by zeroing hugetlb pages to be explicitly accounted to the actual user, you can not enable this behavior (it's off by default).

Quoted 6 lines

I guess I can summarize my thoughts like this: while I understand the argument against doing this outside of the context of the actual user of the pages, this is 1) optional, and 2) so far the other solutions introduce interfaces that I don't think are that great, or would require maintaining a hugetlb 'shadow pool' in userspace through hugetlbfs files.

One more thing: any userspace solution has one basic problem: the hugetlb pages will be unavailable while they are being zeroed out, as the userspace process or thread will have to map+touch them, taking them off the freelist. So now another process that needs the hugetlb pages, expecting them to be there based on initial configuration and what it's done so far, may end up getting unexpected -ENOMEM because one or more pages have been temporarily allocated by userspace prezero threads.

My patch doesn't have that issue - the pages stay on the freelist, the total number of available pages does not change. In the rare case that a freshly allocated page is being prezeroed, you'll just have to wait until it's done (taking up no more time than doing it yourself).

Now, you can implement something like this in userspace (if I get -ENOMEM, check with the prezero thread or process), but it's not great.

- Frank

#11 lore raw
In reply to Mateusz Guzik
Quoted 15 lines · Mateusz Guzik
Quoted 5 lines · Frank van der Linden

Fresh hugetlb pages are zeroed out when they are faulted in, just like with all other page types. This can take up a good amount of time for larger page sizes (e.g. around 40 milliseconds for a 1G page on a recent AMD-based system).

Quoted 8 lines

This normally isn't a problem, since hugetlb pages are typically mapped by the application for a long time, and the initial delay when touching them isn't much of an issue.

However, there are some use cases where a large number of hugetlb pages are touched when an application (such as a VM backed by these pages) starts. For 256 1G pages and 40ms per page, this would take 10 seconds, a noticeable delay.

Quoted 7 lines

The current huge page zeroing code is not that great to begin with.

There was a patchset posted some time ago to remedy at least some of it:
https://lore.kernel.org/all/[email protected]/

but it apparently fell through the cracks.

It didn't fell through the cracks for sure

Just had a detour into preempt=auto before resuming the main work. But that seems to be done in the last merge window with the lazy preempt stuff. I think Ankur was planning on following that series above soon-ish.

Adding him here, such that he keeps me honest :)

#12 lore raw
Quoted 24 lines · Joao Martins
Quoted 15 lines · Mateusz Guzik
Quoted 5 lines · Frank van der Linden

Fresh hugetlb pages are zeroed out when they are faulted in, just like with all other page types. This can take up a good amount of time for larger page sizes (e.g. around 40 milliseconds for a 1G page on a recent AMD-based system).

Quoted 8 lines

This normally isn't a problem, since hugetlb pages are typically mapped by the application for a long time, and the initial delay when touching them isn't much of an issue.

However, there are some use cases where a large number of hugetlb pages are touched when an application (such as a VM backed by these pages) starts. For 256 1G pages and 40ms per page, this would take 10 seconds, a noticeable delay.

Quoted 7 lines

The current huge page zeroing code is not that great to begin with.

There was a patchset posted some time ago to remedy at least some of it:
https://lore.kernel.org/all/[email protected]/

but it apparently fell through the cracks.

Quoted 7 lines

It didn't fell through the cracks for sure

Just had a detour into preempt=auto before resuming the main work. But that seems to be done in the last merge window with the lazy preempt stuff. I think Ankur was planning on following that series above soon-ish.

Adding him here, such that he keeps me honest :)

Thanks for Ccing Joao.

--
ankur

#13 lore raw
In reply to Mateusz Guzik
Quoted 10 lines · Mateusz Guzik
Quoted 9 lines · Frank van der Linden

Fresh hugetlb pages are zeroed out when they are faulted in, just like with all other page types. This can take up a good amount of time for larger page sizes (e.g. around 40 milliseconds for a 1G page on a recent AMD-based system).

This normally isn't a problem, since hugetlb pages are typically mapped by the application for a long time, and the initial delay when touching them isn't much of an issue.

Quoted 6 lines
Quoted 4 lines

However, there are some use cases where a large number of hugetlb pages are touched when an application (such as a VM backed by these pages) starts. For 256 1G pages and 40ms per page, this would take 10 seconds, a noticeable delay.

The current huge page zeroing code is not that great to begin with.

Yeah definitely suboptimal. The current huge page zeroing code is both slow and it trashes the cache while zeroing.

Quoted 4 lines

There was a patchset posted some time ago to remedy at least some of it:
https://lore.kernel.org/all/[email protected]/

but it apparently fell through the cracks.

As Joao mentioned that got side tracked due to the preempt-lazy stuff. Now that lazy is in, I plan to follow up on the zeroing work.

Quoted 3 lines

Any games with "background zeroing" are notoriously crappy and I would argue one should exhaust other avenues before going there -- at the end of the day the cost of zeroing will have to get paid.

Yeah and the background zeroing has dual cost: the cost in CPU time plus the indirect cost to other processes due to the trashing of L3 etc.

Ankur

#14 lore raw
Quoted 32 lines · Ankur Arora
Quoted 10 lines · Mateusz Guzik
Quoted 9 lines · Frank van der Linden

Fresh hugetlb pages are zeroed out when they are faulted in, just like with all other page types. This can take up a good amount of time for larger page sizes (e.g. around 40 milliseconds for a 1G page on a recent AMD-based system).

This normally isn't a problem, since hugetlb pages are typically mapped by the application for a long time, and the initial delay when touching them isn't much of an issue.

Quoted 6 lines
Quoted 4 lines

However, there are some use cases where a large number of hugetlb pages are touched when an application (such as a VM backed by these pages) starts. For 256 1G pages and 40ms per page, this would take 10 seconds, a noticeable delay.

The current huge page zeroing code is not that great to begin with.

Yeah definitely suboptimal. The current huge page zeroing code is both slow and it trashes the cache while zeroing.

Quoted 4 lines

There was a patchset posted some time ago to remedy at least some of it:
https://lore.kernel.org/all/[email protected]/

but it apparently fell through the cracks.

As Joao mentioned that got side tracked due to the preempt-lazy stuff. Now that lazy is in, I plan to follow up on the zeroing work.

Quoted 6 lines
Quoted 3 lines

Any games with "background zeroing" are notoriously crappy and I would argue one should exhaust other avenues before going there -- at the end of the day the cost of zeroing will have to get paid.

Yeah and the background zeroing has dual cost: the cost in CPU time plus the indirect cost to other processes due to the trashing of L3 etc.

I'm not sure what you mean here - any caching side effects of zeroing happen regardless of who does it, right? It doesn't matter if it's a kthread or the calling thread.

If you're concerned about the caching side effects in general, using non-temporal instructions helps (e.g. movnti on x86). See the link I mentioned for a patch that was sent years ago (
https://lore.kernel.org/all/[email protected]/
). Using movnti on x86 definitely helps performance (up to 50% in my experiments). Which is great, but it still leaves considerable delay for the use case I mentioned.

- Frank

#15 lore raw
Quoted 37 lines · Frank van der Linden
Quoted 36 lines · Ankur Arora
Quoted 10 lines · Mateusz Guzik
Quoted 9 lines · Frank van der Linden

Fresh hugetlb pages are zeroed out when they are faulted in, just like with all other page types. This can take up a good amount of time for larger page sizes (e.g. around 40 milliseconds for a 1G page on a recent AMD-based system).

This normally isn't a problem, since hugetlb pages are typically mapped by the application for a long time, and the initial delay when touching them isn't much of an issue.

Quoted 6 lines
Quoted 4 lines

However, there are some use cases where a large number of hugetlb pages are touched when an application (such as a VM backed by these pages) starts. For 256 1G pages and 40ms per page, this would take 10 seconds, a noticeable delay.

The current huge page zeroing code is not that great to begin with.

Yeah definitely suboptimal. The current huge page zeroing code is both slow and it trashes the cache while zeroing.

Quoted 4 lines

There was a patchset posted some time ago to remedy at least some of it:
https://lore.kernel.org/all/[email protected]/

but it apparently fell through the cracks.

As Joao mentioned that got side tracked due to the preempt-lazy stuff. Now that lazy is in, I plan to follow up on the zeroing work.

Quoted 3 lines

Any games with "background zeroing" are notoriously crappy and I would argue one should exhaust other avenues before going there -- at the end of the day the cost of zeroing will have to get paid.

Quoted 5 lines
Quoted 2 lines

Yeah and the background zeroing has dual cost: the cost in CPU time plus the indirect cost to other processes due to the trashing of L3 etc.

I'm not sure what you mean here - any caching side effects of zeroing happen regardless of who does it, right?

Sure.

Quoted 2 lines

It doesn't matter if it's a
kthread or the calling thread.

As other people point out it's more a matter of accruing it to the right context. The noise will always spill over but userspace can use cpu cgroups etc to to limit how far these effects are seen.

Additionally, this kthread will be doing bulk zeroing while a user thread would zero as needed. Though I guess for the VM prefaulting case it's likely similar.

Quoted 7 lines

If you're concerned about the caching side effects in general, using non-temporal instructions helps (e.g. movnti on x86). See the link I mentioned for a patch that was sent years ago (
https://lore.kernel.org/all/[email protected]/
). Using movnti on x86 definitely helps performance (up to 50% in my experiments). Which is great, but it still leaves considerable delay for the use case I mentioned.

In my testing at least on AMD you can get a lot more than 50% improvement.

See for instance the CLZERO (or the REP STOS) numbers here: https://lore.kernel.org/lkml/[email protected]/

--
ankur

#16 lore raw
In reply to Frank van der Linden
kernel test robot noticed the following build warnings:

Hi Frank,

kernel test robot noticed the following build warnings:

[auto build test WARNING on v6.12]
[cannot apply to akpm-mm/mm-everything brauner-vfs/vfs.all linus/master v6.13-rc1 next-20241128] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Frank-van-der-Linden/mm-hugetlb-optionally-pre-zero-hugetlb-pages/20241203-042817
base:   v6.12
patch link:    https://lore.kernel.org/r/20241202202058.3249628-1-fvdl%40google.com
patch subject: [PATCH] mm/hugetlb: optionally pre-zero hugetlb pages
config: powerpc-ps3_defconfig (https://download.01.org/0day-ci/archive/20241203/[email protected]/config)
compiler: powerpc64-linux-gcc (GCC) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241203/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

mm/hugetlb.c: In function 'prezero_enabled_store':

>> mm/hugetlb.c:4290:13: warning: variable 'err' set but not used [-Wunused-but-set-variable]
    4290 |         int err;
         |             ^~~

vim +/err +4290 mm/hugetlb.c

4284
4285 static ssize_t prezero_enabled_store(struct kobject *kobj,
4286 struct kobj_attribute *attr,
4287 const char *buf, size_t count)
4288 {
4289 struct hstate *h;

> 4290		int err;

4291 long val;
4292 bool prezero_enabled;
4293
4294 err = kstrtol(buf, 10, &val);
4295 if (val != 0 && val != 1)
4296 return -EINVAL;
4297
4298 prezero_enabled = !!val;
4299
4300 h = kobj_to_hstate(kobj, NULL);
4301 if (prezero_enabled == h->prezero_enabled)
4302 return count;
4303
4304 h->prezero_enabled = prezero_enabled;
4305
4306 mem_hotplug_begin();
4307 khzerod_run_hstate(h);
4308 mem_hotplug_done();
4309
4310 return count;
4311 }
4312 HSTATE_ATTR(prezero_enabled);
4313

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki