mm: split thp synchronously on MADV_DONTNEED

This patch let such applications not worry about the low level handling of THPs in the kernel and splits the THPs synchronously on MADV_DONTNEED.

Quoted 3 lines · Shakeel Butt

This patch let such applications not worry about the low level handling of THPs in the kernel and splits the THPs synchronously on MADV_DONTNEED.

I don't mind the synchronous split, but I have concerns about the implementation. I don't think it's worth another pointer in task_struct.

It's also the case that splitting is likely to succeed, so I think a better implementation would try to split and then put the page on the global deferred list if splitting fails.

Many applications do sophisticated management of their heap memory for better performance but with low cost. We have a bunch of such applications running on our production and examples include caching and data storage services. These applications keep their hot data on the THPs for better performance and release the cold data through MADV_DONTNEED to keep the memory cost low.

The kernel defers the split and release of THPs until there is memory pressure. This causes complicates the memory management of these sophisticated applications which then needs to look into low level kernel handling of THPs to better gauge their headroom for expansion. In addition these applications are very latency sensitive and would prefer to not face memory reclaim due to non-deterministic nature of reclaim.

This patch let such applications not worry about the low level handling of THPs in the kernel and splits the THPs synchronously on MADV_DONTNEED.

Signed-off-by Shakeel Butt <[email protected]>

---
 include/linux/mmzone.h   |  5 ++++
 include/linux/sched.h    |  4 ++++
 include/linux/sched/mm.h | 11 +++++++++
 kernel/fork.c            |  3 +++
 mm/huge_memory.c         | 50 ++++++++++++++++++++++++++++++++++++++++
 mm/madvise.c             |  8 +++++++
 6 files changed, 81 insertions(+)

include/linux/mmzone.h +5 -0

index 58e744b78c2c..7fa0035128b9 100644
@@ -795,6 +795,11 @@ struct deferred_split {
 	struct list_head split_queue;
 	unsigned long split_queue_len;
 };
+void split_local_deferred_list(struct list_head *defer_list);
+#else
+static inline void split_local_deferred_list(struct list_head *defer_list)
+{
+}
 #endif
 
 /*

include/linux/sched.h +0 -0

index 9d27fd0ce5df..a984bb6509d9 100644

include/linux/sched.h +31 -0

+void split_local_deferred_list(struct list_head *defer_list)
+{
+	struct list_head *pos, *next;
+	struct page *page;
+
+	/* First iteration for split. */
+	list_for_each_safe(pos, next, defer_list) {
+		page = list_entry((void *)pos, struct page, deferred_list);
+		page = compound_head(page);
+
+		if (!trylock_page(page))
+			continue;
+
+		if (split_huge_page(page)) {
+			unlock_page(page);
+			continue;
+		}
+		/* split_huge_page() removes page from list on success */
+		unlock_page(page);
+
+		/* corresponding get in deferred_split_huge_page. */
+		put_page(page);
+	}
+
+	/* Second iteration to putback failed pages. */
+	list_for_each_safe(pos, next, defer_list) {
+		struct deferred_split *ds_queue;
+		unsigned long flags;
+
+		page = list_entry((void *)pos, struct page, deferred_list);
+		page = compound_head(page);

Have you considered impact on short-living tasks where paying splitting tax would hurt performace without any benefits? Maybe a sparete madvise opration needed? I donno.

Quoted 5 lines

include/linux/sched.h +0 -0

index 9d27fd0ce5df..a984bb6509d9 100644

include/linux/sched.h struct task_struct { · line 1421

Quoted 10 lines

include/linux/sched.h +4 -0

@@ -1412,6 +1412,10 @@ struct task_struct {
      struct mem_cgroup               *active_memcg;
 #endif
 
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+     struct list_head                *deferred_split_list;
+#endif
+
 #ifdef CONFIG_BLK_CGROUP
      struct request_queue            *throttle_queue;
 #endif

It looks dirty. We really don't have options to pass it down?

Maybe passdown the list via zap_details and call a new rmap remove helper if the list is present?

Quoted 2 lines

+void split_local_deferred_list(struct list_head *defer_list)

Looks like a lot of copy-paste from deferred_split_scan(). Can we get them consolidated?

Many applications do sophisticated management of their heap memory for better performance but with low cost. We have a bunch of such applications running on our production and examples include caching and data storage services. These applications keep their hot data on the THPs for better performance and release the cold data through MADV_DONTNEED to keep the memory cost low.

The kernel defers the split and release of THPs until there is memory pressure. This causes complicates the memory management of these sophisticated applications which then needs to look into low level kernel handling of THPs to better gauge their headroom for expansion. In addition these applications are very latency sensitive and would prefer to not face memory reclaim due to non-deterministic nature of reclaim.

This patch let such applications not worry about the low level handling of THPs in the kernel and splits the THPs synchronously on MADV_DONTNEED.

Quoted 14 lines · Shakeel Butt

Many applications do sophisticated management of their heap memory for better performance but with low cost. We have a bunch of such applications running on our production and examples include caching and data storage services. These applications keep their hot data on the THPs for better performance and release the cold data through MADV_DONTNEED to keep the memory cost low.

The kernel defers the split and release of THPs until there is memory pressure. This causes complicates the memory management of these sophisticated applications which then needs to look into low level kernel handling of THPs to better gauge their headroom for expansion. In addition these applications are very latency sensitive and would prefer to not face memory reclaim due to non-deterministic nature of reclaim.

Quoted 3 lines

This patch let such applications not worry about the low level handling of THPs in the kernel and splits the THPs synchronously on MADV_DONTNEED.

I've been wondering about whether this is really the right strategy (and this goes wider than just this one, new case)

We chose to use a 2MB page here, based on whatever heuristics are currently in play. Now userspace is telling us we were wrong and should have used smaller pages.

Quoted 18 lines

Quoted 14 lines

Many applications do sophisticated management of their heap memory for better performance but with low cost. We have a bunch of such applications running on our production and examples include caching and data storage services. These applications keep their hot data on the THPs for better performance and release the cold data through MADV_DONTNEED to keep the memory cost low.

The kernel defers the split and release of THPs until there is memory pressure. This causes complicates the memory management of these sophisticated applications which then needs to look into low level kernel handling of THPs to better gauge their headroom for expansion. In addition these applications are very latency sensitive and would prefer to not face memory reclaim due to non-deterministic nature of reclaim.

Quoted 3 lines

This patch let such applications not worry about the low level handling of THPs in the kernel and splits the THPs synchronously on MADV_DONTNEED.

Quoted 6 lines

I've been wondering about whether this is really the right strategy (and this goes wider than just this one, new case)

We chose to use a 2MB page here, based on whatever heuristics are currently in play. Now userspace is telling us we were wrong and should have used smaller pages.

IIUC, not necessarily, unfortunately.

User space might be discarding the whole 2MB either via a single call (MADV_DONTNEED a 2MB range as done by virtio-balloon with "free page reporting" or by virtio-mem in QEMU). In that case, there is nothing to migrate and we were not doing anything wrong.

But more extreme, user space might be discarding the whole THP in small pieces shortly over time. This for example happens when a VM inflates the memory balloon via virtio-balloon. All inflation requests are 4k, resulting in a 4k MADV_DONTNEED calls. If we end up inflating a THP range inside of the VM, mapping to a THP range inside the hypervisor, we'll essentially free a THP in the hypervisor piece by piece using individual MADV_DONTNEED calls -- this happens frequently. Something similar can happen when de-fragmentation inside the VM "moves around" inflated 4k pages piece by piece to essentially form a huge inflated range -- this happens less frequently as of now. In both cases, migration is counter-productive, as we're just about to free the whole range either way.

(yes, there are ways to optimize, for example using hugepage ballooning or merging MADV_DONTNEED calls in the hypervisor, but what I described is what we currently implement in hypervisors like QEMU, because there are corner cases for everything)

Long story short: it's hard to tell what will happen next based on a single MADV_DONTNEED call. Page compaction, in comparison, doesn't care and optimized the layout as it observes it.

Many applications do sophisticated management of their heap memory for better performance but with low cost. We have a bunch of such applications running on our production and examples include caching and data storage services. These applications keep their hot data on the THPs for better performance and release the cold data through MADV_DONTNEED to keep the memory cost low.

The kernel defers the split and release of THPs until there is memory pressure. This causes complicates the memory management of these sophisticated applications which then needs to look into low level kernel handling of THPs to better gauge their headroom for expansion.

In
addition these applications are very latency sensitive and would prefer to not face memory reclaim due to non-deterministic nature of reclaim.

This patch let such applications not worry about the low level handling of THPs in the kernel and splits the THPs synchronously on MADV_DONTNEED.

Quoted 7 lines · Shakeel Butt

Many applications do sophisticated management of their heap memory for better performance but with low cost. We have a bunch of such applications running on our production and examples include caching and data storage services. These applications keep their hot data on the THPs for better performance and release the cold data through MADV_DONTNEED to keep the memory cost low.

Quoted 4 lines

The kernel defers the split and release of THPs until there is memory pressure. This causes complicates the memory management of these sophisticated applications which then needs to look into low level kernel handling of THPs to better gauge their headroom for expansion.

Can you elaborate a bit on that point? What exactly does such an application do? I would have assumed that it's mostly transparent for applications.

Quoted 3 lines

In
addition these applications are very latency sensitive and would prefer to not face memory reclaim due to non-deterministic nature of reclaim.

That makes sense.

Quoted 4 lines

This patch let such applications not worry about the low level handling of THPs in the kernel and splits the THPs synchronously on MADV_DONTNEED.

The main user I'm concerned about is virtio-balloon, which ends up discarding VM memory via MADV_DONTNEED when inflating the balloon in the guest in 4k granularity, but also during "free page reporting" continuously when e.g., a 2MiB page becomes free in the guest. We want both activities to be fast, and especially during "free page reporting", to defer any heavy work.

Do we have a performance evaluation how much overhead is added e.g., for a single 4k MADV_DONTNEED call on a THP or on a MADV_DONTNEED call that covers the whole THP?

Quoted 9 lines · David Hildenbrand

Quoted 7 lines · Shakeel Butt

Many applications do sophisticated management of their heap memory for better performance but with low cost. We have a bunch of such applications running on our production and examples include caching and data storage services. These applications keep their hot data on the THPs for better performance and release the cold data through MADV_DONTNEED to keep the memory cost low.

Quoted 9 lines

Quoted 4 lines

The kernel defers the split and release of THPs until there is memory pressure. This causes complicates the memory management of these sophisticated applications which then needs to look into low level kernel handling of THPs to better gauge their headroom for expansion.

Can you elaborate a bit on that point? What exactly does such an application do? I would have assumed that it's mostly transparent for applications.

The application monitors its cgroup usage to decide if it can expand the memory footprint or release some (unneeded/cold) buffer. It releases madvise(MADV_DONTNEED) to release the memory which basically puts the THP into defer list. These deferred THPs are still charged to the cgroup which leads to bloated usage read by the application and making wrong decisions. Internally we added a cgroup interface to trigger the split of deferred THPs for that cgroup but this is hacky and exposing kernel internals to users. I want to solve this problem in a more general way for the users.

Quoted 12 lines

Quoted 4 lines

In
addition these applications are very latency sensitive and would prefer to not face memory reclaim due to non-deterministic nature of reclaim.

That makes sense.

Quoted 4 lines

This patch let such applications not worry about the low level handling of THPs in the kernel and splits the THPs synchronously on MADV_DONTNEED.

Quoted 6 lines

The main user I'm concerned about is virtio-balloon, which ends up discarding VM memory via MADV_DONTNEED when inflating the balloon in the guest in 4k granularity, but also during "free page reporting" continuously when e.g., a 2MiB page becomes free in the guest. We want both activities to be fast, and especially during "free page reporting", to defer any heavy work.

Thanks for the info. What is the source virtio-balloon used for free pages?

Quoted 5 lines

Do we have a performance evaluation how much overhead is added e.g., for a single 4k MADV_DONTNEED call on a THP or on a MADV_DONTNEED call that covers the whole THP?

I did a simple benchmark of madvise(MADV_DONTNEED) on 10000 THPs on x86 for both settings you suggested. I don't see any statistically significant difference with and without the patch. Let me know if you want me to try something else.

Thanks for the details, that makes sense to me. It's essentially like another kernel buffer charged to the process, only reclaimed on memory reclaim.

(can we add that to the patch description?)

Quoted 5 lines

I did a simple benchmark of madvise(MADV_DONTNEED) on 10000 THPs on x86 for both settings you suggested. I don't see any statistically significant difference with and without the patch. Let me know if you want me to try something else.

Awesome, thanks for benchmarking. I did not check, but I assume on re-access, we won't actually re-use pages from the underlying, partially unmapped, THP, correct?

So after MADV_DONTNEED, the zapped sub-pages are essentially lost until reclaimed by splitting the THP?

If they could get
reused, there would be value in the deferred split when partially unmapping a THP.

I do wonder which purpose the deferred split serves nowadays at all. Fortunately, there is documentation: Documentation/vm/transhuge.rst:

"
Unmapping part of THP (with munmap() or other way) is not going to free memory immediately. Instead, we detect that a subpage of THP is not in use in page_remove_rmap() and queue the THP for splitting if memory pressure comes. Splitting will free up unused subpages.

Splitting the page right away is not an option due to locking context in the place where we can detect partial unmap. It also might be counterproductive since in many cases partial unmap happens during exit(2) if a THP crosses a VMA boundary.

The function deferred_split_huge_page() is used to queue a page for splitting. The splitting itself will happen when we get memory pressure via shrinker interface.
"

I do wonder which these locking contexts are exactly, and if we could also do the same thing on ordinary munmap -- because I assume it can be similarly problematic for some applications.

The "exit()" case might
indeed be interesting, but I really do wonder if this is even observable in actual number: I'm not so sure about the "many cases" but I might be wrong, of course.

Quoted 19 lines

I do wonder which purpose the deferred split serves nowadays at all. Fortunately, there is documentation: Documentation/vm/transhuge.rst:

"
Unmapping part of THP (with munmap() or other way) is not going to free memory immediately. Instead, we detect that a subpage of THP is not in use in page_remove_rmap() and queue the THP for splitting if memory pressure comes. Splitting will free up unused subpages.

Splitting the page right away is not an option due to locking context in the place where we can detect partial unmap. It also might be counterproductive since in many cases partial unmap happens during exit(2) if a THP crosses a VMA boundary.

The function deferred_split_huge_page() is used to queue a page for splitting. The splitting itself will happen when we get memory pressure via shrinker interface.
"

Quoted 3 lines

I do wonder which these locking contexts are exactly, and if we could also do the same thing on ordinary munmap -- because I assume it can be similarly problematic for some applications.

This is a good question regarding munmap. One main difference is munmap takes mmap_lock in write mode and usually performance critical applications avoid such operations.

Quoted 5 lines

The "exit()" case might
indeed be interesting, but I really do wonder if this is even observable in actual number: I'm not so sure about the "many cases" but I might be wrong, of course.

I am not worried about the exit(). The whole THP will get freed and be removed from the deferred list as well. Note that deferred list does not hold reference to the THP and has a hook in the THP destructor.

Quoted 8 lines

Quoted 4 lines

I do wonder which these locking contexts are exactly, and if we could also do the same thing on ordinary munmap -- because I assume it can be similarly problematic for some applications.

This is a good question regarding munmap. One main difference is munmap takes mmap_lock in write mode and usually performance critical applications avoid such operations.

Maybe we can extend it too most page zapping, if that makes things simpler.

Quoted 1 line · David Hildenbrand

[...]

Quoted 9 lines

Quoted 8 lines

Quoted 4 lines

I do wonder which these locking contexts are exactly, and if we could also do the same thing on ordinary munmap -- because I assume it can be similarly problematic for some applications.

This is a good question regarding munmap. One main difference is munmap takes mmap_lock in write mode and usually performance critical applications avoid such operations.

Quoted 2 lines

Maybe we can extend it too most page zapping, if that makes things simpler.

Do you mean doing sync THP split for most of page zapping functions (but only if that makes things simpler)?

Quoted 12 lines · Shakeel Butt

Quoted 1 line · David Hildenbrand

[...]

Quoted 9 lines

Quoted 8 lines

Quoted 4 lines

I do wonder which these locking contexts are exactly, and if we could also do the same thing on ordinary munmap -- because I assume it can be similarly problematic for some applications.

This is a good question regarding munmap. One main difference is munmap takes mmap_lock in write mode and usually performance critical applications avoid such operations.

Quoted 6 lines

Quoted 2 lines

Maybe we can extend it too most page zapping, if that makes things simpler.

Do you mean doing sync THP split for most of page zapping functions (but only if that makes things simpler)?

Yes -- if there are no downsides.

Quoted 21 lines · David Hildenbrand

Quoted 18 lines · Shakeel Butt

Quoted 1 line · David Hildenbrand

[...]

Quoted 11 lines

Quoted 8 lines

Quoted 4 lines

I do wonder which these locking contexts are exactly, and if we could also do the same thing on ordinary munmap -- because I assume it can be similarly problematic for some applications.

This is a good question regarding munmap. One main difference is munmap takes mmap_lock in write mode and usually performance critical applications avoid such operations.

Maybe we can extend it too most page zapping, if that makes things simpler.

Do you mean doing sync THP split for most of page zapping functions (but only if that makes things simpler)?

Quoted 2 lines

Yes -- if there are no downsides.

I will try. At the moment the assumption of "Not null zap_details implies leave swap entries" is giving me a headache.

Quoted 25 lines · Shakeel Butt

Quoted 23 lines · David Hildenbrand

Quoted 12 lines · Shakeel Butt

Quoted 1 line · David Hildenbrand

[...]

Quoted 9 lines

Quoted 8 lines

Quoted 4 lines

I do wonder which these locking contexts are exactly, and if we could also do the same thing on ordinary munmap -- because I assume it can be similarly problematic for some applications.

This is a good question regarding munmap. One main difference is munmap takes mmap_lock in write mode and usually performance critical applications avoid such operations.

Quoted 6 lines

Quoted 2 lines

Maybe we can extend it too most page zapping, if that makes things simpler.

Do you mean doing sync THP split for most of page zapping functions (but only if that makes things simpler)?

Yes -- if there are no downsides.

Quoted 2 lines

I will try. At the moment the assumption of "Not null zap_details implies leave swap entries" is giving me a headache.

Not only you, did you stumble over

https://lkml.kernel.org/r/[email protected]

already?

Quoted 27 lines · David Hildenbrand

Quoted 25 lines · Shakeel Butt

Quoted 23 lines · David Hildenbrand

Quoted 12 lines · Shakeel Butt

Quoted 1 line · David Hildenbrand

[...]

Quoted 9 lines

Quoted 8 lines

Quoted 4 lines

I do wonder which these locking contexts are exactly, and if we could also do the same thing on ordinary munmap -- because I assume it can be similarly problematic for some applications.

This is a good question regarding munmap. One main difference is munmap takes mmap_lock in write mode and usually performance critical applications avoid such operations.

Quoted 6 lines

Quoted 2 lines

Maybe we can extend it too most page zapping, if that makes things simpler.

Do you mean doing sync THP split for most of page zapping functions (but only if that makes things simpler)?

Yes -- if there are no downsides.

Quoted 6 lines

Quoted 2 lines

I will try. At the moment the assumption of "Not null zap_details implies leave swap entries" is giving me a headache.

Not only you, did you stumble over

https://lkml.kernel.org/r/[email protected]

Quoted 3 lines

already?

Oh thanks for the pointer. I missed that. I will take a look. Thanks again.

Hi, Shakeel,

I saw your v2 has started to pass in zap_details, then we need know the side effect on that skip-swap-entry thing because with your v2 code munmap() will start to skip swap entry too (while it was not before).

I saw that you didn't mention this in v2 patch either in commit message or code, not sure whether you digged that up. I think it needs some double check (or feel free to start this digging by reviewing my small patch above :).

Quoted 1 line · Peter Xu

[...]

Quoted 11 lines

Hi, Shakeel,

I saw your v2 has started to pass in zap_details, then we need know the side effect on that skip-swap-entry thing because with your v2 code munmap() will start to skip swap entry too (while it was not before).

I saw that you didn't mention this in v2 patch either in commit message or code, not sure whether you digged that up. I think it needs some double check (or feel free to start this digging by reviewing my small patch above :).

I remember mentioning in the patch that this has dependency on your rfc patch. Just checked again and yeah I only mentioned in the 'changes since v1' section. I will add it more explicitly in the following versions.

I will take a close look at your patch as well.

Quoted 3 lines

Do we have a performance evaluation how much overhead is added e.g., for a single 4k MADV_DONTNEED call on a THP or on a MADV_DONTNEED call that covers the whole THP?

I did a simple benchmark of madvise(MADV_DONTNEED) on 10000 THPs on x86 for both settings you suggested. I don't see any statistically significant difference with and without the patch. Let me know if you want me to try something else.

Quoted 8 lines · Shakeel Butt

Quoted 3 lines

Do we have a performance evaluation how much overhead is added e.g., for a single 4k MADV_DONTNEED call on a THP or on a MADV_DONTNEED call that covers the whole THP?

I did a simple benchmark of madvise(MADV_DONTNEED) on 10000 THPs on x86 for both settings you suggested. I don't see any statistically significant difference with and without the patch. Let me know if you want me to try something else.

I'm a bit surprised that sync split thp didn't bring any extra overhead.

"unmap whole thp" is understandable from that pov, because afaict that won't even trigger any thp split anyway even delayed, if this is the simplest case that only this process mapped this thp, and it mapped once.

For "unmap 4k upon thp" IIUC that's the worst case and zapping 4k should be fast; while what I don't understand since thp split requires all hand-made work for copying thp flags into small pages and so on, so I thought there should at least be some overhead measured. Shakeel, could there be something overlooked in the test, or maybe it's me that overlooked?

I had the same concern as what Kirill/Matthew raised in the other thread - I'm worried proactively splitting simply because any 4k page is zapped might quickly free up 2m thps in the system and I'm not sure whether it'll exaggerate the defragmentation of the system memory in general. I'm also not sure whether that's ideal for some very common workload that frequently uses DONTNEED to proactively drop some pages.

Quoted 22 lines · Peter Xu

Quoted 8 lines · Shakeel Butt

Quoted 3 lines

Do we have a performance evaluation how much overhead is added e.g., for a single 4k MADV_DONTNEED call on a THP or on a MADV_DONTNEED call that covers the whole THP?

I did a simple benchmark of madvise(MADV_DONTNEED) on 10000 THPs on x86 for both settings you suggested. I don't see any statistically significant difference with and without the patch. Let me know if you want me to try something else.

I'm a bit surprised that sync split thp didn't bring any extra overhead.

"unmap whole thp" is understandable from that pov, because afaict that won't even trigger any thp split anyway even delayed, if this is the simplest case that only this process mapped this thp, and it mapped once.

For "unmap 4k upon thp" IIUC that's the worst case and zapping 4k should be fast; while what I don't understand since thp split requires all hand-made work for copying thp flags into small pages and so on, so I thought there should at least be some overhead measured. Shakeel, could there be something overlooked in the test, or maybe it's me that overlooked?

Quoted 6 lines

I had the same concern as what Kirill/Matthew raised in the other thread - I'm worried proactively splitting simply because any 4k page is zapped might quickly free up 2m thps in the system and I'm not sure whether it'll exaggerate the defragmentation of the system memory in general. I'm also not sure whether that's ideal for some very common workload that frequently uses DONTNEED to proactively drop some pages.

The pageblock corresponding to the THP is movable. So (unless we start spilling unmovable allocations into movable pageblocks) we'd only place movable allocations in there. Compaction will be able to migrate to re-create a free THP.

In contrast I think, compaction will happily skip over the THP and ignore it, because it has no clue that the THP could be repurposed by split+migrate (at least I am not aware of code that does it).

Unless I am missing something, with the above in mind it could make sense to split as soon as possible, even before we're under memory pressure -- for example, for proactive compaction.

[proactive compaction could try splitting first as well I think]

Quoted 39 lines · David Hildenbrand

Quoted 22 lines · Peter Xu

Quoted 8 lines · Shakeel Butt

Quoted 3 lines

Do we have a performance evaluation how much overhead is added e.g., for a single 4k MADV_DONTNEED call on a THP or on a MADV_DONTNEED call that covers the whole THP?

I did a simple benchmark of madvise(MADV_DONTNEED) on 10000 THPs on x86 for both settings you suggested. I don't see any statistically significant difference with and without the patch. Let me know if you want me to try something else.

I'm a bit surprised that sync split thp didn't bring any extra overhead.

"unmap whole thp" is understandable from that pov, because afaict that won't even trigger any thp split anyway even delayed, if this is the simplest case that only this process mapped this thp, and it mapped once.

For "unmap 4k upon thp" IIUC that's the worst case and zapping 4k should be fast; while what I don't understand since thp split requires all hand-made work for copying thp flags into small pages and so on, so I thought there should at least be some overhead measured. Shakeel, could there be something overlooked in the test, or maybe it's me that overlooked?

Quoted 6 lines

I had the same concern as what Kirill/Matthew raised in the other thread - I'm worried proactively splitting simply because any 4k page is zapped might quickly free up 2m thps in the system and I'm not sure whether it'll exaggerate the defragmentation of the system memory in general. I'm also not sure whether that's ideal for some very common workload that frequently uses DONTNEED to proactively drop some pages.

The pageblock corresponding to the THP is movable. So (unless we start spilling unmovable allocations into movable pageblocks) we'd only place movable allocations in there. Compaction will be able to migrate to re-create a free THP.

In contrast I think, compaction will happily skip over the THP and ignore it, because it has no clue that the THP could be repurposed by split+migrate (at least I am not aware of code that does it).

Quoted 5 lines

Unless I am missing something, with the above in mind it could make sense to split as soon as possible, even before we're under memory pressure -- for example, for proactive compaction.

[proactive compaction could try splitting first as well I think]

But we can't rely on proactive compaction for rapid operations, because it's still adding overhead to the overall system by split+merge, right?

(quoted diff) +8 -0

+compaction_proactiveness
+========================
+ ...
+Note that compaction has a non-trivial system-wide impact as pages
+belonging to different processes are moved around, which could also lead
+to latency spikes in unsuspecting applications. The kernel employs
+various heuristics to avoid wasting CPU cycles if it detects that
+proactive compaction is not being effective.

Delaying split makes sense to me because after all the kernel is not aware of the userspace's preference, so the best thing is to do nothing until necessary.

Proactively split thps in dontneed/unmap added an assumption that the userspace wants to break the pages by default. It's 100% true for Shakeel's use case, but I'm not sure whether it may always be true. That's why I thought maybe a new interface is more proper, so we at least won't break anyone by accident.

Quoted 40 lines · Peter Xu

Quoted 39 lines · David Hildenbrand

Quoted 22 lines · Peter Xu

Quoted 8 lines · Shakeel Butt

Quoted 3 lines

Do we have a performance evaluation how much overhead is added e.g., for a single 4k MADV_DONTNEED call on a THP or on a MADV_DONTNEED call that covers the whole THP?

I did a simple benchmark of madvise(MADV_DONTNEED) on 10000 THPs on x86 for both settings you suggested. I don't see any statistically significant difference with and without the patch. Let me know if you want me to try something else.

I'm a bit surprised that sync split thp didn't bring any extra overhead.

"unmap whole thp" is understandable from that pov, because afaict that won't even trigger any thp split anyway even delayed, if this is the simplest case that only this process mapped this thp, and it mapped once.

For "unmap 4k upon thp" IIUC that's the worst case and zapping 4k should be fast; while what I don't understand since thp split requires all hand-made work for copying thp flags into small pages and so on, so I thought there should at least be some overhead measured. Shakeel, could there be something overlooked in the test, or maybe it's me that overlooked?

Quoted 6 lines

I had the same concern as what Kirill/Matthew raised in the other thread - I'm worried proactively splitting simply because any 4k page is zapped might quickly free up 2m thps in the system and I'm not sure whether it'll exaggerate the defragmentation of the system memory in general. I'm also not sure whether that's ideal for some very common workload that frequently uses DONTNEED to proactively drop some pages.

The pageblock corresponding to the THP is movable. So (unless we start spilling unmovable allocations into movable pageblocks) we'd only place movable allocations in there. Compaction will be able to migrate to re-create a free THP.

In contrast I think, compaction will happily skip over the THP and ignore it, because it has no clue that the THP could be repurposed by split+migrate (at least I am not aware of code that does it).

Quoted 8 lines

Quoted 5 lines

Unless I am missing something, with the above in mind it could make sense to split as soon as possible, even before we're under memory pressure -- for example, for proactive compaction.

[proactive compaction could try splitting first as well I think]

But we can't rely on proactive compaction for rapid operations, because it's still adding overhead to the overall system by split+merge, right?

Yes, but there is also direct compaction that can be triggered without the shrinker getting involved. I think we can summarize as "there might not be a right or wrong when to split". An application that MADV_DONTNEEDs/munmap sub-THP memory told us that it doesn't want to consume memory, yet it looks like it's still consuming that memory.

I do wonder how THP on the deferred split queue behave in respect to page migration -- memory offlining, alloc_contig_range(). I saw reports

that there are some cases where THP can be problematic when stress-testing THP:
https://lkml.kernel.org/r/20210903162102.GA10039@mam-gdavis-dt

But not sure if that's related to deferred splitting. Most probably not.

Quoted 10 lines

(quoted diff) +8 -0

+compaction_proactiveness
+========================
+ ...
+Note that compaction has a non-trivial system-wide impact as pages
+belonging to different processes are moved around, which could also lead
+to latency spikes in unsuspecting applications. The kernel employs
+various heuristics to avoid wasting CPU cycles if it detects that
+proactive compaction is not being effective.

Quoted 7 lines

Delaying split makes sense to me because after all the kernel is not aware of the userspace's preference, so the best thing is to do nothing until necessary.

Proactively split thps in dontneed/unmap added an assumption that the userspace wants to break the pages by default. It's 100% true for Shakeel's use case, but I'm not sure whether it may always be true. That's why I thought maybe a new interface is more proper, so we at least won't break anyone by accident.

Well, we already broke the PMD into PTEs. So the performance gain at least for that user is really gone until we "fix that" again via khugepaged -- which might just be configured to not "fix" if there are empty PTEs.

It for sure is interesting if you have a COW huge page and only one party zaps/unmaps some part.

--
Thanks,

David / dhildenb

Quoted 8 lines · Shakeel Butt

Quoted 3 lines

Do we have a performance evaluation how much overhead is added e.g., for a single 4k MADV_DONTNEED call on a THP or on a MADV_DONTNEED call that covers the whole THP?

I did a simple benchmark of madvise(MADV_DONTNEED) on 10000 THPs on x86 for both settings you suggested. I don't see any statistically significant difference with and without the patch. Let me know if you want me to try something else.

I'm a bit surprised that sync split thp didn't bring any extra overhead.

"unmap whole thp" is understandable from that pov, because afaict that won't even trigger any thp split anyway even delayed, if this is the simplest case that only this process mapped this thp, and it mapped once.

For "unmap 4k upon thp" IIUC that's the worst case and zapping 4k should be fast; while what I don't understand since thp split requires all hand-made work for copying thp flags into small pages and so on, so I thought there should at least be some overhead measured. Shakeel, could there be something overlooked in the test, or maybe it's me that overlooked?

Quoted 17 lines · Peter Xu

Quoted 8 lines · Shakeel Butt

Quoted 3 lines

Do we have a performance evaluation how much overhead is added e.g., for a single 4k MADV_DONTNEED call on a THP or on a MADV_DONTNEED call that covers the whole THP?

I did a simple benchmark of madvise(MADV_DONTNEED) on 10000 THPs on x86 for both settings you suggested. I don't see any statistically significant difference with and without the patch. Let me know if you want me to try something else.

I'm a bit surprised that sync split thp didn't bring any extra overhead.

"unmap whole thp" is understandable from that pov, because afaict that won't even trigger any thp split anyway even delayed, if this is the simplest case that only this process mapped this thp, and it mapped once.

Quoted 6 lines

For "unmap 4k upon thp" IIUC that's the worst case and zapping 4k should be fast; while what I don't understand since thp split requires all hand-made work for copying thp flags into small pages and so on, so I thought there should at least be some overhead measured. Shakeel, could there be something overlooked in the test, or maybe it's me that overlooked?

Thanks for making me rerun this and yes indeed I had a very silly bug in the benchmark code (i.e. madvise the same page for the whole loop) and this is indeed several times slower than without the patch (sorry David for misleading you).

To better understand what is happening, I profiled the benchmark:

- 31.27% 0.01% dontneed [kernel.kallsyms] [k] zap_page_range_sync
- 31.27% zap_page_range_sync
- 30.25% split_local_deferred_list
- 30.16% split_huge_page_to_list
- 21.05% try_to_migrate
+ rmap_walk_anon
- 7.47% remove_migration_ptes
+ 7.34% rmap_walk_locked
+ 1.02% zap_page_range_details

The overhead is not due to copying page flags but rather due to two rmap walks. I don't think this much overhead is justified for current users of MADV_DONTNEED and munmap. I have to rethink the approach.

Quoted 5 lines

Thanks for making me rerun this and yes indeed I had a very silly bug in the benchmark code (i.e. madvise the same page for the whole loop) and this is indeed several times slower than without the patch (sorry David for misleading you).

Quoted 12 lines

To better understand what is happening, I profiled the benchmark:

- 31.27% 0.01% dontneed [kernel.kallsyms] [k] zap_page_range_sync
- 31.27% zap_page_range_sync
- 30.25% split_local_deferred_list
- 30.16% split_huge_page_to_list
- 21.05% try_to_migrate
+ rmap_walk_anon
- 7.47% remove_migration_ptes
+ 7.34% rmap_walk_locked
+ 1.02% zap_page_range_details

Makes sense, thanks for verifying it, Shakeel. I forgot it'll also walk itself.

I believe this effect will be exaggerated when the mapping is shared, e.g. shmem file thp between processes. What's worse is that when one process DONTNEED one 4k page, all the rest mms will need to split the huge pmd without even being noticed, so that's a direct suffer from perf degrade.

Quoted 5 lines

The overhead is not due to copying page flags but rather due to two rmap walks. I don't think this much overhead is justified for current users of MADV_DONTNEED and munmap. I have to rethink the approach.

Some side notes: I digged out the old MADV_COLLAPSE proposal right after I thought about MADV_SPLIT (or any of its variance):

https://lore.kernel.org/all/[email protected]/

My memory was that there's some issue to be solved so that was blocked, however when I read the thread it sounds like the list was mostly reaching a consensus on considering MADV_COLLAPSE being beneficial. Still copying DavidR in case I missed something important.

If we think MADV_COLLAPSE can help to implement an userspace (and more importantly, data-aware) khugepaged, then MADV_SPLIT can be the other side of kcompactd, perhaps.

That's probably a bit off topic of this specific discussion on the specific use case, but so far it seems all reasonable and discussable.

Would this really apply to MADV_DONTNEED on shmem, and would deferred splitting apply on shmem? I'm constantly confused about shmem vs. anon, but I would have assumed that shmem is fd-based and we wouldn't end up in rmap_walk_anon. For shmem, the pagecache would contain the THP which would stick around and deferred splits don't even apply.

User space can trigger a split manually using some MADV hackery. But it can only be used for the use case here, where we actually want to zap a page.

1. MADV_FREE a single 4k page in the range. This will split the PMD->PTE and the compound page.

2. MADV_DONTNEED either the complete range or the single 4k page.

Quoted 6 lines

Quoted 6 lines

Thanks for making me rerun this and yes indeed I had a very silly bug in the benchmark code (i.e. madvise the same page for the whole loop) and this is indeed several times slower than without the patch (sorry David for misleading you).

No worries, BUGs happen :)

Quoted 13 lines

Quoted 12 lines

To better understand what is happening, I profiled the benchmark:

- 31.27% 0.01% dontneed [kernel.kallsyms] [k] zap_page_range_sync
- 31.27% zap_page_range_sync
- 30.25% split_local_deferred_list
- 30.16% split_huge_page_to_list
- 21.05% try_to_migrate
+ rmap_walk_anon
- 7.47% remove_migration_ptes
+ 7.34% rmap_walk_locked
+ 1.02% zap_page_range_details

Quoted 7 lines

Makes sense, thanks for verifying it, Shakeel. I forgot it'll also walk itself.

I believe this effect will be exaggerated when the mapping is shared, e.g. shmem file thp between processes. What's worse is that when one process DONTNEED one 4k page, all the rest mms will need to split the huge pmd without even being noticed, so that's a direct suffer from perf degrade.

Would this really apply to MADV_DONTNEED on shmem, and would deferred splitting apply on shmem? I'm constantly confused about shmem vs. anon, but I would have assumed that shmem is fd-based and we wouldn't end up in rmap_walk_anon. For shmem, the pagecache would contain the THP which would stick around and deferred splits don't even apply.

But again, I'm constantly confused so I'd love to be enlighted.

Quoted 5 lines

Quoted 5 lines

The overhead is not due to copying page flags but rather due to two rmap walks. I don't think this much overhead is justified for current users of MADV_DONTNEED and munmap. I have to rethink the approach.

Most probably not.

Quoted 11 lines

Some side notes: I digged out the old MADV_COLLAPSE proposal right after I thought about MADV_SPLIT (or any of its variance):

https://lore.kernel.org/all/[email protected]/

My memory was that there's some issue to be solved so that was blocked, however when I read the thread it sounds like the list was mostly reaching a consensus on considering MADV_COLLAPSE being beneficial. Still copying DavidR in case I missed something important.

Quoted 6 lines

If we think MADV_COLLAPSE can help to implement an userspace (and more importantly, data-aware) khugepaged, then MADV_SPLIT can be the other side of kcompactd, perhaps.

That's probably a bit off topic of this specific discussion on the specific use case, but so far it seems all reasonable and discussable.

User space can trigger a split manually using some MADV hackery. But it can only be used for the use case here, where we actually want to zap a page.

1. MADV_FREE a single 4k page in the range. This will split the PMD->PTE and the compound page.
2. MADV_DONTNEED either the complete range or the single 4k page.

--
Thanks,

David / dhildenb

Some side notes: I digged out the old MADV_COLLAPSE proposal right after I thought about MADV_SPLIT (or any of its variance):

https://lore.kernel.org/all/[email protected]/

My memory was that there's some issue to be solved so that was blocked, however when I read the thread it sounds like the list was mostly reaching a consensus on considering MADV_COLLAPSE being beneficial. Still copying DavidR in case I missed something important.

If we think MADV_COLLAPSE can help to implement an userspace (and more importantly, data-aware) khugepaged, then MADV_SPLIT can be the other side of kcompactd, perhaps.

That's probably a bit off topic of this specific discussion on the specific use case, but so far it seems all reasonable and discussable.