mm: Add callback for defining compaction completion

For some applications we need to allocate almost all memory as hugepages. However, on a running system, higher order allocations can fail if the memory is fragmented. Linux kernel currently does on-demand compaction as we request more hugepages but this style of compaction incurs very high latency. Experiments with one-time full memory compaction (followed by hugepage allocations) shows that kernel is able to restore a highly fragmented memory state to a fairly compacted memory state within <1 sec for a 32G system. Such data suggests that a more proactive compaction can help us allocate a large fraction of memory as hugepages keeping allocation latencies low.

In general, compaction can introduce unexpected latencies for applications that don't even have strong requirements for contiguous allocations. It is also hard to efficiently determine if the current system state can be easily compacted due to mixing of unmovable memory. Due to these reasons, automatic background compaction by the kernel itself is hard to get right in a way which does not hurt unsuspecting applications or waste CPU cycles.

Even with these caveats, pro-active compaction can still be very useful in certain scenarios to reduce hugepage allocation latencies. This callback interface allows drivers to drive compaction based on their own policies like the current level of external fragmentation for a particular order, system load etc.

Signed-off-by Nitin Gupta <[email protected]>

---
 include/linux/compaction.h | 10 ++++++++++
 mm/compaction.c            | 20 ++++++++++++++------
 mm/internal.h              |  2 ++
 3 files changed, 26 insertions(+), 6 deletions(-)

include/linux/compaction.h +10 -0

index 9569e7c786d3..1ea828450fa2 100644
@@ -58,6 +58,16 @@ enum compact_result {
 	COMPACT_SUCCESS,
 };
 
+/* Callback function to determine if compaction is finished. */
+typedef enum compact_result (*compact_finished_cb)(
+	struct zone *zone, int order);
+
+enum compact_result compact_zone_order(struct zone *zone, int order,
+		gfp_t gfp_mask, enum compact_priority prio,
+		unsigned int alloc_flags, int classzone_idx,
+		struct page **capture,
+		compact_finished_cb compact_finished_cb);
+
 struct alloc_context; /* in mm/internal.h */
 
 /*

mm/compaction.c +14 -6

index 952dc2fb24e5..73e2e9246bc4 100644
@@ -1872,6 +1872,9 @@ static enum compact_result __compact_finished(struct compact_control *cc)
 			return COMPACT_PARTIAL_SKIPPED;
 	}
 
+	if (cc->compact_finished_cb)
+		return cc->compact_finished_cb(cc->zone, cc->order);
+
 	if (is_via_compact_memory(cc->order))
 		return COMPACT_CONTINUE;
 

@@ -2274,10 +2277,11 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
 	return ret;
 }
 
-static enum compact_result compact_zone_order(struct zone *zone, int order,
+enum compact_result compact_zone_order(struct zone *zone, int order,
 		gfp_t gfp_mask, enum compact_priority prio,
 		unsigned int alloc_flags, int classzone_idx,
-		struct page **capture)
+		struct page **capture,
+		compact_finished_cb compact_finished_cb)
 {
 	enum compact_result ret;
 	struct compact_control cc = {

@@ -2293,10 +2297,11 @@ static enum compact_result compact_zone_order(struct zone *zone, int order,
 					MIGRATE_ASYNC :	MIGRATE_SYNC_LIGHT,
 		.alloc_flags = alloc_flags,
 		.classzone_idx = classzone_idx,
-		.direct_compaction = true,
+		.direct_compaction = !compact_finished_cb,
 		.whole_zone = (prio == MIN_COMPACT_PRIORITY),
 		.ignore_skip_hint = (prio == MIN_COMPACT_PRIORITY),
-		.ignore_block_suitable = (prio == MIN_COMPACT_PRIORITY)
+		.ignore_block_suitable = (prio == MIN_COMPACT_PRIORITY),
+		.compact_finished_cb = compact_finished_cb
 	};
 	struct capture_control capc = {
 		.cc = &cc,

@@ -2313,11 +2318,13 @@ static enum compact_result compact_zone_order(struct zone *zone, int order,
 	VM_BUG_ON(!list_empty(&cc.freepages));
 	VM_BUG_ON(!list_empty(&cc.migratepages));
 
-	*capture = capc.page;
+	if (capture)
+		*capture = capc.page;
 	current->capture_control = NULL;
 
 	return ret;
 }
+EXPORT_SYMBOL(compact_zone_order);
 
 int sysctl_extfrag_threshold = 500;
 

@@ -2361,7 +2368,8 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
 		}
 
 		status = compact_zone_order(zone, order, gfp_mask, prio,
-				alloc_flags, ac_classzone_idx(ac), capture);
+				alloc_flags, ac_classzone_idx(ac), capture,
+				NULL);
 		rc = max(status, rc);
 
 		/* The allocation should succeed, stop compacting */

mm/internal.h +2 -0

index e32390802fd3..f873f7c2b9dc 100644
@@ -11,6 +11,7 @@
 #include <linux/mm.h>
 #include <linux/pagemap.h>
 #include <linux/tracepoint-defs.h>
+#include <linux/compaction.h>
 
 /*
  * The set of flags that only affect watermark checking and reclaim

@@ -203,6 +204,7 @@ struct compact_control {
 	bool whole_zone;		/* Whole zone should/has been scanned */
 	bool contended;			/* Signal lock or sched contention */
 	bool rescan;			/* Rescanning the same pageblock */
+	compact_finished_cb compact_finished_cb;
 };
 
 /*

Signature · 1 line

2.21.0

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Michal Hocko
Sent: Tuesday, September 10, 2019 1:19 PM
To: Nitin Gupta <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Matthew Wilcox <[email protected]>; Yu Zhao <[email protected]>; Qian Cai <[email protected]>; Andrey Ryabinin <[email protected]>; Allison Randal <[email protected]>; Mike Rapoport <[email protected]>; Thomas Gleixner <[email protected]>; Arun KS <[email protected]>; Wei Yang <[email protected]>; [email protected]; linux- [email protected]
Subject: Re: [PATCH] mm: Add callback for defining compaction completion

Quoted 1 line · Nitin Gupta

For some applications we need to allocate almost all memory as

hugepages.

Quoted 2 lines

However, on a running system, higher order allocations can fail if the memory is fragmented. Linux kernel currently does on-demand

compaction

Quoted 8 lines

as we request more hugepages but this style of compaction incurs very high latency. Experiments with one-time full memory compaction (followed by hugepage allocations) shows that kernel is able to restore a highly fragmented memory state to a fairly compacted memory state within <1 sec for a 32G system. Such data suggests that a more proactive compaction can help us allocate a large fraction of memory as hugepages keeping allocation latencies low.

Quoted 6 lines

In general, compaction can introduce unexpected latencies for applications that don't even have strong requirements for contiguous allocations. It is also hard to efficiently determine if the current system state can be easily compacted due to mixing of unmovable memory. Due to these reasons, automatic background compaction by the kernel itself is hard to get right in a way which does not hurt unsuspecting

applications or waste CPU cycles.

We do trigger background compaction on a high order pressure from the page allocator by waking up kcompactd. Why is that not sufficient?

Quoted 5 lines

Even with these caveats, pro-active compaction can still be very useful in certain scenarios to reduce hugepage allocation latencies. This callback interface allows drivers to drive compaction based on their own policies like the current level of external fragmentation for a particular order, system load etc.

So we do not trust the core MM to make a reasonable decision while we give a free ticket to modules. How does this make any sense at all? How is a random module going to make a more informed decision when it has less visibility on the overal MM situation.

If you need to control compaction from the userspace you have an interface for that. It is also completely unexplained why you need a completion callback.

mm/compaction.c static enum compact_result

Quoted 34 lines

Signed-off-by Nitin Gupta <[email protected]>

---
 include/linux/compaction.h | 10 ++++++++++
 mm/compaction.c            | 20 ++++++++++++++------
 mm/internal.h              |  2 ++
 3 files changed, 26 insertions(+), 6 deletions(-)

include/linux/compaction.h +10 -0

index 9569e7c786d3..1ea828450fa2 100644
@@ -58,6 +58,16 @@ enum compact_result {
 	COMPACT_SUCCESS,
 };
 
+/* Callback function to determine if compaction is finished. */
+typedef enum compact_result (*compact_finished_cb)(
+	struct zone *zone, int order);
+
+enum compact_result compact_zone_order(struct zone *zone, int order,
+		gfp_t gfp_mask, enum compact_priority prio,
+		unsigned int alloc_flags, int classzone_idx,
+		struct page **capture,
+		compact_finished_cb compact_finished_cb);
+
 struct alloc_context; /* in mm/internal.h */
 
 /*

mm/compaction.c index renamed from mm/compaction.c +0 -0

952dc2fb24e5..73e2e9246bc4 100644

mm/compaction.c +0 -0

@@ -1872,6 +1872,9 @@ static enum compact_result

__compact_finished(struct compact_control *cc)

mm/compaction.c compact_zone(struct compact_control *cc,

Quoted 10 lines

mm/compaction.c +3 -0

 			return COMPACT_PARTIAL_SKIPPED;
 	}
 
+	if (cc->compact_finished_cb)
+		return cc->compact_finished_cb(cc->zone, cc->order);
+
 	if (is_via_compact_memory(cc->order))
 		return COMPACT_CONTINUE;
 

@@ -2274,10 +2277,11 @@ compact_zone(struct compact_control *cc,

struct capture_control *capc)

mm/compaction.c static enum compact_result

Quoted 15 lines

mm/compaction.c +3 -2

 	return ret;
 }
 
-static enum compact_result compact_zone_order(struct zone *zone, int
 order,
+enum compact_result compact_zone_order(struct zone *zone, int order,
 		gfp_t gfp_mask, enum compact_priority prio,
 		unsigned int alloc_flags, int classzone_idx,
-		struct page **capture)
+		struct page **capture,
+		compact_finished_cb compact_finished_cb)
 {
 	enum compact_result ret;

mm/compaction.c +0 -0

 	struct compact_control cc = {

@@ -2293,10 +2297,11 @@ static enum compact_result

compact_zone_order(struct zone *zone, int order,

mm/compaction.c static enum compact_result · line 2297

Quoted 1 line

mm/compaction.c +0 -0

 					MIGRATE_ASYNC :

MIGRATE_SYNC_LIGHT,

mm/compaction.c static enum compact_result compact_zone_order(struct zone *zone, int order,

Quoted 8 lines

mm/compaction.c +2 -2

 		.alloc_flags = alloc_flags,
 		.classzone_idx = classzone_idx,
-		.direct_compaction = true,
+		.direct_compaction = !compact_finished_cb,
 		.whole_zone = (prio == MIN_COMPACT_PRIORITY),
 		.ignore_skip_hint = (prio == MIN_COMPACT_PRIORITY),
-		.ignore_block_suitable = (prio == MIN_COMPACT_PRIORITY)
+		.ignore_block_suitable = (prio ==

MIN_COMPACT_PRIORITY),

mm/compaction.c static enum compact_result

Quoted 5 lines

mm/compaction.c +1 -0

+		.compact_finished_cb = compact_finished_cb
 	};
 	struct capture_control capc = {
 		.cc = &cc,

@@ -2313,11 +2318,13 @@ static enum compact_result

compact_zone_order(struct zone *zone, int order,

mm/compaction.c enum compact_result

Quoted 15 lines

mm/compaction.c +3 -1

 	VM_BUG_ON(!list_empty(&cc.freepages));
 	VM_BUG_ON(!list_empty(&cc.migratepages));
 
-	*capture = capc.page;
+	if (capture)
+		*capture = capc.page;
 	current->capture_control = NULL;
 
 	return ret;
 }
+EXPORT_SYMBOL(compact_zone_order);
 
 int sysctl_extfrag_threshold = 500;
 

@@ -2361,7 +2368,8 @@ enum compact_result

try_to_compact_pages(gfp_t gfp_mask, unsigned int order,

Quoted 22 lines

mm/compaction.c +2 -1

 		}
 
 		status = compact_zone_order(zone, order, gfp_mask, prio,
-				alloc_flags, ac_classzone_idx(ac), capture);
+				alloc_flags, ac_classzone_idx(ac), capture,
+				NULL);
 		rc = max(status, rc);
 
 		/* The allocation should succeed, stop compacting */ diff --git

a/mm/internal.h b/mm/internal.h index e32390802fd3..f873f7c2b9dc 100644

mm/internal.h +1 -0

@@ -11,6 +11,7 @@
 #include <linux/mm.h>
 #include <linux/pagemap.h>
 #include <linux/tracepoint-defs.h>
+#include <linux/compaction.h>
 
 /*
  * The set of flags that only affect watermark checking and reclaim

mm/internal.h struct compact_control { · line 204

Quoted 1 line

mm/internal.h +0 -0

@@ -203,6 +204,7 @@ struct compact_control {
 	bool whole_zone;		/* Whole zone should/has been

scanned */

mm/internal.h struct compact_control { · line 205

Quoted 1 line

mm/internal.h +0 -0

 	bool contended;			/* Signal lock or sched

contention */

Quoted 7 lines

mm/internal.h +1 -1

 	bool rescan;			/* Rescanning the same pageblock */
+	compact_finished_cb compact_finished_cb;
 };
 
 /*
--

2.21.0

--
Michal Hocko
SUSE Labs

Quoted 14 lines · Nitin Gupta

Quoted 13 lines

For some applications we need to allocate almost all memory as hugepages.
However, on a running system, higher order allocations can fail if the memory is fragmented. Linux kernel currently does on-demand compaction
as we request more hugepages but this style of compaction incurs very high latency. Experiments with one-time full memory compaction (followed by hugepage allocations) shows that kernel is able to restore a highly fragmented memory state to a fairly compacted memory state within <1 sec for a 32G system. Such data suggests that a more proactive compaction can help us allocate a large fraction of memory as hugepages keeping allocation latencies low.

Quoted 3 lines

Quoted 3 lines

In general, compaction can introduce unexpected latencies for applications that don't even have strong requirements for contiguous allocations.

Quoted 4 lines

Quoted 4 lines

It is also hard to efficiently determine if the current
system state can be easily compacted due to mixing of unmovable memory. Due to these reasons, automatic background compaction by the kernel itself is hard to get right in a way which does not hurt unsuspecting

Quoted 5 lines

applications or waste CPU cycles.

We do trigger background compaction on a high order pressure from the page allocator by waking up kcompactd. Why is that not sufficient?

Whenever kcompactd is woken up, it does just enough work to create one free page of the given order (compaction_control.order) or higher.

Such a design causes very high latency for workloads where we want to allocate lots of hugepages in short period of time. With pro-active compaction we can hide much of this latency. For some more background discussion and data, please see this thread:

https://patchwork.kernel.org/patch/11098289/

Quoted 6 lines

Quoted 5 lines

Even with these caveats, pro-active compaction can still be very useful in certain scenarios to reduce hugepage allocation latencies. This callback interface allows drivers to drive compaction based on their own policies like the current level of external fragmentation for a particular order, system load etc.

Quoted 5 lines

So we do not trust the core MM to make a reasonable decision while we give a free ticket to modules. How does this make any sense at all? How is a random module going to make a more informed decision when it has less visibility on the overal MM situation.

Embedding any specific policy (like: keep external fragmentation for order-9 between 30-40%) within MM core looks like a bad idea.

As a driver, we
can easily measure parameters like system load, current fragmentation level for any order in any zone etc. to make an informed decision. See the thread I refereed above for more background discussion.

Quoted 4 lines

If you need to control compaction from the userspace you have an interface for that. It is also completely unexplained why you need a completion callback.

/proc/sys/vm/compact_memory does whole system compaction which is often too much as a pro-active compaction strategy. To get more control over how to compaction work to do, I have added a compaction callback which controls how much work is done in one compaction cycle.

On Tue 10-09-19 22:27:53, Nitin Gupta wrote:
[...]

Quoted 14 lines

Quoted 14 lines · Nitin Gupta

Quoted 13 lines

For some applications we need to allocate almost all memory as hugepages.
However, on a running system, higher order allocations can fail if the memory is fragmented. Linux kernel currently does on-demand compaction
as we request more hugepages but this style of compaction incurs very high latency. Experiments with one-time full memory compaction (followed by hugepage allocations) shows that kernel is able to restore a highly fragmented memory state to a fairly compacted memory state within <1 sec for a 32G system. Such data suggests that a more proactive compaction can help us allocate a large fraction of memory as hugepages keeping allocation latencies low.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

In general, compaction can introduce unexpected latencies for applications that don't even have strong requirements for contiguous allocations.

Could you expand on this a bit please? Gfp flags allow to express how much the allocator try and compact for a high order allocations. Hugetlb allocations tend to require retrying and heavy compaction to succeed and the success rate tends to be pretty high from my experience. Why that is not case in your case?

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

It is also hard to efficiently determine if the current system state can be easily compacted due to mixing of unmovable memory. Due to these reasons, automatic background compaction by the kernel itself is hard to get right in a way which does not hurt unsuspecting

Quoted 8 lines

Quoted 5 lines

applications or waste CPU cycles.

We do trigger background compaction on a high order pressure from the page allocator by waking up kcompactd. Why is that not sufficient?

Whenever kcompactd is woken up, it does just enough work to create one free page of the given order (compaction_control.order) or higher.

This is an implementation detail IMHO. I am pretty sure we can do a better auto tuning when there is an indication of a constant flow of high order requests. This is no different from the memory reclaim in principle. Just because the kswapd autotuning not fitting with your particular workload you wouldn't want to export direct reclaim functionality and call it from a random module. That is just doomed to fail because different subsystems in control just leads to decisions going against each other.

Quoted 6 lines

Such a design causes very high latency for workloads where we want to allocate lots of hugepages in short period of time. With pro-active compaction we can hide much of this latency. For some more background discussion and data, please see this thread:

https://patchwork.kernel.org/patch/11098289/

I am aware of that thread. And there are two things. You claim the allocation success rate is unnecessarily lower and that the direct latency is high. You simply cannot assume both low latency and high success rate. Compaction is not free. Somebody has to do the work. Hiding it into the background means that you are eating a lot of cycles from everybody else (think of a workload running in a restricted cpu controller just doing a lot of work in an unaccounted context).

That being said you really have to be prepared to pay a price for precious resource like high order pages.

On the other hand I do understand that high latency is not really desired for a more optimistic allocation requests with a reasonable fallback strategy. Those would benefit from kcompactd not giving up too early.

Quoted 14 lines

Quoted 7 lines

Quoted 6 lines

Even with these caveats, pro-active compaction can still be very useful in certain scenarios to reduce hugepage allocation latencies. This callback interface allows drivers to drive compaction based on their own policies like the current level of external fragmentation for a particular order, system load etc.

Quoted 6 lines

So we do not trust the core MM to make a reasonable decision while we give
a free ticket to modules. How does this make any sense at all? How is a random module going to make a more informed decision when it has less visibility on the overal MM situation.

Quoted 3 lines

Embedding any specific policy (like: keep external fragmentation for order-9
between 30-40%) within MM core looks like a bad idea.

Agreed

Quoted 5 lines

As a driver, we
can easily measure parameters like system load, current fragmentation level
for any order in any zone etc. to make an informed decision. See the thread I refereed above for more background discussion.

Do that from the userspace then. If there is an insufficient interface to do that then let's talk about what is missing.

Quoted 10 lines

Quoted 5 lines

If you need to control compaction from the userspace you have an interface
for that. It is also completely unexplained why you need a completion callback.

/proc/sys/vm/compact_memory does whole system compaction which is often too much as a pro-active compaction strategy. To get more control over how to compaction work to do, I have added a compaction callback which controls how much work is done in one compaction cycle.

Why is a more fine grained control really needed? Sure compacting everything is heavy weight but how often do you have to do that. Your changelog starts with a usecase when there is a high demand for large pages at the startup. What prevents you do compaction at that time. If the workload is longterm then the initial price should just pay back, no?

Quoted 20 lines · Michal Hocko

On Tue 10-09-19 22:27:53, Nitin Gupta wrote:
[...]

Quoted 14 lines

Quoted 14 lines · Nitin Gupta

Quoted 13 lines

For some applications we need to allocate almost all memory as hugepages.
However, on a running system, higher order allocations can fail if the memory is fragmented. Linux kernel currently does on-demand compaction
as we request more hugepages but this style of compaction incurs very high latency. Experiments with one-time full memory compaction (followed by hugepage allocations) shows that kernel is able to restore a highly fragmented memory state to a fairly compacted memory state within <1 sec for a 32G system. Such data suggests that a more proactive compaction can help us allocate a large fraction of memory as hugepages keeping allocation latencies low.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

In general, compaction can introduce unexpected latencies for applications that don't even have strong requirements for contiguous allocations.

Quoted 6 lines

Could you expand on this a bit please? Gfp flags allow to express how much the allocator try and compact for a high order allocations. Hugetlb allocations tend to require retrying and heavy compaction to succeed and the success rate tends to be pretty high from my experience. Why that is not case in your case?

Yes, I have the same observation: with GFP_TRANSHUGE | __GFP_RETRY_MAYFAIL I get very good success rate (~90% of free RAM allocated as hugepages). However, what I'm trying to point out is that this high success rate comes with high allocation latencies (90th percentile latency of 2206us). On the same system, the same high-order allocations which hit the fast path have latency <5us.

Quoted 14 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

It is also hard to efficiently determine if the current system state can be easily compacted due to mixing of unmovable memory. Due to these reasons, automatic background compaction by the kernel itself is hard to get right in a way which does not hurt unsuspecting

Quoted 8 lines

Quoted 5 lines

applications or waste CPU cycles.

We do trigger background compaction on a high order pressure from the page allocator by waking up kcompactd. Why is that not sufficient?

Whenever kcompactd is woken up, it does just enough work to create one free page of the given order (compaction_control.order) or higher.

Quoted 9 lines

This is an implementation detail IMHO. I am pretty sure we can do a better auto tuning when there is an indication of a constant flow of high order requests. This is no different from the memory reclaim in principle. Just because the kswapd autotuning not fitting with your particular workload you wouldn't want to export direct reclaim functionality and call it from a random module. That is just doomed to fail because different subsystems in control just leads to decisions going against each other.

I don't want to go the route of adding any auto-tuning/perdiction code to control compaction in the kernel. I'm more inclined towards extending existing interfaces to allow compaction behavior to be controlled either from userspace or a kernel driver. Letting a random module control compaction or a root process pumping new tunables from sysfs is the same in principle.

This patch is in the spirit of simple extension to existing compaction_zone_order() which allows either a kernel driver or userspace (through sysfs) to control compaction.

Also, we should avoid driving hard parallels between reclaim and
compaction: the former is often necessary for forward progress while the latter is often an optimization. Since contiguous allocations are mostly optimizations it's good to expose hooks from the kernel that let user (through a driver or userspace) control it using its own heuristics.

I thought hard about whats lacking in current userspace interface (sysfs):
- /proc/sys/vm/compact_memory: full system compaction is not an option as a viable pro-active compaction strategy.

- possibly expose [low, high] threshold values for each node and let kcompactd act on them. This was my approach for my original patch I linked earlier. Problem here is that it introduces too many tunables.

Considering the above, I came up with this callback approach which make it trivial to introduce user specific policies for compaction. It puts the onus of system stability, responsive in the hands of user without burdening admins with more tunables or adding crystal balls to kernel.

Quoted 3 lines

Do that from the userspace then. If there is an insufficient interface to do that then let's talk about what is missing.

Currently we only have a proc interface to do full system compaction. Here's what missing from this interface: ability to set per-node, per-zone, per-order, [low, high] extfrag thresholds.

Quoted 20 lines · Michal Hocko

On Tue 10-09-19 22:27:53, Nitin Gupta wrote:
[...]

Quoted 17 lines

Quoted 17 lines · Nitin Gupta

Quoted 16 lines

For some applications we need to allocate almost all memory as hugepages.
However, on a running system, higher order allocations can fail if the memory is fragmented. Linux kernel currently does on-demand compaction
as we request more hugepages but this style of compaction incurs very high latency. Experiments with one-time full memory compaction (followed by hugepage allocations) shows that kernel is able to restore a highly fragmented memory state to a fairly compacted memory state within <1 sec for a 32G system. Such data suggests that a more proactive compaction can help us allocate a large fraction of memory as hugepages keeping allocation latencies low.

In general, compaction can introduce unexpected latencies for applications that don't even have strong requirements for contiguous allocations.

Quoted 6 lines

Could you expand on this a bit please? Gfp flags allow to express how much the allocator try and compact for a high order allocations. Hugetlb allocations tend to require retrying and heavy compaction to succeed and the success rate tends to be pretty high from my experience. Why that is not case in your case?

Yes, I have the same observation: with GFP_TRANSHUGE | __GFP_RETRY_MAYFAIL I get very good success rate (~90% of free RAM allocated as hugepages). However, what I'm trying to point out is that this high success rate comes with high allocation latencies (90th percentile latency of 2206us). On the same system, the same high-order allocations which hit the fast path have latency <5us.

Quoted 14 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

It is also hard to efficiently determine if the current system state can be easily compacted due to mixing of unmovable memory. Due to these reasons, automatic background compaction by the kernel itself is hard to get right in a way which does not hurt unsuspecting

Quoted 8 lines

Quoted 5 lines

applications or waste CPU cycles.

We do trigger background compaction on a high order pressure from the page allocator by waking up kcompactd. Why is that not sufficient?

Whenever kcompactd is woken up, it does just enough work to create one free page of the given order (compaction_control.order) or higher.

Quoted 9 lines

This is an implementation detail IMHO. I am pretty sure we can do a better auto tuning when there is an indication of a constant flow of high order requests. This is no different from the memory reclaim in principle. Just because the kswapd autotuning not fitting with your particular workload you wouldn't want to export direct reclaim functionality and call it from a random module. That is just doomed to fail because different subsystems in control just leads to decisions going against each other.

I don't want to go the route of adding any auto-tuning/perdiction code to control compaction in the kernel. I'm more inclined towards extending existing interfaces to allow compaction behavior to be controlled either from userspace or a kernel driver. Letting a random module control compaction or a root process pumping new tunables from sysfs is the same in principle.

This patch is in the spirit of simple extension to existing compaction_zone_order() which allows either a kernel driver or userspace (through sysfs) to control compaction.

Also, we should avoid driving hard parallels between reclaim and
compaction: the former is often necessary for forward progress while the latter is often an optimization. Since contiguous allocations are mostly optimizations it's good to expose hooks from the kernel that let user (through a driver or userspace) control it using its own heuristics.

I thought hard about whats lacking in current userspace interface (sysfs):
- /proc/sys/vm/compact_memory: full system compaction is not an option as a viable pro-active compaction strategy.

- possibly expose [low, high] threshold values for each node and let kcompactd act on them. This was my approach for my original patch I linked earlier. Problem here is that it introduces too many tunables.

Considering the above, I came up with this callback approach which make it trivial to introduce user specific policies for compaction. It puts the onus of system stability, responsive in the hands of user without burdening admins with more tunables or adding crystal balls to kernel.

Quoted 15 lines

Quoted 6 lines

Such a design causes very high latency for workloads where we want to allocate lots of hugepages in short period of time. With pro-active compaction we can hide much of this latency. For some more background discussion and data, please see this thread:

https://patchwork.kernel.org/patch/11098289/

I am aware of that thread. And there are two things. You claim the allocation success rate is unnecessarily lower and that the direct latency is high. You simply cannot assume both low latency and high success rate. Compaction is not free. Somebody has to do the work. Hiding it into the background means that you are eating a lot of cycles from everybody else (think of a workload running in a restricted cpu controller just doing a lot of work in an unaccounted context).

Quoted 7 lines

That being said you really have to be prepared to pay a price for precious resource like high order pages.

On the other hand I do understand that high latency is not really desired for a more optimistic allocation requests with a reasonable fallback strategy. Those would benefit from kcompactd not giving up too early.

Doing pro-active compaction in background has merits in reducing reducing high-order alloc latency. Its true that it would end up burning cycles with little benefit in some cases. Its upto the user of this new interface to back off if it detects such a case.

Quoted 20 lines

Quoted 14 lines

Quoted 7 lines

Quoted 6 lines

Even with these caveats, pro-active compaction can still be very useful in certain scenarios to reduce hugepage allocation latencies. This callback interface allows drivers to drive compaction based on their own policies like the current level of external fragmentation for a particular order, system load etc.

Quoted 6 lines

So we do not trust the core MM to make a reasonable decision while we give
a free ticket to modules. How does this make any sense at all? How is a random module going to make a more informed decision when it has less visibility on the overal MM situation.

Quoted 3 lines

Embedding any specific policy (like: keep external fragmentation for order-9
between 30-40%) within MM core looks like a bad idea.

Agreed

Quoted 9 lines

Quoted 5 lines

As a driver, we
can easily measure parameters like system load, current fragmentation level
for any order in any zone etc. to make an informed decision. See the thread I refereed above for more background discussion.

Do that from the userspace then. If there is an insufficient interface to do that then let's talk about what is missing.

Currently we only have a proc interface to do full system compaction. Here's what missing from this interface: ability to set per-node, per-zone, per-order, [low, high] extfrag thresholds. This is what I exposed in my earlier patch titled 'proactive compaction'. Discussion there made me realize these are too many tunables and any admin would always get them wrong. Even if intended user of these sysfs node is some monitoring daemon, its tempting to mess with them.

With a callback extension to compact_zone_order() implementing any of the per-node, per-zone, per-order limits is straightforward and if needed the driver can expose debugfs/sysfs nodes if needed at all. (nvcompact.c driver[1] exposes these tunables as debugfs nodes, for example).

[1] https://gitlab.com/nigupta/linux/snippets/1894161

Quoted 12 lines

Quoted 11 lines

Quoted 6 lines

If you need to control compaction from the userspace you have an interface
for that. It is also completely unexplained why you need a completion callback.

/proc/sys/vm/compact_memory does whole system compaction which is often too much as a pro-active compaction strategy. To get more control over how to compaction work to do, I have added a compaction callback which controls how much work is done in one compaction cycle.

Quoted 7 lines

Why is a more fine grained control really needed? Sure compacting everything is heavy weight but how often do you have to do that. Your changelog starts with a usecase when there is a high demand for large pages at the startup. What prevents you do compaction at that time. If the workload is longterm then the initial price should just pay back, no?

Compacting all NUMA nodes is not practical on large systems in response to, say, launching a DB process on a certain node. Also, the frequency of hugepage allocation burts may be completely unpredictable. That's why background compaction can keep extfrag in check, say while system is lightly loaded (adhoc policy), keeping high-order allocation latencies low whenever the burst shows up.

- Nitin

Hi Nitin,

Quoted 27 lines · Nitin Gupta

Quoted 26 lines · Michal Hocko

On Tue 10-09-19 22:27:53, Nitin Gupta wrote:
[...]

Quoted 23 lines

Quoted 23 lines · Nitin Gupta

Quoted 22 lines

For some applications we need to allocate almost all memory as hugepages.
However, on a running system, higher order allocations can fail if the
memory is fragmented. Linux kernel currently does on-demand compaction
as we request more hugepages but this style of compaction incurs very
high latency. Experiments with one-time full memory compaction (followed by hugepage allocations) shows that kernel is able to restore a highly fragmented memory state to a fairly compacted memory
state within <1 sec for a 32G system. Such data suggests that a more
proactive compaction can help us allocate a large fraction of memory
as hugepages keeping allocation latencies low.

In general, compaction can introduce unexpected latencies for applications that don't even have strong requirements for contiguous
allocations.

Quoted 6 lines

Quoted 6 lines

Could you expand on this a bit please? Gfp flags allow to express how much the allocator try and compact for a high order allocations. Hugetlb allocations tend to require retrying and heavy compaction to succeed and the success rate tends to be pretty high from my experience. Why that is not case in your case?

The link to the driver you send on gitlab is not working :(

Quoted 34 lines

Yes, I have the same observation: with GFP_TRANSHUGE | __GFP_RETRY_MAYFAIL I get very good success rate (~90% of free RAM allocated as hugepages). However, what I'm trying to point out is that this
high success rate comes with high allocation latencies (90th percentile latency of 2206us). On the same system, the same high-order allocations which hit the fast path have latency <5us.

Quoted 16 lines

Quoted 8 lines

Quoted 8 lines

Quoted 6 lines

It is also hard to efficiently determine if the current system state can be easily compacted due to mixing of unmovable memory. Due to these reasons, automatic background compaction by the
kernel itself is hard to get right in a way which does not hurt unsuspecting

applications or waste CPU cycles.

Quoted 7 lines

Quoted 4 lines

We do trigger background compaction on a high order pressure from the
page allocator by waking up kcompactd. Why is that not sufficient?

Whenever kcompactd is woken up, it does just enough work to create one free page of the given order (compaction_control.order) or higher.

Quoted 9 lines

This is an implementation detail IMHO. I am pretty sure we can do a better auto tuning when there is an indication of a constant flow of high order requests. This is no different from the memory reclaim in principle. Just because the kswapd autotuning not fitting with your particular workload you wouldn't want to export direct reclaim functionality and call it from a random module. That is just doomed to fail because different subsystems in control just leads to decisions going against each other.

Quoted 7 lines

I don't want to go the route of adding any auto-tuning/perdiction code to control compaction in the kernel. I'm more inclined towards extending existing interfaces to allow compaction behavior to be controlled either from userspace or a kernel driver. Letting a random module control compaction or a root process pumping new tunables from sysfs is the same in
principle.

Do you think a kernel module and root user process have the same privileges? You can only export so much info to sysfs to use? Also wouldn't this introduce more tunables, per driver tunables to be more specific?

Quoted 9 lines

This patch is in the spirit of simple extension to existing compaction_zone_order() which allows either a kernel driver or userspace (through sysfs) to control compaction.

Also, we should avoid driving hard parallels between reclaim and
compaction: the former is often necessary for forward progress while the latter is often an optimization. Since contiguous allocations are mostly optimizations it's good to expose hooks from the kernel that let user (through a driver or userspace) control it using its own heuristics.

How is compaction an optimization? If I have a memory zone which has memory pages more than zone_highwmark and if higher order allocations a re failing because the memory is awfully fragmented, We need compaction to furthur progress here. I have seen workloads where kswapd won't help in progressing furthur because memory is so awfully fragmented. The workload I am quoting is the thpscale_workload from Mel Gorman's mmtests workloads.

Quoted 3 lines

I thought hard about whats lacking in current userspace interface (sysfs):
- /proc/sys/vm/compact_memory: full system compaction is not an option as a viable pro-active compaction strategy.

Don't we have a sysfs interface to compact memory per node through /sys/devices/system/node/node<node_number>/compact? CONFIG_SYSFS AND CONFIG_NUMA are enabled on a lot of systems? Why are we not talking about this?
I don't think kcompactd can go finer grain than per node. per-zone is an option but then that would be overkill I feel.

Quoted 4 lines

- possibly expose [low, high] threshold values for each node and let kcompactd act on them. This was my approach for my original patch I linked earlier. Problem here is that it introduces too many tunables.

Quoted 5 lines

Considering the above, I came up with this callback approach which make it trivial to introduce user specific policies for compaction. It puts the onus of system stability, responsive in the hands of user without burdening
admins with more tunables or adding crystal balls to kernel.

I have the same question as Michal, that is won't this cause conflicts among different subsystems? If you did answer it in your previous mails, could you point to as I may have missed it :)

Quoted 23 lines

Quoted 22 lines

Quoted 6 lines

Such a design causes very high latency for workloads where we want to allocate lots of hugepages in short period of time. With pro-active compaction we can hide much of this latency. For some more background discussion and data, please see this thread:

https://patchwork.kernel.org/patch/11098289/

I am aware of that thread. And there are two things. You claim the allocation success rate is unnecessarily lower and that the direct latency is high. You simply cannot assume both low latency and high success rate. Compaction is not free. Somebody has to do the work. Hiding it into the background means that you are eating a lot of cycles from everybody else (think of a workload running in a restricted cpu controller just doing a lot of work in an unaccounted context).

That being said you really have to be prepared to pay a price for precious resource like high order pages.

On the other hand I do understand that high latency is not really desired for a more optimistic allocation requests with a reasonable fallback strategy. Those would benefit from kcompactd not giving up too early.

Quoted 5 lines

Doing pro-active compaction in background has merits in reducing reducing high-order alloc latency. Its true that it would end up burning cycles with
little benefit in some cases. Its upto the user of this new interface to back off if it detects such a case.

Are these cycles worth considering in the big picture of reducing high order allocation latency?

Quoted 47 lines

Quoted 26 lines

Quoted 20 lines

Quoted 10 lines

Quoted 9 lines

Even with these caveats, pro-active compaction can still be very useful in certain scenarios to reduce hugepage allocation latencies.
This callback interface allows drivers to drive compaction based on
their own policies like the current level of external fragmentation
for a particular order, system load etc.

Quoted 9 lines

So we do not trust the core MM to make a reasonable decision while we
give
a free ticket to modules. How does this make any sense at all? How is a
random module going to make a more informed decision when it has less
visibility on the overal MM situation.

Quoted 3 lines

Embedding any specific policy (like: keep external fragmentation for order-9
between 30-40%) within MM core looks like a bad idea.

Agreed

Quoted 9 lines

Quoted 5 lines

As a driver, we
can easily measure parameters like system load, current fragmentation level
for any order in any zone etc. to make an informed decision. See the thread I refereed above for more background discussion.

Do that from the userspace then. If there is an insufficient interface to do that then let's talk about what is missing.

Currently we only have a proc interface to do full system compaction. Here's what missing from this interface: ability to set per-node, per- zone,
per-order, [low, high] extfrag thresholds. This is what I exposed in my earlier patch titled 'proactive compaction'. Discussion there made me realize
these are too many tunables and any admin would always get them wrong. Even
if intended user of these sysfs node is some monitoring daemon, its tempting to mess with them.

Quoted 6 lines

With a callback extension to compact_zone_order() implementing any of the per-node, per-zone, per-order limits is straightforward and if needed the driver can expose debugfs/sysfs nodes if needed at all. (nvcompact.c driver[1] exposes these tunables as debugfs nodes, for example).

[1] https://gitlab.com/nigupta/linux/snippets/1894161

Now, your proposing a system where we have interfaces from each driver. That could be more confusing for a sys admin to configure I feel?

But what your proposing really made me think about what kind of tunables do we want? Rather than just talking about tunables from the mm subsystem, can we introduce tunables that indicate the behaviour of workloads? Using this information from the user, we can look to optimize reclaim and compaction for that workload.
If we have a tunable which can indicate that the kernel is running in an environment where the where the workload will be performing a lot of higher order allocations, we can improve memory reclaim and compaction considering these parameters. One optimization I can think of extending kcompactd to compact more memory when a higher order allocation fails.

One of the biggest issues with having a discussion on proactive reclaim/compaction is that the workloads are really unpredictable. Rather than working on tunables from the mm subsystem which help us take action on memory pressure, can we talk about interfaces to hint about workloads so that we can make better informed decisions in the mm subsystem rather than involving other drivers?