mm: thp: Redefine default THP defrag behaviour disable it by default

This patch only makes sense on mmotm because it's heavily relying on an existing swapping-related fix and indirectly relying on the kcompactd patches. Even though the kernel says "4.4.0", the swapping and kcompactd patches have been cherry-picked from mmotm for the purposes of testing.

THP defrag is enabled by default to direct reclaim/compact but not wake kswapd in the event of a THP allocation failure. The problem is that THP allocation requests potentially enter reclaim/compaction. This potentially incurs a severe stall that is not guaranteed to be offset by reduced TLB misses. While there has been considerable effort to reduce the impact of reclaim/compaction, it is still a high cost and workloads that should fit in memory fail to do so. Specifically, a simple anon/file streaming workload will enter direct reclaim on NUMA at least even though the working set size is 80% of RAM. It's been years and it's time to throw in the towel.

First, this patch redefines what THP defrag means;

o GFP_TRANSHUGE by default will neither reclaim/compact nor wake kswapd
o For faults, defrag will not direct/reclaim but will wake kswapd
o For khugepaged, defrag will enter direct/reclaim but not wake kswapd

This means that a THP fault will no longer stall but may incur reclaim/compaction via kswapd reclaiming and kcompactd compacting. This is potentially destructive so the patch disables THP defrag by default. THP defrag for khugepaged remains enabled and will enter direct/reclaim but no wakeup kswapd or kcompactd.

After this patch a THP allocation failure will quickly fallback and rely on khugepaged to recover the situation at some time in the future. In some cases, this will reduce THP usage but the benefit of THP is hard to measure and not a universal win where as a stall to reclaim/compaction is definitely measurable and can be painful.

The first test for this is using "usemem" to read a large file and write a large anonymous mapping (to avoid the zero page) multiple times. The total size of the mappings is 80% of RAM and the benchmark simply measures how long it takes to complete. It uses multiple threads to see if that is a factor. On UMA, the performance is almost identical so is not reported but on NUMA, we see this

usemem

                                   4.4.0                 4.4.0
                          kcompactd-v1r1         nodefrag-v1r3
Amean    System-1       102.86 (  0.00%)       46.81 ( 54.50%)
Amean    System-4        37.85 (  0.00%)       34.02 ( 10.12%)
Amean    System-7        48.12 (  0.00%)       46.89 (  2.56%)
Amean    System-12       51.98 (  0.00%)       56.96 ( -9.57%)
Amean    System-21       80.16 (  0.00%)       79.05 (  1.39%)
Amean    System-30      110.71 (  0.00%)      107.17 (  3.20%)
Amean    System-48      127.98 (  0.00%)      124.83 (  2.46%)
Amean    Elapsd-1       185.84 (  0.00%)      105.51 ( 43.23%)
Amean    Elapsd-4        26.19 (  0.00%)       25.58 (  2.33%)
Amean    Elapsd-7        21.65 (  0.00%)       21.62 (  0.16%)
Amean    Elapsd-12       18.58 (  0.00%)       17.94 (  3.43%)
Amean    Elapsd-21       17.53 (  0.00%)       16.60 (  5.33%)
Amean    Elapsd-30       17.45 (  0.00%)       17.13 (  1.84%)
Amean    Elapsd-48       15.40 (  0.00%)       15.27 (  0.82%)

For a single thread, the benchmark completes 43.23% faster with this patch applied with smaller benefits as the thread increases. Similar, notice the large reduction in most cases in system CPU usage. The overall CPU time is

               4.4.0       4.4.0
        kcompactd-v1r1 nodefrag-v1r3
User        10357.65    10438.33
System       3988.88     3543.94
Elapsed      2203.01     1634.41

Which is substantial. Now, the reclaim figures

                                 4.4.0       4.4.0
                          kcompactd-v1r1nodefrag-v1r3
Minor Faults                 128458477   278352931
Major Faults                   2174976         225
Swap Ins                      16904701           0
Swap Outs                     17359627           0
Allocation stalls                43611           0
DMA allocs                           0           0
DMA32 allocs                  19832646    19448017
Normal allocs                614488453   580941839
Movable allocs                       0           0
Direct pages scanned          24163800           0
Kswapd pages scanned                 0           0
Kswapd pages reclaimed               0           0
Direct pages reclaimed        20691346           0
Compaction stalls                42263           0
Compaction success                 938           0
Compaction failures              41325           0

This patch eliminates almost all swapping and direct reclaim activity. There is still overhead but it's from NUMA balancing which does not identify that it's pointless trying to do anything with this workload.

I also tried the thpscale benchmark which forces a corner case where compaction can be used heavily and measures the latency of whether base or huge pages were used

thpscale Fault Latencies

                                       4.4.0                 4.4.0
                              kcompactd-v1r1         nodefrag-v1r3
Amean    fault-base-1      5288.84 (  0.00%)     2817.12 ( 46.73%)
Amean    fault-base-3      6365.53 (  0.00%)     3499.11 ( 45.03%)
Amean    fault-base-5      6526.19 (  0.00%)     4363.06 ( 33.15%)
Amean    fault-base-7      7142.25 (  0.00%)     4858.08 ( 31.98%)
Amean    fault-base-12    13827.64 (  0.00%)    10292.11 ( 25.57%)
Amean    fault-base-18    18235.07 (  0.00%)    13788.84 ( 24.38%)
Amean    fault-base-24    21597.80 (  0.00%)    24388.03 (-12.92%)
Amean    fault-base-30    26754.15 (  0.00%)    19700.55 ( 26.36%)
Amean    fault-base-32    26784.94 (  0.00%)    19513.57 ( 27.15%)
Amean    fault-huge-1      4223.96 (  0.00%)     2178.57 ( 48.42%)
Amean    fault-huge-3      2194.77 (  0.00%)     2149.74 (  2.05%)
Amean    fault-huge-5      2569.60 (  0.00%)     2346.95 (  8.66%)
Amean    fault-huge-7      3612.69 (  0.00%)     2997.70 ( 17.02%)
Amean    fault-huge-12     3301.75 (  0.00%)     6727.02 (-103.74%)
Amean    fault-huge-18     6696.47 (  0.00%)     6685.72 (  0.16%)
Amean    fault-huge-24     8000.72 (  0.00%)     9311.43 (-16.38%)
Amean    fault-huge-30    13305.55 (  0.00%)     9750.45 ( 26.72%)
Amean    fault-huge-32     9981.71 (  0.00%)    10316.06 ( -3.35%)

The average time to fault pages is substantially reduced in the majority of caseds but with the obvious caveat that fewer THPs are actually used in this adverse workload

                                   4.4.0                 4.4.0
                          kcompactd-v1r1         nodefrag-v1r3
Percentage huge-1         0.71 (  0.00%)       14.04 (1865.22%)
Percentage huge-3        10.77 (  0.00%)       33.05 (206.85%)
Percentage huge-5        60.39 (  0.00%)       38.51 (-36.23%)
Percentage huge-7        45.97 (  0.00%)       34.57 (-24.79%)
Percentage huge-12       68.12 (  0.00%)       40.07 (-41.17%)
Percentage huge-18       64.93 (  0.00%)       47.82 (-26.35%)
Percentage huge-24       62.69 (  0.00%)       44.23 (-29.44%)
Percentage huge-30       43.49 (  0.00%)       55.38 ( 27.34%)
Percentage huge-32       50.72 (  0.00%)       51.90 (  2.35%)

                                 4.4.0       4.4.0
                          kcompactd-v1r1nodefrag-v1r3
Minor Faults                  37429143    47564000
Major Faults                      1916        1558
Swap Ins                          1466        1079

Swap Outs 2936863 149626
Allocation stalls 62510 3
DMA allocs 0 0
DMA32 allocs 6566458 6401314
Normal allocs 216361697 216538171
Movable allocs 0 0
Direct pages scanned 25977580 17998
Kswapd pages scanned 0 3638931
Kswapd pages reclaimed 0 207236
Direct pages reclaimed 8833714 88

Compaction stalls               103349           5
Compaction success                 270           4
Compaction failures             103079           1

Note again that while this does swap as it's an aggressive workload, the direct relcim activity and allocation stalls is substantially reduced. There is some kswapd activity but ftrace showed that the kswapd activity was due to normal wakeups from 4K pages being allocated. Compaction-related stalls and activity are almost eliminated.

I also tried the stutter benchmark. For this, I do not have figures for NUMA but it's something that does impact UMA so I'll report what is available

stutter

                                 4.4.0                 4.4.0
                        kcompactd-v1r1         nodefrag-v1r3
Min         mmap      7.3571 (  0.00%)      7.3438 (  0.18%)
1st-qrtle   mmap      7.5278 (  0.00%)     17.9200 (-138.05%)
2nd-qrtle   mmap      7.6818 (  0.00%)     21.6055 (-181.25%)
3rd-qrtle   mmap     11.0889 (  0.00%)     21.8881 (-97.39%)
Max-90%     mmap     27.8978 (  0.00%)     22.1632 ( 20.56%)
Max-93%     mmap     28.3202 (  0.00%)     22.3044 ( 21.24%)
Max-95%     mmap     28.5600 (  0.00%)     22.4580 ( 21.37%)
Max-99%     mmap     29.6032 (  0.00%)     25.5216 ( 13.79%)
Max         mmap   4109.7289 (  0.00%)   4813.9832 (-17.14%)
Mean        mmap     12.4474 (  0.00%)     19.3027 (-55.07%)

This benchmark is trying to fault an anonymous mapping while there is a heavy IO load -- a scenario that desktop users used to complain about frequently. This shows a mix because the ideal case of mapping with THP is not hit as often. However, note that 99% of the mappings complete 13.79% faster. The CPU usage here is particularly interesting

               4.4.0       4.4.0
        kcompactd-v1r1nodefrag-v1r3

User 67.50 0.99
System 1327.88 91.30
Elapsed 2079.00 2128.98

And once again we look at the reclaim figures

                                 4.4.0       4.4.0
                          kcompactd-v1r1nodefrag-v1r3
Minor Faults                 335241922  1314582827
Major Faults                       715         819
Swap Ins                             0           0
Swap Outs                            0           0
Allocation stalls               532723           0
DMA allocs                           0           0
DMA32 allocs                1822364341  1177950222
Normal allocs               1815640808  1517844854
Movable allocs                       0           0
Direct pages scanned          21892772           0
Kswapd pages scanned          20015890    41879484
Kswapd pages reclaimed        19961986    41822072
Direct pages reclaimed        21892741           0
Compaction stalls              1065755           0
Compaction success                 514           0
Compaction failures            1065241           0

Allocation stalls and all direct reclaim activity is eliminated as well as compaction-related stalls.

THP gives impressive gains in some cases but only if they are quickly available. We're not going to reach the point where they are completely free so lets take the costs out of the fast paths finally and defer the cost to kswapd, kcompactd and khugepaged where it belongs.

---
 include/linux/gfp.h |  2 +-
 mm/huge_memory.c    | 26 ++++++++++++++++----------
 2 files changed, 17 insertions(+), 11 deletions(-)

index 8942af0813e3..e4a0287e5d0b 100644
@@ -248,7 +248,7 @@ struct vm_area_struct;
 #define GFP_HIGHUSER_MOVABLE	(GFP_HIGHUSER | __GFP_MOVABLE)
 #define GFP_TRANSHUGE	((GFP_HIGHUSER_MOVABLE | __GFP_COMP | \
 			 __GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN) & \
-			 ~__GFP_KSWAPD_RECLAIM)
+			 ~__GFP_RECLAIM)
 
 /* Convert GFP flags to their corresponding migrate type */
 #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE)

index 62fe06bb7d04..2708e9766e37 100644
@@ -46,7 +46,6 @@ unsigned long transparent_hugepage_flags __read_mostly =
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE_MADVISE
 	(1<<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG)|
 #endif
-	(1<<TRANSPARENT_HUGEPAGE_DEFRAG_FLAG)|
 	(1<<TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG)|
 	(1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG);
 

@@ -784,9 +783,17 @@ static int __do_huge_pmd_anonymous_page(struct mm_struct *mm,
 	return 0;
 }
 
-static inline gfp_t alloc_hugepage_gfpmask(int defrag, gfp_t extra_gfp)
+/* Defrag for allocation during fault will wake kswapd if necessary */
+static inline gfp_t alloc_hugepage_direct_gfpmask(struct vm_area_struct *vma)
 {
-	return (GFP_TRANSHUGE & ~(defrag ? 0 : __GFP_RECLAIM)) | extra_gfp;
+	bool defrag = transparent_hugepage_defrag(vma);
+	return GFP_TRANSHUGE | (defrag ? __GFP_KSWAPD_RECLAIM : 0);
+}
+
+/* Defrag for khugepaged will enter direct reclaim/compaction if necessary */
+static inline gfp_t alloc_hugepage_khugepaged_gfpmask(void)
+{
+	return GFP_TRANSHUGE | (khugepaged_defrag() ? __GFP_DIRECT_RECLAIM : 0);
 }
 
 /* Caller must hold page table lock. */

@@ -859,7 +866,7 @@ int do_huge_pmd_anonymous_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		}
 		return ret;
 	}
-	gfp = alloc_hugepage_gfpmask(transparent_hugepage_defrag(vma), 0);
+	gfp = alloc_hugepage_direct_gfpmask(vma);
 	page = alloc_hugepage_vma(gfp, vma, haddr, HPAGE_PMD_ORDER);
 	if (unlikely(!page)) {
 		count_vm_event(THP_FAULT_FALLBACK);

@@ -1185,7 +1192,7 @@ int do_huge_pmd_wp_page(struct mm_struct *mm, struct vm_area_struct *vma,
 alloc:
 	if (transparent_hugepage_enabled(vma) &&
 	    !transparent_hugepage_debug_cow()) {
-		huge_gfp = alloc_hugepage_gfpmask(transparent_hugepage_defrag(vma), 0);
+		huge_gfp = alloc_hugepage_direct_gfpmask(vma);
 		new_page = alloc_hugepage_vma(huge_gfp, vma, haddr, HPAGE_PMD_ORDER);
 	} else
 		new_page = NULL;

@@ -2440,9 +2447,9 @@ static int khugepaged_find_target_node(void)
 	return 0;
 }
 
-static inline struct page *alloc_hugepage(int defrag)
+static inline struct page *alloc_khugepaged_hugepage(void)
 {
-	return alloc_pages(alloc_hugepage_gfpmask(defrag, 0),
+	return alloc_pages(alloc_hugepage_khugepaged_gfpmask(),
 			   HPAGE_PMD_ORDER);
 }
 

@@ -2451,7 +2458,7 @@ static struct page *khugepaged_alloc_hugepage(bool *wait)
 	struct page *hpage;
 
 	do {
-		hpage = alloc_hugepage(khugepaged_defrag());
+		hpage = alloc_khugepaged_hugepage();
 		if (!hpage) {
 			count_vm_event(THP_COLLAPSE_ALLOC_FAILED);
 			if (!*wait)

@@ -2523,8 +2530,7 @@ static void collapse_huge_page(struct mm_struct *mm,
 	VM_BUG_ON(address & ~HPAGE_PMD_MASK);
 
 	/* Only allocate from the target node */
-	gfp = alloc_hugepage_gfpmask(khugepaged_defrag(), __GFP_OTHER_NODE) |
-		__GFP_THISNODE;
+	gfp = alloc_hugepage_khugepaged_gfpmask() | __GFP_OTHER_NODE | __GFP_THISNODE;
 
 	/* release the mmap_sem read lock. */
 	new_page = khugepaged_alloc_page(hpage, gfp, mm, address, node);

2.6.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I agree with your conclusions, but with the caveat that if we do not try to defragment memory for THP at fault time, mlocked programs might not have any opportunity at all to get transparent huge pages.

I wonder if we should consider mlock one of the slow paths where we should try to actually take the time to create THPs.

Also, we might consider doing THP collapse from the NUMA page migration opportunistically, if there is a free 2MB page available on the destination host.

Having said all that ...

--
All rights reversed

It would be a significant rework of mlock because it's not just mlocking memory, it's doing something similar to khugepaged and actively trying to collapse pages. I'm not against the idea as such but I'm not sure how much of a benefit it would be really.

While not necessarily a bad idea, it goes back to an old problem whereby there can be false sharing of NUMA pages within a THP boundary. Consider for example if threads are calculating 4K blocks and then it gets migrated as a THP including unrelated threads. It's not necessarily a win. We knew that THP false sharing was a potential problem at the start but never went much further than acknowleding it's a theoritical issue.

Thanks.

Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

It depends on the workload: with virtual machines THP is essential from the start without having to wait half a khugepaged cycle in average, especially on large systems. We see this effect for example in postcopy live migraiton where --postcopy-after-precopy is essential to reach peak performance during database workloads in guest, immediately after postcopy completes. With --postcopy-after-precopy only those pages that may be triggering userfaults will need to be collapsed with khugepaged and all the rest that was previously passed over with precopy has an high probability to be immediately THP backed also thanks to defrag/direct-compaction. Failing at starting the destination node largely THP backed is very visible in benchmark (even if a full precopy pass is done first). Later on the performance increases again as khugepaged fixes things, but it takes some time.

So unless we've a very good kcompatd or a workqueue doing the job of providing enough THP for page faults, I'm skeptical of this. If something I'd rather set defrag to "madvise" as qemu and other loads that critically need THP or they run literally half as slow (for example with enterprise workloads in the guest) will still be able not to rely solely on khugepaged which can takes some time to act. So your benchmark would run the same but the VM could still start fully THP backed.

Another problem is that khugepaged isn't able to collapse shared readonly anon pages, mostly because of the rmap complexities. I agree with Kirill we should be looking into how make this work, although I doubt the simpler refcounting is going to help much in this regard as the problem is in dealing with rmap, not so much with refcounts.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Which is a specialised case that does not apply to all users. Remember that the data showed that a basic streaming write of an anon mapping on a freshly booted NUMA system was enough to stall the process for long periods of time.

Even in the specialised case, a single VM reaching its peak performance may rely on getting THP but if that's at the cost of reclaiming other pages that may be hot to a second VM then it's an overall loss.

Finally, for the specialised case, if it really is that critical then pages could be freed preemptively from userspace before the VM starts. For example, allocate and free X hugetlbfs pages before the migration.

Right now, there are numerous tuning guides out there that are suggest disabling THP entirely due to the stalls. On my own desktop, I occasionally see a new process halt the system for a few seconds and it was possible to see that THP allocations were happening at the time.

If it's critical that the performance is identical then I would suggest a pre-migration step of alloc/free of hugetlbfs pages to force the defragmentation. Alternatively trigger compaction from proc and if necessary use memhog to allocate/free the required memory followed by a proc compaction. It's a little less tidy but it solves the corner case while leaving the common case free of stalls.

Unfortunately, it'll never be perfect. We went through a cycle of having really high success rates of allocations in 3.0 days and the cost in reclaim and disruption was way too high.

I think that's important but I'm not seeing right now how it's related to preventing processes stalling for long periods of time in direct reclaim and compaction.

Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

You're mixing the concern of that THP will use more memory with the cost of defragmentation. If you've memory issues and you are ok to sacrifice performance for swapping less you should disable THP, set it to never, and that's it.

The issues we're discussing here are about a system that isn't nearly in swap but with all memory fragmented or in dirty pagecache (which isn't too far from swapping from an I/O/writeback standpoint, but it's different than being low on memory: using more memory in anonymous THP memory because of THP won't move the needle in terms of the trouble dirty page causes to the VM unless you're in the corner case where that extra memory actually makes a difference, but if there's a streaming writer THP on or off won't make any significant difference).

In general VM (as in virtual machine) is a case where THP will not use any additional memory.

Good userland should just use MADV_HUGEPAGE, it should not be required to get root privilege to do such things by hand and try to defrag the system by hand. It'd be also overkill to do that, perhaps the app won't know exactly how many pages it really needs until the computation runs.

Here I'm not insisting to call compaction for all cases, I'd be ok with me if the default just relies on khugepaged, my problem is for those apps using MADV_HUGEPAGE that needs the THP immediately. You must live a way for the application to tell the kernel it is ok to take time to allocate the THP as long as it gets it and that's a fine semantic for MADV_HUGEPAGE.

khugepaged can take way more than a dozen minutes to pass over the address space, it's simply not ok if certain workloads would run half a slow despite they used MADV_HUGEPAGE and in turn they are telling the kernel "this is definitely a long lived allocation for a computation that needs THP, do everything you can to map THP here".

A "echo 3 >drop_caches; echo >compact_memory" is reasonably quick and if the allocations are contiguous and there's not much else churning the buddy over the NUMA zones where the app is running on, the total cost of direct compaction won't be very different from the above two commands. Slowing down a computation that used MADV_HUGEPAGE 50% to save the "echo 3 >drop_caches; echo >compact_memory" total runtime doesn't sound ok with me. It's also not ok with me that root is required if the app tries to fixup by hand and use hugetlbfs or /proc/sys/vm tricks to fix the layout of the buddy before starting.

It's not just qemu though, this is a tradeoff between short lived allocation or long lived allocation where the apps knows it's going to compute a lot. If the allocation is short lived there is a risk in doing direct compaction. if the allocation is long lived and the app notified the kernel with MADV_HUGEPAGE that it is going to run slow without THP, there is a risk in not doing direct compaction.

Let's first agree if direct compaction is going to hurt also for the MADV_HUGEPAGE case. I say MADV_HUGEPAGE benefits from direct compaction and is not hurt by not doing direct compaction. If you agree with this concept, I'd ask to change your patch, because your patch in turn is hurting MADV_HUGEPAGE users.

Providing a really lightweight compaction that won't stall, so it can be used always by direct reclaim can be done later anyway, that's a secondary concern, the primary concern is not to break MADV_HUEGPAGE.

In fact after this change I think you could make MADV_HUEGPAGE call compaction more aggressively as then we know we're not in the once-only short lived usage case were we risk only wasting CPU. I agree it's very hard to manage compaction for the average case were we have no clue if compaction is going to payoff or not, but for MADV_HUGEPAGE we know.

compaction done in the background only really can work with a reservation above the high watermark that can only be accessed by emergency allocations or THP allocations, i.e. you need to spend more RAM to make it work, like we spend RAM in the high-low wmark ranges to make kswapd work with hysteresis. If you leave the compacted pages in the buddy there's no way it can pay off as they'd be fragmented by background churn before the THP fault can get an hold on them. You should try again with a reservation.

If an app cares and forks and wants low TLB overhead it'd be nice if it could still be guarantee to get it with MADV_HUGEPAGE as that is a case khugepaged can't fix.

I'm still very skeptical about this patch, and the reason isn't desktop load but MADV_HUGEPAGE apps, for those I don't think this patch is ok.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

^^^ drop this not sorry for any confusion :)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

There are three cases

1. THP was allocated when the application only required 4K and consumes more memory. This has always been the case but not the concern here
2. Memory is fragmented but there are enough free pages. In this case, only compaction is required and the memory footprint is the same
3. Memory is fragmentation and pages have to be freed before compaction.

It's 3 I was referred to even though all the cases are important.

I want to get to the half-way point where THP is used if easily available without worrying that there will be stalls at some point in the future or requiring application modification for madvise. That's better than the all or nothing approach that users are currently faced with. I wince every time I see a tuning guide suggesting THP be disabled and have handled too many bugs where disabling THP was a workaround.

That said, you made a number of important points. I'm not going to respond to them individually because I believe I understand your concerns and now agree with them. I've prototyped a patch that modifies the defrag tunable as follows;

1. By default, "madvise" and direct reclaim/compaction for applications that specifically requested that behaviour. This will avoid breaking MADV_HUGEPAGE which you mentioned in a few places
2. "never" will never reclaim anything and was the default behaviour of version 1 but will not be the default in version 2.
3. "defer" will wake kswapd which will reclaim or wake kcompactd whichever is necessary. This is new but avoids stalls while helping khugepaged do its work quickly in the near future.
4. "always" will direct reclaim/compact just like todays behaviour

I'm testing it at the moment to make sure each of the options behave correctly.

Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hello Mel,

Defragging memory synchronously only under madvise is fine with me.

This is an kABI visible change, but it should be ok. I'm not aware of any program that parses that file and could get confused.

"defer" sounds an interesting default option if it could be made to work better.

I suspect there are a number of apps that took advantage of the "always" setting without realizing it, but we only could notice the ones that don't. In any case those apps can start to call MADV_HUGEPAGE if they don't already and that will provide a definitive fix. With this approach MADV_HUGEPAGE will provide the same reliability in allocation as before so there will be no problem then.

Thanks,
Andrea

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I think this is a sensible default though. As you pointed out, those applications specifically requested it and a delay *should* be acceptable. If not, then it's a one-liner to change the behaviour.

Neither am I but it'll be a wait and see approach unfortunately to see do I get the dreaded "you broke an ABI that applications depend upon" report.

I was tempted to set it but given that there was a host of reclaim-related bugs recently I backed off. For example, the last three releases has a serious bug whereby NUMA machines swapped heavily and no one reported it (or I missed it). There is still one excessive reclaiming bug open that has a potential patch that hasn't been tested so that's still an issue. I didn't want to muddy the waters further.

Agreed but in itself, it'll be interesting to see if anyone notices. With the new default, applications still get huge pages in a lot of cases. It'll be interesting to report if someone complains about long-term behaviour where THP utilisation is lower for periods of time until khugepaged recovers it.

Yes or else they set the tunable to always and carry on.

Yes.

As I believe your concerns have been addressed, can I get an ack on this patch?

Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Could you elaborate on problems with rmap? I have looked into this deeply yet.

Do you see anything what would prevent following basic scheme:

- Identify series of small pages as candidate for collapsing into a compound page. Not sure how difficult it would be. I guess it can be done by looking for adjacent pages which belong to the same anon_vma.

- Setup migration entries for pte which maps these pages.

- Collapse small pages into compound page. IIUC, it only will be possible if these pages are not pinned.

- Replace migration entries with ptes which point to subpages of the new compound page.

- Scan over all vmas mapping this compound page, looking for VMA suitable for huge page. We cannot collapse it right away due lock inversion of anon_vma->rwsem vs. mmap_sem.

- For found VMAs, collapse page table into PMD one VMA a time under down_write(mmap_sem).

Even if would fail to create any PMDs, we would reduce LRU pressure by collapsing small pages into compound one.

 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Just like if there was no other process sharing them yes.

I see how your new refcounting simplifies things as we don't have to do create hugepmds immediately, but we still have to modify all ptes of all sharers, not just those belonging to the vma we collapsed (or we'd be effectively copying-on-collapse in turn losing the sharing).

If we'd defer it and leave temporarily new THP and old 4k pages both allocated and independently mapped, a process running in the old ptes could gup_fast and a process in the new ptes could gup_fast too and we'd up with double memory usage, so we'd need a way to redirect gup_fast in the old pte to the new THP, so the future pins goes to the new THP always. Some new linkage between old ptes and new ptes would also be needed to keep walking it slowly and it shall be invalidated during COWs.

Doing it incrementally and not updating all ptes at once wouldn't be straightforward. Doing it not incrementally would mean paying the cost of updating (in the worst case) up to hundred thousand ptes at full CPU usage for a later gain we're not sure about. Said that I think it's worthy goal to achieve, especially if we remove compaction from direct reclaim.

Thanks,
Andrea

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

The cornercases Rik pointed out aside, if the mapping isn't long-lived enough that it can wait for khugepaged, what are the odds that the defrag work will be offset by the TLB savings? However, for mappings where it would pay off, having to do the same defrag work but doing it at a later time is actually a net loss. Should we consider keeping direct reclaim and compaction as a configurable option as least?

Regardless, this looks like much saner defaults than what we have.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yes, I think so. I've a prototype now that makes it configurable and am running some tests. I'll preserve your and Rik's ack in V2 as the patch will be different but the default behaviour will be very similar.

Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>