mm, thp: relax __GFP_THISNODE for MADV_HUGEPAGE mappings

From: Michal Hocko <[email protected]>

Andrea has noticed [1] that a THP allocation might be really disruptive when allocated on NUMA system with the local node full or hard to reclaim. Stefan has posted an allocation stall report on 4.12 based SLES kernel which suggests the same issue:

[245513.362669] kvm: page allocation stalls for 194572ms, order:9, mode:0x4740ca(__GFP_HIGHMEM|__GFP_IO|__GFP_FS|__GFP_COMP|__GFP_NOMEMALLOC|__GFP_HARDWALL|__GFP_THISNODE|__GFP_MOVABLE|__GFP_DIRECT_RECLAIM), nodemask=(null)
[245513.363983] kvm cpuset=/ mems_allowed=0-1
[245513.364604] CPU: 10 PID: 84752 Comm: kvm Tainted: G        W 4.12.0+98-ph <a href="/view.php?id=1" title="[geschlossen] Integration Ramdisk" class="resolved">0000001</a> SLE15 (unreleased)
[245513.365258] Hardware name: Supermicro SYS-1029P-WTRT/X11DDW-NT, BIOS 2.0 12/05/2017
[245513.365905] Call Trace:
[245513.366535]  dump_stack+0x5c/0x84
[245513.367148]  warn_alloc+0xe0/0x180
[245513.367769]  __alloc_pages_slowpath+0x820/0xc90
[245513.368406]  ? __slab_free+0xa9/0x2f0
[245513.369048]  ? __slab_free+0xa9/0x2f0
[245513.369671]  __alloc_pages_nodemask+0x1cc/0x210
[245513.370300]  alloc_pages_vma+0x1e5/0x280
[245513.370921]  do_huge_pmd_wp_page+0x83f/0xf00
[245513.371554]  ? set_huge_zero_page.isra.52.part.53+0x9b/0xb0
[245513.372184]  ? do_huge_pmd_anonymous_page+0x631/0x6d0
[245513.372812]  __handle_mm_fault+0x93d/0x1060
[245513.373439]  handle_mm_fault+0xc6/0x1b0
[245513.374042]  __do_page_fault+0x230/0x430
[245513.374679]  ? get_vtime_delta+0x13/0xb0
[245513.375411]  do_page_fault+0x2a/0x70
[245513.376145]  ? page_fault+0x65/0x80
[245513.376882]  page_fault+0x7b/0x80
[...]
[245513.382056] Mem-Info:
[245513.382634] active_anon:126315487 inactive_anon:1612476 isolated_anon:5
                 active_file:60183 inactive_file:245285 isolated_file:0
                 unevictable:15657 dirty:286 writeback:1 unstable:0
                 slab_reclaimable:75543 slab_unreclaimable:2509111
                 mapped:81814 shmem:31764 pagetables:370616 bounce:0
                 free:32294031 free_pcp:6233 free_cma:0
[245513.386615] Node 0 active_anon:254680388kB inactive_anon:1112760kB active_file:240648kB inactive_file:981168kB unevictable:13368kB isolated(anon):0kB isolated(file):0kB mapped:280240kB dirty:1144kB writeback:0kB shmem:95832kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 81225728kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[245513.388650] Node 1 active_anon:250583072kB inactive_anon:5337144kB active_file:84kB inactive_file:0kB unevictable:49260kB isolated(anon):20kB isolated(file):0kB mapped:47016kB dirty:0kB writeback:4kB shmem:31224kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 31897600kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no

The defrag mode is "madvise" and from the above report it is clear that the THP has been allocated for MADV_HUGEPAGA vma.

Andrea has identified that the main source of the problem is __GFP_THISNODE usage:

: The problem is that direct compaction combined with the NUMA : __GFP_THISNODE logic in mempolicy.c is telling reclaim to swap very : hard the local node, instead of failing the allocation if there's no : THP available in the local node.
:
: Such logic was ok until __GFP_THISNODE was added to the THP allocation : path even with MPOL_DEFAULT.
:
: The idea behind the __GFP_THISNODE addition, is that it is better to : provide local memory in PAGE_SIZE units than to use remote NUMA THP : backed memory. That largely depends on the remote latency though, on : threadrippers for example the overhead is relatively low in my : experience.
:
: The combination of __GFP_THISNODE and __GFP_DIRECT_RECLAIM results in : extremely slow qemu startup with vfio, if the VM is larger than the : size of one host NUMA node. This is because it will try very hard to : unsuccessfully swapout get_user_pages pinned pages as result of the : __GFP_THISNODE being set, instead of falling back to PAGE_SIZE : allocations and instead of trying to allocate THP on other nodes (it : would be even worse without vfio type1 GUP pins of course, except it'd : be swapping heavily instead).

Fix this by removing __GFP_THISNODE handling from alloc_pages_vma where it doesn't belong and move it to alloc_hugepage_direct_gfpmask where we juggle gfp flags for different allocation modes. The rationale is that __GFP_THISNODE is helpful in relaxed defrag modes because falling back to a different node might be more harmful than the benefit of a large page. If the user really requires THP (e.g. by MADV_HUGEPAGE) then the THP has a higher priority than local NUMA placement.

Be careful when the vma has an explicit numa binding though, because __GFP_THISNODE is not playing well with it. We want to follow the explicit numa policy rather than enforce a node which happens to be local to the cpu we are running on.

[1] http://lkml.kernel.org/r/[email protected]

Fixes 5265047ac301 ("mm, thp: really limit transparent hugepage allocation to local node")

Reported-by Stefan Priebe <[email protected]>

Debugged-by Andrea Arcangeli <[email protected]>

Tested-by Stefan Priebe <[email protected]>

Tested-by Zi Yan <[email protected]>

Signed-off-by Michal Hocko <[email protected]>

---

Hi,
this is a follow up for [1]. Anrea has proposed two approaches to solve the regression. This is an alternative implementation of the second approach [2]. The reason for an alternative approach is that I strongly believe that all the subtle THP gfp manipulation should be at a single place (alloc_hugepage_direct_gfpmask) rather than spread in multiple places with additional fixup. There is one notable difference to [2] and that is defrag=allways behavior where I am preserving the original behavior. The reason for that is that defrag=always has always had tendency to stall and reclaim and we have addressed that by defining a new default defrag mode. We can discuss this behavior later but I believe the default mode and a regression noticed by multiple users should be closed regardless. Hence this patch.

[2] http://lkml.kernel.org/r/[email protected]

 include/linux/mempolicy.h |  2 ++
 mm/huge_memory.c          | 26 ++++++++++++++++++--------
 mm/mempolicy.c            | 28 +---------------------------
 3 files changed, 21 insertions(+), 35 deletions(-)

include/linux/mempolicy.h +2 -0

index 5228c62af416..bac395f1d00a 100644
@@ -139,6 +139,8 @@ struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp,
 struct mempolicy *get_task_policy(struct task_struct *p);
 struct mempolicy *__get_vma_policy(struct vm_area_struct *vma,
 		unsigned long addr);
+struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
+						unsigned long addr);
 bool vma_policy_mof(struct vm_area_struct *vma);
 
 extern void numa_default_policy(void);

mm/huge_memory.c +18 -8

index c3bc7e9c9a2a..56c9aac4dc86 100644
@@ -629,21 +629,31 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
  *	    available
  * never: never stall for any thp allocation
  */
-static inline gfp_t alloc_hugepage_direct_gfpmask(struct vm_area_struct *vma)
+static inline gfp_t alloc_hugepage_direct_gfpmask(struct vm_area_struct *vma, unsigned long addr)
 {
 	const bool vma_madvised = !!(vma->vm_flags & VM_HUGEPAGE);
+	gfp_t this_node = 0;
+	struct mempolicy *pol;
+
+#ifdef CONFIG_NUMA
+	/* __GFP_THISNODE makes sense only if there is no explicit binding */
+	pol = get_vma_policy(vma, addr);
+	if (pol->mode != MPOL_BIND)
+		this_node = __GFP_THISNODE;
+	mpol_cond_put(pol);
+#endif
 
 	if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags))
-		return GFP_TRANSHUGE | (vma_madvised ? 0 : __GFP_NORETRY);
+		return GFP_TRANSHUGE | (vma_madvised ? 0 : __GFP_NORETRY | this_node);
 	if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags))
-		return GFP_TRANSHUGE_LIGHT | __GFP_KSWAPD_RECLAIM;
+		return GFP_TRANSHUGE_LIGHT | __GFP_KSWAPD_RECLAIM | this_node;
 	if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags))
 		return GFP_TRANSHUGE_LIGHT | (vma_madvised ? __GFP_DIRECT_RECLAIM :
-							     __GFP_KSWAPD_RECLAIM);
+							     __GFP_KSWAPD_RECLAIM | this_node);
 	if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags))
 		return GFP_TRANSHUGE_LIGHT | (vma_madvised ? __GFP_DIRECT_RECLAIM :
-							     0);
-	return GFP_TRANSHUGE_LIGHT;
+							     this_node);
+	return GFP_TRANSHUGE_LIGHT | this_node;
 }
 
 /* Caller must hold page table lock. */

@@ -715,7 +725,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
 			pte_free(vma->vm_mm, pgtable);
 		return ret;
 	}
-	gfp = alloc_hugepage_direct_gfpmask(vma);
+	gfp = alloc_hugepage_direct_gfpmask(vma, haddr);
 	page = alloc_hugepage_vma(gfp, vma, haddr, HPAGE_PMD_ORDER);
 	if (unlikely(!page)) {
 		count_vm_event(THP_FAULT_FALLBACK);

@@ -1290,7 +1300,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd)
 alloc:
 	if (transparent_hugepage_enabled(vma) &&
 	    !transparent_hugepage_debug_cow()) {
-		huge_gfp = alloc_hugepage_direct_gfpmask(vma);
+		huge_gfp = alloc_hugepage_direct_gfpmask(vma, haddr);
 		new_page = alloc_hugepage_vma(huge_gfp, vma, haddr, HPAGE_PMD_ORDER);
 	} else
 		new_page = NULL;

mm/mempolicy.c +1 -27

index da858f794eb6..75bbfc3d6233 100644
@@ -1648,7 +1648,7 @@ struct mempolicy *__get_vma_policy(struct vm_area_struct *vma,
  * freeing by another task.  It is the caller's responsibility to free the
  * extra reference for shared policies.
  */
-static struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
+struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
 						unsigned long addr)
 {
 	struct mempolicy *pol = __get_vma_policy(vma, addr);

@@ -2026,32 +2026,6 @@ alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma,
 		goto out;
 	}
 
-	if (unlikely(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && hugepage)) {
-		int hpage_node = node;
-
-		/*
-		 * For hugepage allocation and non-interleave policy which
-		 * allows the current node (or other explicitly preferred
-		 * node) we only try to allocate from the current/preferred
-		 * node and don't fall back to other nodes, as the cost of
-		 * remote accesses would likely offset THP benefits.
-		 *
-		 * If the policy is interleave, or does not allow the current
-		 * node in its nodemask, we allocate the standard way.
-		 */
-		if (pol->mode == MPOL_PREFERRED &&
-						!(pol->flags & MPOL_F_LOCAL))
-			hpage_node = pol->v.preferred_node;
-
-		nmask = policy_nodemask(gfp, pol);
-		if (!nmask || node_isset(hpage_node, *nmask)) {
-			mpol_cond_put(pol);
-			page = __alloc_pages_node(hpage_node,
-						gfp | __GFP_THISNODE, order);
-			goto out;
-		}
-	}
-
 	nmask = policy_nodemask(gfp, pol);
 	preferred_nid = policy_node(gfp, pol, node);
 	page = __alloc_pages_nodemask(gfp, order, preferred_nid, nmask);

[305146.987742] khugepaged: page allocation stalls for 224236ms, order:9,

mode:0x4740ca(__GFP_HIGHMEM|__GFP_IO|__GFP_FS|__GFP_COMP|__GFP_NOMEMALLOC|__GFP_HARDWALL|__GFP_THISNODE|__GFP_MOVABLE|__GFP_DIRECT_RECLAIM), nodemask=(null)

Hello,

whlie using this path i got another stall - which i never saw under kernel 4.4. Here is the trace:

[305111.932698] INFO: task ksmtuned:1399 blocked for more than 120 seconds.
[305111.933612]       Tainted: G                   4.12.0+105-ph #1
[305111.934456] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[305111.935323] ksmtuned        D    0  1399      1 0x00080000
[305111.936207] Call Trace:
[305111.937118]  ? __schedule+0x3bc/0x830
[305111.937991]  schedule+0x32/0x80
[305111.938837]  schedule_preempt_disabled+0xa/0x10
[305111.939687]  __mutex_lock.isra.4+0x287/0x4c0

From: Michal Hocko <[email protected]>

Andrea has noticed [1] that a THP allocation might be really disruptive when allocated on NUMA system with the local node full or hard to reclaim. Stefan has posted an allocation stall report on 4.12 based SLES kernel which suggests the same issue:

[245513.362669] kvm: page allocation stalls for 194572ms, order:9, mode:0x4740ca(__GFP_HIGHMEM|__GFP_IO|__GFP_FS|__GFP_COMP|__GFP_NOMEMALLOC|__GFP_HARDWALL|__GFP_THISNODE|__GFP_MOVABLE|__GFP_DIRECT_RECLAIM), nodemask=(null)
[245513.363983] kvm cpuset=/ mems_allowed=0-1
[245513.364604] CPU: 10 PID: 84752 Comm: kvm Tainted: G        W 4.12.0+98-ph <a href="/view.php?id=1" title="[geschlossen] Integration Ramdisk" class="resolved">0000001</a> SLE15 (unreleased)
[245513.365258] Hardware name: Supermicro SYS-1029P-WTRT/X11DDW-NT, BIOS 2.0 12/05/2017
[245513.365905] Call Trace:
[245513.366535]  dump_stack+0x5c/0x84
[245513.367148]  warn_alloc+0xe0/0x180
[245513.367769]  __alloc_pages_slowpath+0x820/0xc90
[245513.368406]  ? __slab_free+0xa9/0x2f0
[245513.369048]  ? __slab_free+0xa9/0x2f0
[245513.369671]  __alloc_pages_nodemask+0x1cc/0x210
[245513.370300]  alloc_pages_vma+0x1e5/0x280
[245513.370921]  do_huge_pmd_wp_page+0x83f/0xf00
[245513.371554]  ? set_huge_zero_page.isra.52.part.53+0x9b/0xb0

[245513.372184]  ? do_huge_pmd_anonymous_page+0x631/0x6d0
[245513.372812]  __handle_mm_fault+0x93d/0x1060
[245513.373439]  handle_mm_fault+0xc6/0x1b0
[245513.374042]  __do_page_fault+0x230/0x430
[245513.374679]  ? get_vtime_delta+0x13/0xb0
[245513.375411]  do_page_fault+0x2a/0x70
[245513.376145]  ? page_fault+0x65/0x80
[245513.376882]  page_fault+0x7b/0x80

[...]

[245513.382056] Mem-Info:
[245513.382634] active_anon:126315487 inactive_anon:1612476 isolated_anon:5
                 active_file:60183 inactive_file:245285 isolated_file:0
                 unevictable:15657 dirty:286 writeback:1 unstable:0
                 slab_reclaimable:75543 slab_unreclaimable:2509111
                 mapped:81814 shmem:31764 pagetables:370616 bounce:0
                 free:32294031 free_pcp:6233 free_cma:0
[245513.386615] Node 0 active_anon:254680388kB inactive_anon:1112760kB active_file:240648kB inactive_file:981168kB unevictable:13368kB isolated(anon):0kB isolated(file):0kB mapped:280240kB dirty:1144kB writeback:0kB shmem:95832kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 81225728kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[245513.388650] Node 1 active_anon:250583072kB inactive_anon:5337144kB active_file:84kB inactive_file:0kB unevictable:49260kB isolated(anon):20kB isolated(file):0kB mapped:47016kB dirty:0kB writeback:4kB shmem:31224kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 31897600kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no

The defrag mode is "madvise" and from the above report it is clear that the THP has been allocated for MADV_HUGEPAGA vma.

Andrea has identified that the main source of the problem is __GFP_THISNODE usage:

: The problem is that direct compaction combined with the NUMA : __GFP_THISNODE logic in mempolicy.c is telling reclaim to swap very : hard the local node, instead of failing the allocation if there's no : THP available in the local node.
:
: Such logic was ok until __GFP_THISNODE was added to the THP allocation : path even with MPOL_DEFAULT.
:
: The idea behind the __GFP_THISNODE addition, is that it is better to : provide local memory in PAGE_SIZE units than to use remote NUMA THP : backed memory. That largely depends on the remote latency though, on : threadrippers for example the overhead is relatively low in my : experience.
:
: The combination of __GFP_THISNODE and __GFP_DIRECT_RECLAIM results in : extremely slow qemu startup with vfio, if the VM is larger than the : size of one host NUMA node. This is because it will try very hard to : unsuccessfully swapout get_user_pages pinned pages as result of the : __GFP_THISNODE being set, instead of falling back to PAGE_SIZE : allocations and instead of trying to allocate THP on other nodes (it : would be even worse without vfio type1 GUP pins of course, except it'd : be swapping heavily instead).

Fix this by removing __GFP_THISNODE handling from alloc_pages_vma where it doesn't belong and move it to alloc_hugepage_direct_gfpmask where we juggle gfp flags for different allocation modes. The rationale is that __GFP_THISNODE is helpful in relaxed defrag modes because falling back to a different node might be more harmful than the benefit of a large page. If the user really requires THP (e.g. by MADV_HUGEPAGE) then the THP has a higher priority than local NUMA placement.

Quoted 21 lines

From: Michal Hocko <[email protected]>

Andrea has noticed [1] that a THP allocation might be really disruptive when allocated on NUMA system with the local node full or hard to reclaim. Stefan has posted an allocation stall report on 4.12 based SLES kernel which suggests the same issue:

[245513.362669] kvm: page allocation stalls for 194572ms, order:9, mode:0x4740ca(__GFP_HIGHMEM|__GFP_IO|__GFP_FS|__GFP_COMP|__GFP_NOMEMALLOC|__GFP_HARDWALL|__GFP_THISNODE|__GFP_MOVABLE|__GFP_DIRECT_RECLAIM), nodemask=(null)
[245513.363983] kvm cpuset=/ mems_allowed=0-1
[245513.364604] CPU: 10 PID: 84752 Comm: kvm Tainted: G        W 4.12.0+98-ph <a href="/view.php?id=1" title="[geschlossen] Integration Ramdisk" class="resolved">0000001</a> SLE15 (unreleased)
[245513.365258] Hardware name: Supermicro SYS-1029P-WTRT/X11DDW-NT, BIOS 2.0 12/05/2017
[245513.365905] Call Trace:
[245513.366535]  dump_stack+0x5c/0x84
[245513.367148]  warn_alloc+0xe0/0x180
[245513.367769]  __alloc_pages_slowpath+0x820/0xc90
[245513.368406]  ? __slab_free+0xa9/0x2f0
[245513.369048]  ? __slab_free+0xa9/0x2f0
[245513.369671]  __alloc_pages_nodemask+0x1cc/0x210
[245513.370300]  alloc_pages_vma+0x1e5/0x280
[245513.370921]  do_huge_pmd_wp_page+0x83f/0xf00
[245513.371554]  ? set_huge_zero_page.isra.52.part.53+0x9b/0xb0

Quoted 8 lines

[245513.372184]  ? do_huge_pmd_anonymous_page+0x631/0x6d0
[245513.372812]  __handle_mm_fault+0x93d/0x1060
[245513.373439]  handle_mm_fault+0xc6/0x1b0
[245513.374042]  __do_page_fault+0x230/0x430
[245513.374679]  ? get_vtime_delta+0x13/0xb0
[245513.375411]  do_page_fault+0x2a/0x70
[245513.376145]  ? page_fault+0x65/0x80
[245513.376882]  page_fault+0x7b/0x80

Since we don't have __GFP_REPEAT, this suggests that __alloc_pages_direct_compact() took >100s to complete. The memory capacity of the system isn't shown, but I assume it's around 768GB? This should be with COMPACT_PRIO_ASYNC, and MIGRATE_ASYNC compaction certainly should abort much earlier.

Quoted 39 lines

[245513.382056] Mem-Info:
[245513.382634] active_anon:126315487 inactive_anon:1612476 isolated_anon:5
                 active_file:60183 inactive_file:245285 isolated_file:0
                 unevictable:15657 dirty:286 writeback:1 unstable:0
                 slab_reclaimable:75543 slab_unreclaimable:2509111
                 mapped:81814 shmem:31764 pagetables:370616 bounce:0
                 free:32294031 free_pcp:6233 free_cma:0
[245513.386615] Node 0 active_anon:254680388kB inactive_anon:1112760kB active_file:240648kB inactive_file:981168kB unevictable:13368kB isolated(anon):0kB isolated(file):0kB mapped:280240kB dirty:1144kB writeback:0kB shmem:95832kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 81225728kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[245513.388650] Node 1 active_anon:250583072kB inactive_anon:5337144kB active_file:84kB inactive_file:0kB unevictable:49260kB isolated(anon):20kB isolated(file):0kB mapped:47016kB dirty:0kB writeback:4kB shmem:31224kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 31897600kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no

The defrag mode is "madvise" and from the above report it is clear that the THP has been allocated for MADV_HUGEPAGA vma.

Andrea has identified that the main source of the problem is __GFP_THISNODE usage:

: The problem is that direct compaction combined with the NUMA : __GFP_THISNODE logic in mempolicy.c is telling reclaim to swap very : hard the local node, instead of failing the allocation if there's no : THP available in the local node.
:
: Such logic was ok until __GFP_THISNODE was added to the THP allocation : path even with MPOL_DEFAULT.
:
: The idea behind the __GFP_THISNODE addition, is that it is better to : provide local memory in PAGE_SIZE units than to use remote NUMA THP : backed memory. That largely depends on the remote latency though, on : threadrippers for example the overhead is relatively low in my : experience.
:
: The combination of __GFP_THISNODE and __GFP_DIRECT_RECLAIM results in : extremely slow qemu startup with vfio, if the VM is larger than the : size of one host NUMA node. This is because it will try very hard to : unsuccessfully swapout get_user_pages pinned pages as result of the : __GFP_THISNODE being set, instead of falling back to PAGE_SIZE : allocations and instead of trying to allocate THP on other nodes (it : would be even worse without vfio type1 GUP pins of course, except it'd : be swapping heavily instead).

Quoted 8 lines

Fix this by removing __GFP_THISNODE handling from alloc_pages_vma where it doesn't belong and move it to alloc_hugepage_direct_gfpmask where we juggle gfp flags for different allocation modes. The rationale is that __GFP_THISNODE is helpful in relaxed defrag modes because falling back to a different node might be more harmful than the benefit of a large page. If the user really requires THP (e.g. by MADV_HUGEPAGE) then the THP has a higher priority than local NUMA placement.

That's not entirely true, the remote access latency for remote thp on all of our platforms is greater than local small pages, this is especially true for remote thp that is allocated intersocket and must be accessed through the interconnect.

Our users of MADV_HUGEPAGE are ok with assuming the burden of increased allocation latency, but certainly not remote access latency. There are users who remap their text segment onto transparent hugepages are fine with startup delay if they are access all of their text from local thp. Remote thp would be a significant performance degradation.

When Andrea brought this up, I suggested that the full solution would be a MPOL_F_HUGEPAGE flag that could define thp allocation policy -- the added benefit is that we could replace the thp "defrag" mode default by setting this as part of default_policy. Right now, MADV_HUGEPAGE users are concerned about (1) getting thp when system-wide it is not default and (2) additional fault latency when direct compaction is not default. They are not anticipating the degradation of remote access latency, so overloading the meaning of the mode is probably not a good idea.

When Andrea brought this up, I suggested that the full solution would be a MPOL_F_HUGEPAGE flag that could define thp allocation policy -- the added

benefit is that we could replace the thp "defrag" mode default by setting this as part of default_policy. Right now, MADV_HUGEPAGE users are concerned about (1) getting thp when system-wide it is not default and (2) additional fault latency when direct compaction is not default. They are not anticipating the degradation of remote access latency, so overloading the meaning of the mode is probably not a good idea.

On Fri, 7 Sep 2018, Michal Hocko wrote:

Quoted 8 lines

Fix this by removing __GFP_THISNODE handling from alloc_pages_vma where it doesn't belong and move it to alloc_hugepage_direct_gfpmask where we juggle gfp flags for different allocation modes. The rationale is that __GFP_THISNODE is helpful in relaxed defrag modes because falling back to a different node might be more harmful than the benefit of a large page. If the user really requires THP (e.g. by MADV_HUGEPAGE) then the THP has a higher priority than local NUMA placement.

That's not entirely true, the remote access latency for remote thp on all of our platforms is greater than local small pages, this is especially true for remote thp that is allocated intersocket and must be accessed through the interconnect.

Our users of MADV_HUGEPAGE are ok with assuming the burden of increased allocation latency, but certainly not remote access latency. There are users who remap their text segment onto transparent hugepages are fine with startup delay if they are access all of their text from local thp. Remote thp would be a significant performance degradation.

When Andrea brought this up, I suggested that the full solution would be a MPOL_F_HUGEPAGE flag that could define thp allocation policy -- the added benefit is that we could replace the thp "defrag" mode default by setting this as part of default_policy. Right now, MADV_HUGEPAGE users are concerned about (1) getting thp when system-wide it is not default and (2) additional fault latency when direct compaction is not default. They are not anticipating the degradation of remote access latency, so overloading the meaning of the mode is probably not a good idea.

Quoted 5 lines

That's not entirely true, the remote access latency for remote thp on all of our platforms is greater than local small pages, this is especially true for remote thp that is allocated intersocket and must be accessed through the interconnect.

Quoted 5 lines

Our users of MADV_HUGEPAGE are ok with assuming the burden of increased allocation latency, but certainly not remote access latency. There are users who remap their text segment onto transparent hugepages are fine with startup delay if they are access all of their text from local thp. Remote thp would be a significant performance degradation.

Well, it seems that expectations differ for users. It seems that kvm users do not really agree with your interpretation.

Quoted 8 lines

When Andrea brought this up, I suggested that the full solution would be a MPOL_F_HUGEPAGE flag that could define thp allocation policy -- the added benefit is that we could replace the thp "defrag" mode default by setting this as part of default_policy. Right now, MADV_HUGEPAGE users are concerned about (1) getting thp when system-wide it is not default and (2) additional fault latency when direct compaction is not default. They are not anticipating the degradation of remote access latency, so overloading the meaning of the mode is probably not a good idea.

hugepage specific MPOL flags sounds like yet another step into even more cluttered API and semantic, I am afraid. Why should this be any different from regular page allocations? You are getting off-node memory once your local node is full. You have to use an explicit binding to disallow that. THP should be similar in that regards. Once you have said that you _really_ want THP then you are closer to what we do for regular pages IMHO.

I do realize that this is a gray zone because nobody bothered to define the semantic since the MADV_HUGEPAGE has been introduced (a826e422420b4 is exceptionaly short of information). So we are left with more or less undefined behavior and define it properly now. As we can see this might regress in some workloads but I strongly suspect that an explicit binding sounds more logical approach than a thp specific mpol mode. If anything this should be a more generic memory policy basically saying that a zone/node reclaim mode should be enabled for the particular allocation.

On Tue, 11 Sep 2018, Michal Hocko wrote:

Quoted 8 lines

hugepage specific MPOL flags sounds like yet another step into even more cluttered API and semantic, I am afraid. Why should this be any different from regular page allocations? You are getting off-node memory once your local node is full. You have to use an explicit binding to disallow that. THP should be similar in that regards. Once you have said that you _really_ want THP then you are closer to what we do for regular pages IMHO.

Saying that we really want THP isn't an all-or-nothing decision. We certainly want to try hard to fault hugepages locally especially at task startup when remapping our .text segment to thp, and MADV_HUGEPAGE works very well for that. Remote hugepages would be a regression that we now have no way to avoid because the kernel doesn't provide for it, if we were to remove __GFP_THISNODE that this patch introduces.

On Broadwell, for example, we find 7% slower access to remote hugepages than local native pages. On Naples, that becomes worse: 14% slower access latency for intrasocket hugepages compared to local native pages and 39% slower for intersocket.

From what you have said so far it sounds like you would like to have

Quoted 6 lines

Saying that we really want THP isn't an all-or-nothing decision. We certainly want to try hard to fault hugepages locally especially at task startup when remapping our .text segment to thp, and MADV_HUGEPAGE works very well for that. Remote hugepages would be a regression that we now have no way to avoid because the kernel doesn't provide for it, if we were to remove __GFP_THISNODE that this patch introduces.

Why cannot you use mempolicy to bind to local nodes if you really care about the locality?

From what you have said so far it sounds like you would like to have something like the zone/node reclaim mode fine grained for a specific mapping. If we really want to support something like that then it should be a generic policy rather than THP specific thing IMHO.

As I've said it is hard to come up with a solution that would satisfy everybody but considering that the existing reports are seeing this a regression and cosindering their NUMA requirements are not so strict as yours I would tend to think that stronger NUMA requirements should be expressed explicitly rather than implicit effect of a madvise flag. We do have APIs for that.

Well, it seems that expectations differ for users. It seems that kvm users do not really agree with your interpretation.

I do realize that this is a gray zone because nobody bothered to define the semantic since the MADV_HUGEPAGE has been introduced (a826e422420b4 is exceptionaly short of information). So we are left with more or less undefined behavior and define it properly now. As we can see this might regress in some workloads but I strongly suspect that an explicit binding sounds more logical approach than a thp specific mpol mode. If anything this should be a more generic memory policy basically saying that a zone/node reclaim mode should be enabled for the particular allocation.

Hello,

Quoted 2 lines · Michal Hocko

Well, it seems that expectations differ for users. It seems that kvm users do not really agree with your interpretation.

Like David also mentioned here:

lkml.kernel.org/r/[email protected]

depends on the hardware what is a win, so there's no one size fits all.

For two sockets providing remote THP to KVM is likely a win, but changing the defaults depending on boot-time NUMA topology makes things less deterministic and it's also impossible to define an exact break even point.

Quoted 9 lines

I do realize that this is a gray zone because nobody bothered to define the semantic since the MADV_HUGEPAGE has been introduced (a826e422420b4 is exceptionaly short of information). So we are left with more or less undefined behavior and define it properly now. As we can see this might regress in some workloads but I strongly suspect that an explicit binding sounds more logical approach than a thp specific mpol mode. If anything this should be a more generic memory policy basically saying that a zone/node reclaim mode should be enabled for the particular allocation.

MADV_HUGEPAGE means the allocation is long lived, so the cost of compaction is worth it in direct reclaim. Not much else. That is not the problem.

The problem is that even if you ignore the breakage and regression to real life workloads, what is happening right now obviously would require root privilege but MADV_HUEGPAGE requires no root privilege.

Swapping heavy because MADV_HUGEPAGE when there are gigabytes free on other nodes and not even 4k would be swapped-out with THP turned off in sysfs, is simply not possibly what MADV_HUGEPAGE could have been about, and it's a kernel regression that never existed until that commit that added __GFP_THISNODE to the default THP heuristic in mempolicy.

I think we should defer the problem of what is better between 4k NUMA local or remote THP by default for later, I provided two options myself because it didn't matter so much which option we picked in the short term, as long as the bug was fixed.

I wasn't particularly happy about your patch because it still swaps with certain defrag settings which is still allowing things that shouldn't happen without some kind of privileged capability.

On Wed 12-09-18 09:54:17, Andrea Arcangeli wrote:

Quoted 3 lines

I wasn't particularly happy about your patch because it still swaps with certain defrag settings which is still allowing things that shouldn't happen without some kind of privileged capability.

Well, I am not really sure about defrag=always. I would rather care about the default behavior to plug the regression first. And think about `always' mode on top. Or is this a no-go from your POV?