Skip to contentSkip to conversation

mm,thp,shmem: limit shmem THP alloc gfp_mask

PATCH Superseded Rik van Riel 5 participants 6 messages Oct 22, 2020 – Oct 23, 2020

Subsystems (5)

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 mm,thp,shmem: limit shmem THP alloc gfp_mask

The allocation flags of anonymous transparent huge pages can be controlled through the files in /sys/kernel/mm/transparent_hugepage/defrag, which can help the system from getting bogged down in the page reclaim and compaction code when many THPs are getting allocated simultaneously.

However, the gfp_mask for shmem THP allocations were not limited by those configuration settings, and some workloads ended up with all CPUs stuck on the LRU lock in the page reclaim code, trying to allocate dozens of THPs simultaneously.

This patch applies the same configurated limitation of THPs to shmem hugepage allocations, to prevent that from happening.

This way a THP defrag setting of "never" or "defer+madvise" will result in quick allocation failures without direct reclaim when no 2MB free pages are available.

---
v2: move gfp calculation to shmem_getpage_gfp as suggested by Yu Xu

include/linux/gfp.h +2 -0
index c603237e006c..0a5b164a26d9 100644
@@ -614,6 +614,8 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask);
 extern void pm_restrict_gfp_mask(void);
 extern void pm_restore_gfp_mask(void);
 
+extern gfp_t alloc_hugepage_direct_gfpmask(struct vm_area_struct *vma);
+
 #ifdef CONFIG_PM_SLEEP
 extern bool pm_suspended_storage(void);
 #else
mm/huge_memory.c +1 -1
index 9474dbc150ed..9b08ce5cc387 100644
@@ -649,7 +649,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
  *	    available
  * never: never stall for any thp allocation
  */
-static inline gfp_t alloc_hugepage_direct_gfpmask(struct vm_area_struct *vma)
+gfp_t alloc_hugepage_direct_gfpmask(struct vm_area_struct *vma)
 {
 	const bool vma_madvised = !!(vma->vm_flags & VM_HUGEPAGE);
 
mm/shmem.c +5 -3
index 537c137698f8..9710b9df91e9 100644
@@ -1545,8 +1545,8 @@ static struct page *shmem_alloc_hugepage(gfp_t gfp,
 		return NULL;
 
 	shmem_pseudo_vma_init(&pvma, info, hindex);
-	page = alloc_pages_vma(gfp | __GFP_COMP | __GFP_NORETRY | __GFP_NOWARN,
-			HPAGE_PMD_ORDER, &pvma, 0, numa_node_id(), true);
+	page = alloc_pages_vma(gfp, HPAGE_PMD_ORDER, &pvma, 0, numa_node_id(),
+			       true);
 	shmem_pseudo_vma_destroy(&pvma);
 	if (page)
 		prep_transhuge_page(page);
@@ -1802,6 +1802,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,
 	struct page *page;
 	enum sgp_type sgp_huge = sgp;
 	pgoff_t hindex = index;
+	gfp_t huge_gfp;
 	int error;
 	int once = 0;
 	int alloced = 0;
@@ -1887,7 +1888,8 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,
 	}
 
 alloc_huge:
-	page = shmem_alloc_and_acct_page(gfp, inode, index, true);
+	huge_gfp = alloc_hugepage_direct_gfpmask(vma);
+	page = shmem_alloc_and_acct_page(huge_gfp, inode, index, true);
 	if (IS_ERR(page)) {
 alloc_nohuge:
 		page = shmem_alloc_and_acct_page(gfp, inode,
Acked-by on v2 1/1
#1 lore raw
Quoted 10 lines · Rik van Riel

The allocation flags of anonymous transparent huge pages can be controlled through the files in /sys/kernel/mm/transparent_hugepage/defrag, which can help the system from getting bogged down in the page reclaim and compaction code when many THPs are getting allocated simultaneously.

However, the gfp_mask for shmem THP allocations were not limited by those configuration settings, and some workloads ended up with all CPUs stuck on the LRU lock in the page reclaim code, trying to allocate dozens of THPs simultaneously.

Quoted 8 lines

This patch applies the same configurated limitation of THPs to shmem hugepage allocations, to prevent that from happening.

This way a THP defrag setting of "never" or "defer+madvise" will result in quick allocation failures without direct reclaim when no 2MB free pages are available.

Quoted 63 lines

---
v2: move gfp calculation to shmem_getpage_gfp as suggested by Yu Xu

include/linux/gfp.h +2 -0
index c603237e006c..0a5b164a26d9 100644
@@ -614,6 +614,8 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask);
  extern void pm_restrict_gfp_mask(void);
  extern void pm_restore_gfp_mask(void);
 
+extern gfp_t alloc_hugepage_direct_gfpmask(struct vm_area_struct *vma);
+
  #ifdef CONFIG_PM_SLEEP
  extern bool pm_suspended_storage(void);
  #else
mm/huge_memory.c +1 -1
index 9474dbc150ed..9b08ce5cc387 100644
@@ -649,7 +649,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
   *	    available
   * never: never stall for any thp allocation
   */
-static inline gfp_t alloc_hugepage_direct_gfpmask(struct vm_area_struct *vma)
+gfp_t alloc_hugepage_direct_gfpmask(struct vm_area_struct *vma)
  {
  	const bool vma_madvised = !!(vma->vm_flags & VM_HUGEPAGE);
 
mm/shmem.c +5 -3
index 537c137698f8..9710b9df91e9 100644
@@ -1545,8 +1545,8 @@ static struct page *shmem_alloc_hugepage(gfp_t gfp,
  		return NULL;
 
  	shmem_pseudo_vma_init(&pvma, info, hindex);
-	page = alloc_pages_vma(gfp | __GFP_COMP | __GFP_NORETRY | __GFP_NOWARN,
-			HPAGE_PMD_ORDER, &pvma, 0, numa_node_id(), true);
+	page = alloc_pages_vma(gfp, HPAGE_PMD_ORDER, &pvma, 0, numa_node_id(),
+			       true);
  	shmem_pseudo_vma_destroy(&pvma);
  	if (page)
  		prep_transhuge_page(page);
@@ -1802,6 +1802,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,
  	struct page *page;
  	enum sgp_type sgp_huge = sgp;
  	pgoff_t hindex = index;
+	gfp_t huge_gfp;
  	int error;
  	int once = 0;
  	int alloced = 0;
@@ -1887,7 +1888,8 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,
  	}
 
  alloc_huge:
-	page = shmem_alloc_and_acct_page(gfp, inode, index, true);
+	huge_gfp = alloc_hugepage_direct_gfpmask(vma);
+	page = shmem_alloc_and_acct_page(huge_gfp, inode, index, true);
  	if (IS_ERR(page)) {
  alloc_nohuge:
  		page = shmem_alloc_and_acct_page(gfp, inode,
maintainer
#2 lore raw
In reply to Rik van Riel
Quoted 10 lines · Rik van Riel

The allocation flags of anonymous transparent huge pages can be controlled through the files in /sys/kernel/mm/transparent_hugepage/defrag, which can help the system from getting bogged down in the page reclaim and compaction code when many THPs are getting allocated simultaneously.

However, the gfp_mask for shmem THP allocations were not limited by those configuration settings, and some workloads ended up with all CPUs stuck on the LRU lock in the page reclaim code, trying to allocate dozens of THPs simultaneously.

Quoted 8 lines

This patch applies the same configurated limitation of THPs to shmem hugepage allocations, to prevent that from happening.

This way a THP defrag setting of "never" or "defer+madvise" will result in quick allocation failures without direct reclaim when no 2MB free pages are available.

NAK in its present untested form: see below.

I'm open to change here, particularly to Yu Xu's point (in other mail) about direct reclaim - we avoid that here in Google too: though it's not so much to avoid the direct reclaim, as to avoid the latencies of direct compaction, which __GFP_DIRECT_RECLAIM allows as a side-effect.

Quoted 34 lines

---
v2: move gfp calculation to shmem_getpage_gfp as suggested by Yu Xu

include/linux/gfp.h +2 -0
index c603237e006c..0a5b164a26d9 100644
@@ -614,6 +614,8 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask);
 extern void pm_restrict_gfp_mask(void);
 extern void pm_restore_gfp_mask(void);
 
+extern gfp_t alloc_hugepage_direct_gfpmask(struct vm_area_struct *vma);
+
 #ifdef CONFIG_PM_SLEEP
 extern bool pm_suspended_storage(void);
 #else
mm/huge_memory.c +1 -1
index 9474dbc150ed..9b08ce5cc387 100644
@@ -649,7 +649,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,
  *	    available
  * never: never stall for any thp allocation
  */
-static inline gfp_t alloc_hugepage_direct_gfpmask(struct vm_area_struct *vma)
+gfp_t alloc_hugepage_direct_gfpmask(struct vm_area_struct *vma)
 {
 	const bool vma_madvised = !!(vma->vm_flags & VM_HUGEPAGE);
 
mm/shmem.c +0 -0
index 537c137698f8..9710b9df91e9 100644

mm/shmem.c static struct page *shmem_alloc_hugepage(gfp_t gfp, · line 1549

Quoted 7 lines
mm/shmem.c +2 -2
@@ -1545,8 +1545,8 @@ static struct page *shmem_alloc_hugepage(gfp_t gfp,
 		return NULL;
 
 	shmem_pseudo_vma_init(&pvma, info, hindex);
-	page = alloc_pages_vma(gfp | __GFP_COMP | __GFP_NORETRY | __GFP_NOWARN,
-			HPAGE_PMD_ORDER, &pvma, 0, numa_node_id(), true);
+	page = alloc_pages_vma(gfp, HPAGE_PMD_ORDER, &pvma, 0, numa_node_id(),
+			       true);

Commendably neat so far.

Quoted 12 lines
mm/shmem.c +1 -0
 	shmem_pseudo_vma_destroy(&pvma);
 	if (page)
 		prep_transhuge_page(page);
@@ -1802,6 +1802,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,
 	struct page *page;
 	enum sgp_type sgp_huge = sgp;
 	pgoff_t hindex = index;
+	gfp_t huge_gfp;
 	int error;
 	int once = 0;
 	int alloced = 0;

mm/shmem.c static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, · line 1891

Quoted 5 lines
mm/shmem.c +1 -1
@@ -1887,7 +1888,8 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,
 	}
 
 alloc_huge:
-	page = shmem_alloc_and_acct_page(gfp, inode, index, true);
+	huge_gfp = alloc_hugepage_direct_gfpmask(vma);

Still looks nice: but what about the crash when vma is NULL?

It may work for shmem_fault() (though I'll probably disagree on the details): but tmpfs is a filesystem, so most if not all of the system calls which arrive here have no vma to offer.

Michal is right to remember pushback before, because tmpfs is a filesystem, and "huge=" is a mount option: in using a huge=always filesystem, the user has already declared a preference for huge pages. Whereas the original anon THP had to deduce that preference from sys tunables and vma madvice.

I certainly found it a lot easier to ignore all the shifting sandmaze of the anon THP tunables, and I think Kirill followed me on that.

But it's likely that they have accumulated some defrag wisdom, which tmpfs can take on board - but please accept that in using a huge mount, the preference for huge has already been expressed, so I don't expect anon THP alloc_hugepage_direct_gfpmask() choices will map one to one.

mm/shmem.c static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, · line 1895

Quoted 5 lines
mm/shmem.c +1 -0
+	page = shmem_alloc_and_acct_page(huge_gfp, inode, index, true);
 	if (IS_ERR(page)) {
 alloc_nohuge:
 		page = shmem_alloc_and_acct_page(gfp, inode,

Hugh

#3 lore raw
Quoted 23 lines · Hugh Dickins
Quoted 1 line · Rik van Riel

The allocation flags of anonymous transparent huge pages can be

controlled

Quoted 1 line

through the files in /sys/kernel/mm/transparent_hugepage/defrag,

which can

Quoted 1 line

help the system from getting bogged down in the page reclaim and

compaction

Quoted 3 lines

code when many THPs are getting allocated simultaneously.

However, the gfp_mask for shmem THP allocations were not limited by

those

Quoted 1 line

configuration settings, and some workloads ended up with all CPUs

stuck

Quoted 1 line

on the LRU lock in the page reclaim code, trying to allocate dozens

of

Quoted 3 lines

THPs simultaneously.

This patch applies the same configurated limitation of THPs to

shmem

Quoted 3 lines

hugepage allocations, to prevent that from happening.

This way a THP defrag setting of "never" or "defer+madvise" will

Quoted 8 lines

result

Quoted 1 line

in quick allocation failures without direct reclaim when no 2MB

free

Quoted 3 lines

pages are available.

NAK in its present untested form: see below.

Oops. That issue is easy to fix, but indeed lets figure out what the desired behavior is.

Quoted 9 lines

I'm open to change here, particularly to Yu Xu's point (in other mail)
about direct reclaim - we avoid that here in Google too: though it's not so much to avoid the direct reclaim, as to avoid the latencies of direct compaction, which __GFP_DIRECT_RECLAIM allows as a side- effect.

Review static int shmem_getpage_gfp(struct inode

Quoted 1 line
(quoted diff) +0 -0
@@ -1887,7 +1888,8 @@ static int shmem_getpage_gfp(struct inode

*inode, pgoff_t index,

Quoted 7 lines

mm/shmem.c static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, · line 1891

Quoted 5 lines
mm/shmem.c +1 -1
      }
 
 alloc_huge:
-     page = shmem_alloc_and_acct_page(gfp, inode, index, true);
+     huge_gfp = alloc_hugepage_direct_gfpmask(vma);

Still looks nice: but what about the crash when vma is NULL?

That's a one line fix, but I suppose we should get the discussion on what the code behavior should be out of the way first :)

Quoted 6 lines

Michal is right to remember pushback before, because tmpfs is a filesystem, and "huge=" is a mount option: in using a huge=always filesystem, the user has already declared a preference for huge pages.
Whereas the original anon THP had to deduce that preference from sys tunables and vma madvice.

...

Quoted 5 lines

But it's likely that they have accumulated some defrag wisdom, which tmpfs can take on board - but please accept that in using a huge mount,
the preference for huge has already been expressed, so I don't expect anon THP alloc_hugepage_direct_gfpmask() choices will map one to one.

In my mind, the huge= mount options for tmpfs corresponded to the "enabled" anon THP options, denoting a desired end state, not necessarily how much we will stall allocations to get there immediately.

The underlying allocation behavior has been changed repeatedly, with changes to the direct reclaim code and the compaction deferral code.

The shmem THP gfp_mask never tried really hard anyway, with __GFP_NORETRY being the default, which matches what is used for non-VM_HUGEPAGE anon VMAs.

Likewise, the direct reclaim done from the opportunistic THP allocations done by the shmem code limited itself to reclaiming 32 4kB pages per THP allocation.

In other words, mounting
with huge=always has never behaved
the same as the more aggressive allocations done for MADV_HUGEPAGE VMAs.

This patch would leave shmem THP allocations for non-MADV_HUGEPAGE mapped files opportunistic like today, and make shmem THP allocations for files mapped with MADV_HUGEPAGE more aggressive than today.

However, I would like to know what people think the shmem huge= mount options should do, and how things should behave when memory gets low, before pushing in a patch just because it makes the system run smoother "without changing current behavior too much".

What do people want tmpfs THP allocations to do?

Signature · 1 line
All Rights Reversed.
reviewer
#4 lore raw
Quoted 1 line · Rik van Riel

On Thu, 2020-10-22 at 19:54 -0700, Hugh Dickins wrote:

[...]

Quoted 10 lines
Quoted 5 lines

But it's likely that they have accumulated some defrag wisdom, which tmpfs can take on board - but please accept that in using a huge mount,
the preference for huge has already been expressed, so I don't expect anon THP alloc_hugepage_direct_gfpmask() choices will map one to one.

In my mind, the huge= mount options for tmpfs corresponded to the "enabled" anon THP options, denoting a desired end state, not necessarily how much we will stall allocations to get there immediately.

It is really unfortunate that our configuration space is so huge and messy but we have to live with that now.

Anyway, I would tend to agree that with an absense of per-mount defrag configuration it makes sense to use the global one. Is anybody aware of usecases where a mount specific configuration would make sense?

Signature · 2 lines
Michal Hocko
SUSE Labs
#5 lore raw
In reply to Rik van Riel
Quoted 43 lines · Rik van Riel
Quoted 6 lines · Hugh Dickins

Michal is right to remember pushback before, because tmpfs is a filesystem, and "huge=" is a mount option: in using a huge=always filesystem, the user has already declared a preference for huge pages.
Whereas the original anon THP had to deduce that preference from sys tunables and vma madvice.

...

Quoted 5 lines

But it's likely that they have accumulated some defrag wisdom, which tmpfs can take on board - but please accept that in using a huge mount,
the preference for huge has already been expressed, so I don't expect anon THP alloc_hugepage_direct_gfpmask() choices will map one to one.

In my mind, the huge= mount options for tmpfs corresponded to the "enabled" anon THP options, denoting a desired end state, not necessarily how much we will stall allocations to get there immediately.

The underlying allocation behavior has been changed repeatedly, with changes to the direct reclaim code and the compaction deferral code.

The shmem THP gfp_mask never tried really hard anyway, with __GFP_NORETRY being the default, which matches what is used for non-VM_HUGEPAGE anon VMAs.

Likewise, the direct reclaim done from the opportunistic THP allocations done by the shmem code limited itself to reclaiming 32 4kB pages per THP allocation.

In other words, mounting
with huge=always has never behaved
the same as the more aggressive allocations done for MADV_HUGEPAGE VMAs.

This patch would leave shmem THP allocations for non-MADV_HUGEPAGE mapped files opportunistic like today, and make shmem THP allocations for files mapped with MADV_HUGEPAGE more aggressive than today.

Quoted 7 lines

However, I would like to know what people think the shmem huge= mount options should do, and how things should behave when memory gets low, before pushing in a patch just because it makes the system run smoother "without changing current behavior too much".

What do people want tmpfs THP allocations to do?

I'm also interested for non-tmpfs THP allocations. In my patchset, THPs are no longer limited to being PMD sized, and allocating smaller pages isn't such a tax on the VM. So currently I'm doing:

        gfp_t gfp = readahead_gfp_mask(mapping);
...
        struct page *page = __page_cache_alloc_order(gfp, order);

which translates to:

        mapping_gfp_mask(mapping) | __GFP_NORETRY | __GFP_NOWARN;
        gfp |= GFP_TRANSHUGE_LIGHT;
        gfp &= ~__GFP_DIRECT_RECLAIM;

Everything's very willing to fall back to order-0 pages, but I can see that, eg, for a VM_HUGEPAGE vma, we should perhaps be less willing to fall back to small pages. I would prefer not to add a mount option to every filesystem. People will only get it wrong.