Skip to contentSkip to conversation

mm/page_alloc: avoid high-order page allocation warn with __GFP_NOFAIL

PATCH In review Gao Xiang 5 participants 9 messages Mar 5, 2023 – Mar 7, 2023

Subsystems (3)
lore
Share & tools
lore

Gao Xiang

author
#0 lore raw

PATCH 1/1 mm/page_alloc: avoid high-order page allocation warn with __GFP_NOFAIL

My knowledge of this is somewhat limited, however, since vmalloc already supported __GFP_NOFAIL in commit 9376130c390a ("mm/vmalloc: add support for __GFP_NOFAIL"). __GFP_NOFAIL could trigger the following stack and allocate high-order pages when CONFIG_HAVE_ARCH_HUGE_VMALLOC is enabled:

 __alloc_pages+0x1cb/0x5b0 mm/page_alloc.c:5549
 alloc_pages+0x1aa/0x270 mm/mempolicy.c:2286
 vm_area_alloc_pages mm/vmalloc.c:2989 [inline]
 __vmalloc_area_node mm/vmalloc.c:3057 [inline]
 __vmalloc_node_range+0x978/0x13c0 mm/vmalloc.c:3227
 kvmalloc_node+0x156/0x1a0 mm/util.c:606
 kvmalloc include/linux/slab.h:737 [inline]
 kvmalloc_array include/linux/slab.h:755 [inline]
 kvcalloc include/linux/slab.h:760 [inline]
 (codebase: Linux 6.2-rc2)

Don't warn such cases since high-order pages with __GFP_NOFAIL is somewhat legel.

Cc Mel Gorman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Vlastimil Babka <[email protected]>
---
 mm/page_alloc.c | 6 ------
 1 file changed, 6 deletions(-)
mm/page_alloc.c +0 -6
index 0745aedebb37..0618716c49df 100644
@@ -3822,12 +3822,6 @@ struct page *rmqueue(struct zone *preferred_zone,
 {
 	struct page *page;
 
-	/*
-	 * We most definitely don't want callers attempting to
-	 * allocate greater than order-1 page units with __GFP_NOFAIL.
-	 */
-	WARN_ON_ONCE((gfp_flags & __GFP_NOFAIL) && (order > 1));
-
 	if (likely(pcp_allowed_order(order))) {
 		/*
 		 * MIGRATE_MOVABLE pcplist could have the pages on CMA area and
Signature · 1 line
2.24.4
reviewer
#1 lore raw

[Cc couple of more people recently involved with vmalloc code]

Quoted 18 lines · Gao Xiang

My knowledge of this is somewhat limited, however, since vmalloc already supported __GFP_NOFAIL in commit 9376130c390a ("mm/vmalloc: add support for __GFP_NOFAIL"). __GFP_NOFAIL could trigger the following stack and allocate high-order pages when CONFIG_HAVE_ARCH_HUGE_VMALLOC is enabled:

 __alloc_pages+0x1cb/0x5b0 mm/page_alloc.c:5549
 alloc_pages+0x1aa/0x270 mm/mempolicy.c:2286
 vm_area_alloc_pages mm/vmalloc.c:2989 [inline]

 __vmalloc_area_node mm/vmalloc.c:3057 [inline]
 __vmalloc_node_range+0x978/0x13c0 mm/vmalloc.c:3227
 kvmalloc_node+0x156/0x1a0 mm/util.c:606
 kvmalloc include/linux/slab.h:737 [inline]
 kvmalloc_array include/linux/slab.h:755 [inline]
 kvcalloc include/linux/slab.h:760 [inline]
 (codebase: Linux 6.2-rc2)
Quoted 2 lines

Don't warn such cases since high-order pages with __GFP_NOFAIL is somewhat legel.

OK, this is definitely a bug and it seems my 9376130c390a was incomplete because it hasn't covered the high order case. Not sure how that happened but removing the warning is not the right thing to do here. The higher order allocation is an optimization rather than a must. So it is perfectly fine to fail that allocation and retry rather than go into a very expensive and potentially impossible higher order allocation that must not fail.

The proper fix should look like this unless I am missing something. I would appreciate another pair of eyes on this because I am not fully familiar with the high order optimization part much.

Thanks!
---

mm/vmalloc.c +17 -5
index ef910bf349e1..a8aa2765618a 100644
@@ -2883,6 +2883,8 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 		unsigned int order, unsigned int nr_pages, struct page **pages)
 {
 	unsigned int nr_allocated = 0;
+	gfp_t alloc_gfp = gfp;
+	bool nofail = false;
 	struct page *page;
 	int i;
 
@@ -2931,20 +2933,30 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 			if (nr != nr_pages_request)
 				break;
 		}
+	} else {
+		alloc_gfp &= ~__GFP_NOFAIL;
+		nofail = true;
 	}
 
 	/* High-order pages or fallback path if "bulk" fails. */
-
 	while (nr_allocated < nr_pages) {
 		if (fatal_signal_pending(current))
 			break;
 
 		if (nid == NUMA_NO_NODE)
-			page = alloc_pages(gfp, order);
+			page = alloc_pages(alloc_gfp, order);
 		else
-			page = alloc_pages_node(nid, gfp, order);
-		if (unlikely(!page))
-			break;
+			page = alloc_pages_node(nid, alloc_gfp, order);
+		if (unlikely(!page)) {
+			if (!nofail)
+				break;
+
+			/* fall back to the zero order allocations */
+			alloc_gfp |= __GFP_NOFAIL;
+			order = 0;
+			continue;
+		}
+
 		/*
 		 * Higher order allocations must be able to be treated as
 		 * indepdenent small pages by callers (as they can with
Signature · 2 lines
Michal Hocko
SUSE Labs
author
#2 lore raw
Quoted 32 lines · Michal Hocko

[Cc couple of more people recently involved with vmalloc code]

Quoted 18 lines · Gao Xiang

My knowledge of this is somewhat limited, however, since vmalloc already supported __GFP_NOFAIL in commit 9376130c390a ("mm/vmalloc: add support for __GFP_NOFAIL"). __GFP_NOFAIL could trigger the following stack and allocate high-order pages when CONFIG_HAVE_ARCH_HUGE_VMALLOC is enabled:

  __alloc_pages+0x1cb/0x5b0 mm/page_alloc.c:5549
  alloc_pages+0x1aa/0x270 mm/mempolicy.c:2286
  vm_area_alloc_pages mm/vmalloc.c:2989 [inline]

  __vmalloc_area_node mm/vmalloc.c:3057 [inline]
  __vmalloc_node_range+0x978/0x13c0 mm/vmalloc.c:3227
  kvmalloc_node+0x156/0x1a0 mm/util.c:606
  kvmalloc include/linux/slab.h:737 [inline]
  kvmalloc_array include/linux/slab.h:755 [inline]
  kvcalloc include/linux/slab.h:760 [inline]
  (codebase: Linux 6.2-rc2)
Quoted 2 lines

Don't warn such cases since high-order pages with __GFP_NOFAIL is somewhat legel.

OK, this is definitely a bug and it seems my 9376130c390a was incomplete because it hasn't covered the high order case. Not sure how that happened but removing the warning is not the right thing to do here. The higher order allocation is an optimization rather than a must. So it is perfectly fine to fail that allocation and retry rather than go into a very expensive and potentially impossible higher order allocation that must not fail.

Quoted 3 lines

The proper fix should look like this unless I am missing something. I would appreciate another pair of eyes on this because I am not fully familiar with the high order optimization part much.

I'm fine with the fix. Although I'm not familiar with such vmalloc allocation, I thought about this possibility as well.

The original issue was:
https://lore.kernel.org/r/[email protected]

which I used kvcalloc with __GFP_NOFAIL but it warned, and I made a fix (which now seems wrong) to use kcalloc() but it now warns the same:
https://lore.kernel.org/r/[email protected]

And I then realized it's a bug in kvmalloc() with __GFP_NOFAIL...

Thanks,
Gao Xiang

Quoted 52 lines

Thanks!
---

mm/vmalloc.c +17 -5
index ef910bf349e1..a8aa2765618a 100644
@@ -2883,6 +2883,8 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
  		unsigned int order, unsigned int nr_pages, struct page **pages)
  {
  	unsigned int nr_allocated = 0;
+	gfp_t alloc_gfp = gfp;
+	bool nofail = false;
  	struct page *page;
  	int i;
 
@@ -2931,20 +2933,30 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
  			if (nr != nr_pages_request)
  				break;
  		}
+	} else {
+		alloc_gfp &= ~__GFP_NOFAIL;
+		nofail = true;
  	}
 
  	/* High-order pages or fallback path if "bulk" fails. */
-
  	while (nr_allocated < nr_pages) {
  		if (fatal_signal_pending(current))
  			break;
 
  		if (nid == NUMA_NO_NODE)
-			page = alloc_pages(gfp, order);
+			page = alloc_pages(alloc_gfp, order);
  		else
-			page = alloc_pages_node(nid, gfp, order);
-		if (unlikely(!page))
-			break;
+			page = alloc_pages_node(nid, alloc_gfp, order);
+		if (unlikely(!page)) {
+			if (!nofail)
+				break;
+
+			/* fall back to the zero order allocations */
+			alloc_gfp |= __GFP_NOFAIL;
+			order = 0;
+			continue;
+		}
+
  		/*
  		 * Higher order allocations must be able to be treated as
  		 * indepdenent small pages by callers (as they can with
maintainer
#3 lore raw
In reply to Michal Hocko
Quoted 78 lines · Michal Hocko

[Cc couple of more people recently involved with vmalloc code]

Quoted 18 lines · Gao Xiang

My knowledge of this is somewhat limited, however, since vmalloc already supported __GFP_NOFAIL in commit 9376130c390a ("mm/vmalloc: add support for __GFP_NOFAIL"). __GFP_NOFAIL could trigger the following stack and allocate high-order pages when CONFIG_HAVE_ARCH_HUGE_VMALLOC is enabled:

 __alloc_pages+0x1cb/0x5b0 mm/page_alloc.c:5549
 alloc_pages+0x1aa/0x270 mm/mempolicy.c:2286
 vm_area_alloc_pages mm/vmalloc.c:2989 [inline]

 __vmalloc_area_node mm/vmalloc.c:3057 [inline]
 __vmalloc_node_range+0x978/0x13c0 mm/vmalloc.c:3227
 kvmalloc_node+0x156/0x1a0 mm/util.c:606
 kvmalloc include/linux/slab.h:737 [inline]
 kvmalloc_array include/linux/slab.h:755 [inline]
 kvcalloc include/linux/slab.h:760 [inline]
 (codebase: Linux 6.2-rc2)
Quoted 2 lines

Don't warn such cases since high-order pages with __GFP_NOFAIL is somewhat legel.

OK, this is definitely a bug and it seems my 9376130c390a was incomplete because it hasn't covered the high order case. Not sure how that happened but removing the warning is not the right thing to do here. The higher order allocation is an optimization rather than a must. So it is perfectly fine to fail that allocation and retry rather than go into a very expensive and potentially impossible higher order allocation that must not fail.

The proper fix should look like this unless I am missing something. I would appreciate another pair of eyes on this because I am not fully familiar with the high order optimization part much.

Thanks!
---

mm/vmalloc.c +10 -5
index ef910bf349e1..a8aa2765618a 100644
@@ -2883,6 +2883,8 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 		unsigned int order, unsigned int nr_pages, struct page **pages)
 {
 	unsigned int nr_allocated = 0;
+	gfp_t alloc_gfp = gfp;
+	bool nofail = false;
 	struct page *page;
 	int i;
 
@@ -2931,20 +2933,30 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 			if (nr != nr_pages_request)
 				break;
 		}
+	} else {
+		alloc_gfp &= ~__GFP_NOFAIL;
+		nofail = true;
 	}
 
 	/* High-order pages or fallback path if "bulk" fails. */
-
 	while (nr_allocated < nr_pages) {
 		if (fatal_signal_pending(current))
 			break;
 
 		if (nid == NUMA_NO_NODE)
-			page = alloc_pages(gfp, order);
+			page = alloc_pages(alloc_gfp, order);
 		else
-			page = alloc_pages_node(nid, gfp, order);
-		if (unlikely(!page))
-			break;
+			page = alloc_pages_node(nid, alloc_gfp, order);
+		if (unlikely(!page)) {
+			if (!nofail)
+				break;

mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 2962

Quoted 10 lines
mm/vmalloc.c +7 -0
+
+			/* fall back to the zero order allocations */
+			alloc_gfp |= __GFP_NOFAIL;
+			order = 0;
+			continue;
+		}
+
 		/*
 		 * Higher order allocations must be able to be treated as
 		 * indepdenent small pages by callers (as they can with

Some questions:

1. Could you please add a comment why you want the bulk_gfp without the __GFP_NOFAIL(bulk path)?
2. Could you please add a comment why a high order pages do not want __GFP_NOFAIL? You have already explained.
3. Looking at the patch:

<snip>
+ } else {
+ alloc_gfp &= ~__GFP_NOFAIL;
+ nofail = true;
<snip>

if user does not want to go with __GFP_NOFAIL flag why you force it in
case a high order allocation fails and you switch to 0 order allocations? (for high order-pages scenario you always use __GFP_NOFAIL in the order-0 recovery path).

Thanks!

--
Uladzislau Rezki

reviewer
#4 lore raw

PATCH 1/1 mm, vmalloc: fix high order __GFP_NOFAIL allocations

On Mon 06-03-23 13:14:43, Uladzislau Rezki wrote:
[...]

Quoted 4 lines

Some questions:

1. Could you please add a comment why you want the bulk_gfp without the __GFP_NOFAIL(bulk path)?

The bulk allocator is not documented to fully support __GFP_NOFAIL semantic IIRC. While it uses alloc_pages as fallback I didn't want to make any assumptions based on the current implementation. At least that is my recollection. If we do want to support NOFAIL by the batch allocator then we can drop the special casing here.

Quoted 2 lines

2. Could you please add a comment why a high order pages do not want __GFP_NOFAIL? You have already explained.

See below

Quoted 10 lines

3. Looking at the patch:

<snip>

(quoted diff) +3 -0
+       } else {
+               alloc_gfp &= ~__GFP_NOFAIL;
+               nofail = true;
<snip>

if user does not want to go with __GFP_NOFAIL flag why you force it in case a high order allocation fails and you switch to 0 order allocations?

Not intended. The above should have been else if (gfp & __GFP_NOFAIL). Thanks for catching that!

This would be the full patch with the description:
---
From 3ccfaa15bf2587b8998c129533a0404fedf5a484 Mon Sep 17 00:00:00 2001
From: Michal Hocko <[email protected]>
Date: Mon, 6 Mar 2023 09:15:17 +0100
Subject: [PATCH] mm, vmalloc: fix high order __GFP_NOFAIL allocations

Gao Xiang has reported that the page allocator complains about high order __GFP_NOFAIL request coming from the vmalloc core:

 __alloc_pages+0x1cb/0x5b0 mm/page_alloc.c:5549
 alloc_pages+0x1aa/0x270 mm/mempolicy.c:2286
 vm_area_alloc_pages mm/vmalloc.c:2989 [inline]
 __vmalloc_area_node mm/vmalloc.c:3057 [inline]
 __vmalloc_node_range+0x978/0x13c0 mm/vmalloc.c:3227
 kvmalloc_node+0x156/0x1a0 mm/util.c:606
 kvmalloc include/linux/slab.h:737 [inline]
 kvmalloc_array include/linux/slab.h:755 [inline]
 kvcalloc include/linux/slab.h:760 [inline]

it seems that I have completely missed high order allocation backing vmalloc areas case when implementing __GFP_NOFAIL support. This means that [k]vmalloc at al. can allocate higher order allocations with __GFP_NOFAIL which can trigger OOM killer for non-costly orders easily or cause a lot of reclaim/compaction activity if those requests cannot be satisfied.

Fix the issue by falling back to zero order allocations for __GFP_NOFAIL requests if the high order request fails.

Fixes 9376130c390a ("mm/vmalloc: add support for __GFP_NOFAIL")
---
 mm/vmalloc.c | 28 +++++++++++++++++++++++-----
 1 file changed, 23 insertions(+), 5 deletions(-)
mm/vmalloc.c +23 -5
index ef910bf349e1..bef6cf2b4d46 100644
@@ -2883,6 +2883,8 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 		unsigned int order, unsigned int nr_pages, struct page **pages)
 {
 	unsigned int nr_allocated = 0;
+	gfp_t alloc_gfp = gfp;
+	bool nofail = false;
 	struct page *page;
 	int i;
 
@@ -2893,6 +2895,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 	 * more permissive.
 	 */
 	if (!order) {
+		/* bulk allocator doesn't support nofail req. officially */
 		gfp_t bulk_gfp = gfp & ~__GFP_NOFAIL;
 
 		while (nr_allocated < nr_pages) {
@@ -2931,20 +2934,35 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 			if (nr != nr_pages_request)
 				break;
 		}
+	} else if (gfp & __GFP_NOFAIL) {
+		/*
+		 * Higher order nofail allocations are really expensive and
+		 * potentially dangerous (pre-mature OOM, disruptive reclaim
+		 * and compaction etc.
+		 */
+		alloc_gfp &= ~__GFP_NOFAIL;
+		nofail = true;
 	}
 
 	/* High-order pages or fallback path if "bulk" fails. */
-
 	while (nr_allocated < nr_pages) {
 		if (fatal_signal_pending(current))
 			break;
 
 		if (nid == NUMA_NO_NODE)
-			page = alloc_pages(gfp, order);
+			page = alloc_pages(alloc_gfp, order);
 		else
-			page = alloc_pages_node(nid, gfp, order);
-		if (unlikely(!page))
-			break;
+			page = alloc_pages_node(nid, alloc_gfp, order);
+		if (unlikely(!page)) {
+			if (!nofail)
+				break;
+
+			/* fall back to the zero order allocations */
+			alloc_gfp |= __GFP_NOFAIL;
+			order = 0;
+			continue;
+		}
+
 		/*
 		 * Higher order allocations must be able to be treated as
 		 * indepdenent small pages by callers (as they can with
Signature · 5 lines
2.30.2

-- 
Michal Hocko
SUSE Labs
maintainer Reviewed-by on v1 1/1
#5 lore raw

Re: [PATCH] mm, vmalloc: fix high order __GFP_NOFAIL allocations

Quoted 133 lines · Michal Hocko

On Mon 06-03-23 13:14:43, Uladzislau Rezki wrote:
[...]

Quoted 4 lines

Some questions:

1. Could you please add a comment why you want the bulk_gfp without the __GFP_NOFAIL(bulk path)?

The bulk allocator is not documented to fully support __GFP_NOFAIL semantic IIRC. While it uses alloc_pages as fallback I didn't want to make any assumptions based on the current implementation. At least that is my recollection. If we do want to support NOFAIL by the batch allocator then we can drop the special casing here.

Quoted 2 lines

2. Could you please add a comment why a high order pages do not want __GFP_NOFAIL? You have already explained.

See below

Quoted 10 lines

3. Looking at the patch:

<snip>

mm/vmalloc.c +3 -0
+       } else {
+               alloc_gfp &= ~__GFP_NOFAIL;
+               nofail = true;
<snip>

if user does not want to go with __GFP_NOFAIL flag why you force it in case a high order allocation fails and you switch to 0 order allocations?

Not intended. The above should have been else if (gfp & __GFP_NOFAIL). Thanks for catching that!

This would be the full patch with the description:
---
From 3ccfaa15bf2587b8998c129533a0404fedf5a484 Mon Sep 17 00:00:00 2001
From: Michal Hocko <[email protected]>
Date: Mon, 6 Mar 2023 09:15:17 +0100
Subject: [PATCH] mm, vmalloc: fix high order __GFP_NOFAIL allocations

Gao Xiang has reported that the page allocator complains about high order __GFP_NOFAIL request coming from the vmalloc core:

 __alloc_pages+0x1cb/0x5b0 mm/page_alloc.c:5549
 alloc_pages+0x1aa/0x270 mm/mempolicy.c:2286
 vm_area_alloc_pages mm/vmalloc.c:2989 [inline]
 __vmalloc_area_node mm/vmalloc.c:3057 [inline]
 __vmalloc_node_range+0x978/0x13c0 mm/vmalloc.c:3227
 kvmalloc_node+0x156/0x1a0 mm/util.c:606
 kvmalloc include/linux/slab.h:737 [inline]
 kvmalloc_array include/linux/slab.h:755 [inline]
 kvcalloc include/linux/slab.h:760 [inline]

it seems that I have completely missed high order allocation backing vmalloc areas case when implementing __GFP_NOFAIL support. This means that [k]vmalloc at al. can allocate higher order allocations with __GFP_NOFAIL which can trigger OOM killer for non-costly orders easily or cause a lot of reclaim/compaction activity if those requests cannot be satisfied.

Fix the issue by falling back to zero order allocations for __GFP_NOFAIL requests if the high order request fails.

Fixes 9376130c390a ("mm/vmalloc: add support for __GFP_NOFAIL")
---
 mm/vmalloc.c | 28 +++++++++++++++++++++++-----
 1 file changed, 23 insertions(+), 5 deletions(-)
mm/vmalloc.c +23 -5
index ef910bf349e1..bef6cf2b4d46 100644
@@ -2883,6 +2883,8 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 		unsigned int order, unsigned int nr_pages, struct page **pages)
 {
 	unsigned int nr_allocated = 0;
+	gfp_t alloc_gfp = gfp;
+	bool nofail = false;
 	struct page *page;
 	int i;
 
@@ -2893,6 +2895,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 	 * more permissive.
 	 */
 	if (!order) {
+		/* bulk allocator doesn't support nofail req. officially */
 		gfp_t bulk_gfp = gfp & ~__GFP_NOFAIL;
 
 		while (nr_allocated < nr_pages) {
@@ -2931,20 +2934,35 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 			if (nr != nr_pages_request)
 				break;
 		}
+	} else if (gfp & __GFP_NOFAIL) {
+		/*
+		 * Higher order nofail allocations are really expensive and
+		 * potentially dangerous (pre-mature OOM, disruptive reclaim
+		 * and compaction etc.
+		 */
+		alloc_gfp &= ~__GFP_NOFAIL;
+		nofail = true;
 	}
 
 	/* High-order pages or fallback path if "bulk" fails. */
-
 	while (nr_allocated < nr_pages) {
 		if (fatal_signal_pending(current))
 			break;
 
 		if (nid == NUMA_NO_NODE)
-			page = alloc_pages(gfp, order);
+			page = alloc_pages(alloc_gfp, order);
 		else
-			page = alloc_pages_node(nid, gfp, order);
-		if (unlikely(!page))
-			break;
+			page = alloc_pages_node(nid, alloc_gfp, order);
+		if (unlikely(!page)) {
+			if (!nofail)
+				break;
+
+			/* fall back to the zero order allocations */
+			alloc_gfp |= __GFP_NOFAIL;
+			order = 0;
+			continue;
+		}
+
 		/*
 		 * Higher order allocations must be able to be treated as
 		 * indepdenent small pages by callers (as they can with
Signature · 1 line
2.30.2

--
Uladzislau Rezki

Acked-by on v1 1/1
#6 lore raw

Re: [PATCH] mm, vmalloc: fix high order __GFP_NOFAIL allocations

In reply to Michal Hocko
Quoted 26 lines · Michal Hocko

---
From 3ccfaa15bf2587b8998c129533a0404fedf5a484 Mon Sep 17 00:00:00 2001
From: Michal Hocko <[email protected]>
Date: Mon, 6 Mar 2023 09:15:17 +0100
Subject: [PATCH] mm, vmalloc: fix high order __GFP_NOFAIL allocations

Gao Xiang has reported that the page allocator complains about high order __GFP_NOFAIL request coming from the vmalloc core:

 __alloc_pages+0x1cb/0x5b0 mm/page_alloc.c:5549
 alloc_pages+0x1aa/0x270 mm/mempolicy.c:2286
 vm_area_alloc_pages mm/vmalloc.c:2989 [inline]
 __vmalloc_area_node mm/vmalloc.c:3057 [inline]
 __vmalloc_node_range+0x978/0x13c0 mm/vmalloc.c:3227
 kvmalloc_node+0x156/0x1a0 mm/util.c:606
 kvmalloc include/linux/slab.h:737 [inline]
 kvmalloc_array include/linux/slab.h:755 [inline]
 kvcalloc include/linux/slab.h:760 [inline]

it seems that I have completely missed high order allocation backing vmalloc areas case when implementing __GFP_NOFAIL support. This means that [k]vmalloc at al. can allocate higher order allocations with __GFP_NOFAIL which can trigger OOM killer for non-costly orders easily or cause a lot of reclaim/compaction activity if those requests cannot be satisfied.

Quoted 6 lines

Fix the issue by falling back to zero order allocations for __GFP_NOFAIL requests if the high order request fails.

Fixes 9376130c390a ("mm/vmalloc: add support for __GFP_NOFAIL")
Quoted 26 lines
---
 mm/vmalloc.c | 28 +++++++++++++++++++++++-----
 1 file changed, 23 insertions(+), 5 deletions(-)
mm/vmalloc.c +3 -0
index ef910bf349e1..bef6cf2b4d46 100644
@@ -2883,6 +2883,8 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 		unsigned int order, unsigned int nr_pages, struct page **pages)
 {
 	unsigned int nr_allocated = 0;
+	gfp_t alloc_gfp = gfp;
+	bool nofail = false;
 	struct page *page;
 	int i;
 
@@ -2893,6 +2895,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 	 * more permissive.
 	 */
 	if (!order) {
+		/* bulk allocator doesn't support nofail req. officially */
 		gfp_t bulk_gfp = gfp & ~__GFP_NOFAIL;
 
 		while (nr_allocated < nr_pages) {

mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 2941

Quoted 8 lines
mm/vmalloc.c +5 -0
@@ -2931,20 +2934,35 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 			if (nr != nr_pages_request)
 				break;
 		}
+	} else if (gfp & __GFP_NOFAIL) {
+		/*
+		 * Higher order nofail allocations are really expensive and
+		 * potentially dangerous (pre-mature OOM, disruptive reclaim
+		 * and compaction etc.

^ unclosed parenthesis

Quoted 22 lines
mm/vmalloc.c +8 -5
+		 */
+		alloc_gfp &= ~__GFP_NOFAIL;
+		nofail = true;
 	}
 
 	/* High-order pages or fallback path if "bulk" fails. */
-
 	while (nr_allocated < nr_pages) {
 		if (fatal_signal_pending(current))
 			break;
 
 		if (nid == NUMA_NO_NODE)
-			page = alloc_pages(gfp, order);
+			page = alloc_pages(alloc_gfp, order);
 		else
-			page = alloc_pages_node(nid, gfp, order);
-		if (unlikely(!page))
-			break;
+			page = alloc_pages_node(nid, alloc_gfp, order);
+		if (unlikely(!page)) {
+			if (!nofail)
+				break;

mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 2968

Quoted 10 lines
mm/vmalloc.c +7 -0
+
+			/* fall back to the zero order allocations */
+			alloc_gfp |= __GFP_NOFAIL;
+			order = 0;
+			continue;
+		}
+
 		/*
 		 * Higher order allocations must be able to be treated as
 		 * indepdenent small pages by callers (as they can with

^ while at it the typo could also be fixed

reviewer
#7 lore raw

Re: [PATCH] mm, vmalloc: fix high order __GFP_NOFAIL allocations

Thanks. Here is an incremental diff

mm/vmalloc.c +2 -2
index bef6cf2b4d46..b01295672a31 100644
@@ -2938,7 +2938,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 		/*
 		 * Higher order nofail allocations are really expensive and
 		 * potentially dangerous (pre-mature OOM, disruptive reclaim
-		 * and compaction etc.
+		 * and compaction etc).
 		 */
 		alloc_gfp &= ~__GFP_NOFAIL;
 		nofail = true;
@@ -2965,7 +2965,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
 
 		/*
 		 * Higher order allocations must be able to be treated as
-		 * indepdenent small pages by callers (as they can with
+		 * independent small pages by callers (as they can with
 		 * small-page vmallocs). Some drivers do their own refcounting
 		 * on vmalloc_to_page() pages, some use page->mapping,
 		 * page->lru, etc.
Signature · 2 lines
Michal Hocko
SUSE Labs
#8 lore raw

Re: [PATCH] mm, vmalloc: fix high order __GFP_NOFAIL allocations

In reply to Michal Hocko

n 03/06/23 at 03:03pm, Michal Hocko wrote:
On Mon 06-03-23 13:14:43, Uladzislau Rezki wrote:
[...]

Quoted 4 lines

Some questions:

1. Could you please add a comment why you want the bulk_gfp without the __GFP_NOFAIL(bulk path)?

The bulk allocator is not documented to fully support __GFP_NOFAIL semantic IIRC. While it uses alloc_pages as fallback I didn't want to make any assumptions based on the current implementation. At least that is my recollection. If we do want to support NOFAIL by the batch allocator then we can drop the special casing here.

Quoted 2 lines

2. Could you please add a comment why a high order pages do not want __GFP_NOFAIL? You have already explained.

See below

Quoted 10 lines

3. Looking at the patch:

<snip>

mm/vmalloc.c +3 -0
+       } else {
+               alloc_gfp &= ~__GFP_NOFAIL;
+               nofail = true;
<snip>

if user does not want to go with __GFP_NOFAIL flag why you force it in case a high order allocation fails and you switch to 0 order allocations?

Not intended. The above should have been else if (gfp & __GFP_NOFAIL). Thanks for catching that!

This would be the full patch with the description:
---
From 3ccfaa15bf2587b8998c129533a0404fedf5a484 Mon Sep 17 00:00:00 2001
From: Michal Hocko <[email protected]>
Date: Mon, 6 Mar 2023 09:15:17 +0100
Subject: [PATCH] mm, vmalloc: fix high order __GFP_NOFAIL allocations

Gao Xiang has reported that the page allocator complains about high order __GFP_NOFAIL request coming from the vmalloc core:

  __alloc_pages+0x1cb/0x5b0 mm/page_alloc.c:5549
  alloc_pages+0x1aa/0x270 mm/mempolicy.c:2286
  vm_area_alloc_pages mm/vmalloc.c:2989 [inline]
  __vmalloc_area_node mm/vmalloc.c:3057 [inline]
  __vmalloc_node_range+0x978/0x13c0 mm/vmalloc.c:3227
  kvmalloc_node+0x156/0x1a0 mm/util.c:606
  kvmalloc include/linux/slab.h:737 [inline]
  kvmalloc_array include/linux/slab.h:755 [inline]
  kvcalloc include/linux/slab.h:760 [inline]

it seems that I have completely missed high order allocation backing vmalloc areas case when implementing __GFP_NOFAIL support. This means that [k]vmalloc at al. can allocate higher order allocations with __GFP_NOFAIL which can trigger OOM killer for non-costly orders easily or cause a lot of reclaim/compaction activity if those requests cannot be satisfied.

Fix the issue by falling back to zero order allocations for __GFP_NOFAIL requests if the high order request fails.

Fixes 9376130c390a ("mm/vmalloc: add support for __GFP_NOFAIL")

---

  mm/vmalloc.c | 28 +++++++++++++++++++++++-----
  1 file changed, 23 insertions(+), 5 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index ef910bf349e1..bef6cf2b4d46 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -2883,6 +2883,8 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
unsigned int order, unsigned int nr_pages, struct page **pages)
{
unsigned int nr_allocated = 0;
+ gfp_t alloc_gfp = gfp;
+ bool nofail = false;
struct page *page;
int i;

@@ -2893,6 +2895,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
* more permissive.
*/
if (!order) {
+ /* bulk allocator doesn't support nofail req. officially */
gfp_t bulk_gfp = gfp & ~__GFP_NOFAIL;

while (nr_allocated < nr_pages) {
@@ -2931,20 +2934,35 @@ vm_area_alloc_pages(gfp_t gfp, int nid,
if (nr != nr_pages_request)
break;
}
+ } else if (gfp & __GFP_NOFAIL) {
+ /*
+ * Higher order nofail allocations are really expensive and
+ * potentially dangerous (pre-mature OOM, disruptive reclaim
+ * and compaction etc.
+ */
+ alloc_gfp &= ~__GFP_NOFAIL;
+ nofail = true;
}

/* High-order pages or fallback path if "bulk" fails. */
-
while (nr_allocated < nr_pages) {
if (fatal_signal_pending(current))
break;

if (nid == NUMA_NO_NODE)
- page = alloc_pages(gfp, order);
+ page = alloc_pages(alloc_gfp, order);
else
- page = alloc_pages_node(nid, gfp, order);
- if (unlikely(!page))
- break;
+ page = alloc_pages_node(nid, alloc_gfp, order);
+ if (unlikely(!page)) {
+ if (!nofail)
+ break;
+
+ /* fall back to the zero order allocations */
+ alloc_gfp |= __GFP_NOFAIL;
+ order = 0;
+ continue;
+ }
+
/*
* Higher order allocations must be able to be treated as
* indepdenent small pages by callers (as they can with