Skip to contentSkip to conversation

percpu: improve allocation success rate for non-GFP_KERNEL callers

PATCH Applied Tahsin Erdogan 4 participants 19 messages Feb 25, 2017 – Feb 27, 2017

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 3/3 percpu: improve allocation success rate for non-GFP_KERNEL callers

When pcpu_alloc() is called with gfp != GFP_KERNEL, the likelihood of a failure is higher than GFP_KERNEL case. This is mainly because pcpu_alloc() relies on previously allocated reserves and does not make an effort to add memory to its pools for non-GFP_KERNEL case.

This issue is somewhat mitigated by kicking off a background work when a memory allocation failure occurs. But this doesn't really help the original victim of allocation failure.

This problem affects blkg_lookup_create() callers on machines with a lot of cpus.

This patch reduces failure cases by trying to expand the memory pools. It passes along gfp flag so it is safe to allocate memory this way.

To make this work, a gfp flag aware vmalloc_gfp() function is added. Also, locking around vmap_area_lock has been updated to save/restore irq flags. This was needed to avoid a lockdep problem between request_queue->queue_lock and vmap_area_lock.

---
 include/linux/vmalloc.h |   5 +-
 mm/percpu-km.c          |   8 +--
 mm/percpu-vm.c          | 119 +++++++++++-------------------------
 mm/percpu.c             | 156 ++++++++++++++++++++++++++++--------------------
 mm/vmalloc.c            |  74 ++++++++++++++---------
 5 files changed, 179 insertions(+), 183 deletions(-)
include/linux/vmalloc.h +3 -2
index d68edffbf142..8110a0040b9d 100644
@@ -72,6 +72,7 @@ extern void *vzalloc(unsigned long size);
 extern void *vmalloc_user(unsigned long size);
 extern void *vmalloc_node(unsigned long size, int node);
 extern void *vzalloc_node(unsigned long size, int node);
+extern void *vmalloc_gfp(unsigned long size, gfp_t gfp_mask);
 extern void *vmalloc_exec(unsigned long size);
 extern void *vmalloc_32(unsigned long size);
 extern void *vmalloc_32_user(unsigned long size);
@@ -165,14 +166,14 @@ extern __init void vm_area_register_early(struct vm_struct *vm, size_t align);
 # ifdef CONFIG_MMU
 struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
 				     const size_t *sizes, int nr_vms,
-				     size_t align);
+				     size_t align, gfp_t gfp_mask);
 
 void pcpu_free_vm_areas(struct vm_struct **vms, int nr_vms);
 # else
 static inline struct vm_struct **
 pcpu_get_vm_areas(const unsigned long *offsets,
 		const size_t *sizes, int nr_vms,
-		size_t align)
+		size_t align, gfp_t gfp_mask)
 {
 	return NULL;
 }
mm/percpu-km.c +4 -4
index d66911ff42d9..599a9ce84544 100644
@@ -34,7 +34,7 @@
 #include <linux/log2.h>
 
 static int pcpu_populate_chunk(struct pcpu_chunk *chunk,
-			       int page_start, int page_end)
+			       int page_start, int page_end, gfp_t gfp)
 {
 	return 0;
 }
@@ -45,18 +45,18 @@ static void pcpu_depopulate_chunk(struct pcpu_chunk *chunk,
 	/* nada */
 }
 
-static struct pcpu_chunk *pcpu_create_chunk(void)
+static struct pcpu_chunk *pcpu_create_chunk(gfp_t gfp)
 {
 	const int nr_pages = pcpu_group_sizes[0] >> PAGE_SHIFT;
 	struct pcpu_chunk *chunk;
 	struct page *pages;
 	int i;
 
-	chunk = pcpu_alloc_chunk();
+	chunk = pcpu_alloc_chunk(gfp);
 	if (!chunk)
 		return NULL;
 
-	pages = alloc_pages(GFP_KERNEL, order_base_2(nr_pages));
+	pages = alloc_pages(gfp, order_base_2(nr_pages));
 	if (!pages) {
 		pcpu_free_chunk(chunk);
 		return NULL;
mm/percpu-vm.c +34 -85

191 lines not on this page · raw message

mm/percpu.c +92 -64

365 lines not on this page · raw message

mm/vmalloc.c +46 -28

243 lines not on this page · raw message

Signature · 7 lines
2.11.0.483.g087da7b7c-goog

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#1 lore raw
reproduce: wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # save the attached .config to linux build tree…

Hi Tahsin,

[auto build test ERROR on mmotm/master]
[also build test ERROR on v4.10 next-20170224]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Tahsin-Erdogan/percpu-remove-unused-chunk_alloc-parameter-from-pcpu_get_pages/20170226-052515
base:   git://git.cmpxchg.org/linux-mmotm.git master
config: sh-rsk7269_defconfig (attached as .config)
compiler: sh4-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705

reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=sh

All errors (new ones prefixed by >>):

mm/built-in.o: In function `pcpu_mem_zalloc':

>> percpu.c:(.text+0x12670): undefined reference to `vmalloc_gfp'

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#2 lore raw
In reply to Tahsin Erdogan
reproduce: wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # save the attached .config to linux build tree…

Hi Tahsin,

[auto build test ERROR on mmotm/master]
[also build test ERROR on v4.10 next-20170224]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Tahsin-Erdogan/percpu-remove-unused-chunk_alloc-parameter-from-pcpu_get_pages/20170226-052515
base:   git://git.cmpxchg.org/linux-mmotm.git master
config: blackfin-BF561-EZKIT-SMP_defconfig (attached as .config)
compiler: bfin-uclinux-gcc (GCC) 6.2.0

reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=blackfin

All errors (new ones prefixed by >>):

mm/built-in.o: In function `pcpu_mem_zalloc':

>> mm/percpu.c:314: undefined reference to `vmalloc_gfp'

vim +314 mm/percpu.c

308 if (WARN_ON_ONCE(!slab_is_available()))
309 return NULL;
310
311 if (size <= PAGE_SIZE)
312 return kzalloc(size, gfp);
313 else

 > 314			return vmalloc_gfp(size, gfp | __GFP_HIGHMEM | __GFP_ZERO);

315 }
316
317 /**

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#3 lore raw

PATCH v2 3/3 percpu: improve allocation success rate for non-GFP_KERNEL callers

When pcpu_alloc() is called with gfp != GFP_KERNEL, the likelihood of a failure is higher than GFP_KERNEL case. This is mainly because pcpu_alloc() relies on previously allocated reserves and does not make an effort to add memory to its pools for non-GFP_KERNEL case.

This issue is somewhat mitigated by kicking off a background work when a memory allocation failure occurs. But this doesn't really help the original victim of allocation failure.

This problem affects blkg_lookup_create() callers on machines with a lot of cpus.

This patch reduces failure cases by trying to expand the memory pools. It passes along gfp flag so it is safe to allocate memory this way.

To make this work, a gfp flag aware vmalloc_gfp() function is added. Also, locking around vmap_area_lock has been updated to save/restore irq flags. This was needed to avoid a lockdep problem between request_queue->queue_lock and vmap_area_lock.

---
v2:
added vmalloc_gfp() to mm/nommu.c as well

 include/linux/vmalloc.h |   5 +-
 mm/nommu.c              |   5 ++
 mm/percpu-km.c          |   8 +--
 mm/percpu-vm.c          | 119 +++++++++++-------------------------
 mm/percpu.c             | 156 ++++++++++++++++++++++++++++--------------------
 mm/vmalloc.c            |  74 ++++++++++++++---------
 6 files changed, 184 insertions(+), 183 deletions(-)
include/linux/vmalloc.h +3 -2
index d68edffbf142..8110a0040b9d 100644
@@ -72,6 +72,7 @@ extern void *vzalloc(unsigned long size);
 extern void *vmalloc_user(unsigned long size);
 extern void *vmalloc_node(unsigned long size, int node);
 extern void *vzalloc_node(unsigned long size, int node);
+extern void *vmalloc_gfp(unsigned long size, gfp_t gfp_mask);
 extern void *vmalloc_exec(unsigned long size);
 extern void *vmalloc_32(unsigned long size);
 extern void *vmalloc_32_user(unsigned long size);
@@ -165,14 +166,14 @@ extern __init void vm_area_register_early(struct vm_struct *vm, size_t align);
 # ifdef CONFIG_MMU
 struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
 				     const size_t *sizes, int nr_vms,
-				     size_t align);
+				     size_t align, gfp_t gfp_mask);
 
 void pcpu_free_vm_areas(struct vm_struct **vms, int nr_vms);
 # else
 static inline struct vm_struct **
 pcpu_get_vm_areas(const unsigned long *offsets,
 		const size_t *sizes, int nr_vms,
-		size_t align)
+		size_t align, gfp_t gfp_mask)
 {
 	return NULL;
 }
mm/nommu.c +5 -0
index bc964c26be8c..e81d4724ac07 100644
@@ -359,6 +359,11 @@ void *vzalloc_node(unsigned long size, int node)
 }
 EXPORT_SYMBOL(vzalloc_node);
 
+void *vmalloc_gfp(unsigned long size, gfp_t gfp_mask)
+{
+	return __vmalloc(size, gfp_mask, PAGE_KERNEL);
+}
+
 #ifndef PAGE_KERNEL_EXEC
 # define PAGE_KERNEL_EXEC PAGE_KERNEL
 #endif
mm/percpu-km.c +4 -4
index d66911ff42d9..599a9ce84544 100644
@@ -34,7 +34,7 @@
 #include <linux/log2.h>
 
 static int pcpu_populate_chunk(struct pcpu_chunk *chunk,
-			       int page_start, int page_end)
+			       int page_start, int page_end, gfp_t gfp)
 {
 	return 0;
 }
@@ -45,18 +45,18 @@ static void pcpu_depopulate_chunk(struct pcpu_chunk *chunk,
 	/* nada */
 }
 
-static struct pcpu_chunk *pcpu_create_chunk(void)
+static struct pcpu_chunk *pcpu_create_chunk(gfp_t gfp)
 {
 	const int nr_pages = pcpu_group_sizes[0] >> PAGE_SHIFT;
 	struct pcpu_chunk *chunk;
 	struct page *pages;
 	int i;
 
-	chunk = pcpu_alloc_chunk();
+	chunk = pcpu_alloc_chunk(gfp);
 	if (!chunk)
 		return NULL;
 
-	pages = alloc_pages(GFP_KERNEL, order_base_2(nr_pages));
+	pages = alloc_pages(gfp, order_base_2(nr_pages));
 	if (!pages) {
 		pcpu_free_chunk(chunk);
 		return NULL;
mm/percpu-vm.c +34 -85

191 lines not on this page · raw message

mm/percpu.c +92 -64

365 lines not on this page · raw message

mm/vmalloc.c +46 -28

243 lines not on this page · raw message

Signature · 7 lines
2.11.0.483.g087da7b7c-goog

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#4 lore raw
Quoted 4 lines · Tahsin Erdogan

When pcpu_alloc() is called with gfp != GFP_KERNEL, the likelihood of a failure is higher than GFP_KERNEL case. This is mainly because pcpu_alloc() relies on previously allocated reserves and does not make an effort to add memory to its pools for non-GFP_KERNEL case.

Who is going to use a different mask?

Quoted 7 lines

This issue is somewhat mitigated by kicking off a background work when a memory allocation failure occurs. But this doesn't really help the original victim of allocation failure.

This problem affects blkg_lookup_create() callers on machines with a lot of cpus.

Quoted 7 lines

This patch reduces failure cases by trying to expand the memory pools. It passes along gfp flag so it is safe to allocate memory this way.

To make this work, a gfp flag aware vmalloc_gfp() function is added. Also, locking around vmap_area_lock has been updated to save/restore irq flags. This was needed to avoid a lockdep problem between request_queue->queue_lock and vmap_area_lock.

We already have __vmalloc_gfp, why this cannot be used? Also note that vmalloc dosn't really support arbitrary gfp flags. One have to be really careful because there are some internal allocations which are hardcoded GFP_KERNEL. Also this patch doesn't really add any new callers so it is hard to tell whether what you do actually makes sense and is correct.

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#5 lore raw
Quoted 7 lines · Michal Hocko
Quoted 4 lines · Tahsin Erdogan

When pcpu_alloc() is called with gfp != GFP_KERNEL, the likelihood of a failure is higher than GFP_KERNEL case. This is mainly because pcpu_alloc() relies on previously allocated reserves and does not make an effort to add memory to its pools for non-GFP_KERNEL case.

Who is going to use a different mask?

blkg_create() makes a call with a non-GFP_KERNEL mask:
new_blkg = blkg_alloc(blkcg, q, GFP_NOWAIT | __GFP_NOWARN);

which turns into a call stack like below:

__vmalloc+0x45/0x50
pcpu_mem_zalloc+0x50/0x80
pcpu_populate_chunk+0x3b/0x380
pcpu_alloc+0x588/0x6e0
__alloc_percpu_gfp+0xd/0x10
__percpu_counter_init+0x55/0xc0
blkg_alloc+0x76/0x230
blkg_create+0x489/0x670
blkg_lookup_create+0x9a/0x230
generic_make_request_checks+0x7dd/0x890
generic_make_request+0x1f/0x180
submit_bio+0x61/0x120
Quoted 5 lines

We already have __vmalloc_gfp, why this cannot be used? Also note that vmalloc dosn't really support arbitrary gfp flags. One have to be really careful because there are some internal allocations which are hardcoded GFP_KERNEL. Also this patch doesn't really add any new callers so it is hard to tell whether what you do actually makes sense and is correct.

Did you mean to say __vmalloc? If so, yes, I should use that.

By the way, I now noticed the might_sleep() in alloc_vmap_area() which makes it unsafe to call vmalloc* in GFP_ATOMIC contexts. It was added recently:

commit 5803ed292e63 ("mm: mark all calls into the vmalloc subsystem as potentially sleeping")

Any suggestions on how to deal with that? For instance, would it be safe to replace it with:

might_sleep_if(gfpflags_allow_blocking(gfp_mask));

and then skip purge_vmap_area_lazy() if blocking is not allowed?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw
Quoted 14 lines · Tahsin Erdogan
Quoted 7 lines · Michal Hocko
Quoted 4 lines · Tahsin Erdogan

When pcpu_alloc() is called with gfp != GFP_KERNEL, the likelihood of a failure is higher than GFP_KERNEL case. This is mainly because pcpu_alloc() relies on previously allocated reserves and does not make an effort to add memory to its pools for non-GFP_KERNEL case.

Who is going to use a different mask?

blkg_create() makes a call with a non-GFP_KERNEL mask:

   new_blkg = blkg_alloc(blkcg, q, GFP_NOWAIT | __GFP_NOWARN);

which turns into a call stack like below:

Quoted 12 lines
__vmalloc+0x45/0x50
pcpu_mem_zalloc+0x50/0x80
pcpu_populate_chunk+0x3b/0x380
pcpu_alloc+0x588/0x6e0
__alloc_percpu_gfp+0xd/0x10
__percpu_counter_init+0x55/0xc0
blkg_alloc+0x76/0x230
blkg_create+0x489/0x670
blkg_lookup_create+0x9a/0x230
generic_make_request_checks+0x7dd/0x890
generic_make_request+0x1f/0x180
submit_bio+0x61/0x120

OK, I see. Thanks for the clarification. I am not familiar with the pcp allocator much, but we have

	/*
	 * No space left.  Create a new chunk.  We don't want multiple
	 * tasks to create chunks simultaneously.  Serialize and create iff
	 * there's still no empty chunk after grabbing the mutex.
	 */
	if (is_atomic)
		goto fail;

right before pcpu_populate_chunk so is this actually a problem?

Quoted 7 lines
Quoted 5 lines

We already have __vmalloc_gfp, why this cannot be used? Also note that vmalloc dosn't really support arbitrary gfp flags. One have to be really careful because there are some internal allocations which are hardcoded GFP_KERNEL. Also this patch doesn't really add any new callers so it is hard to tell whether what you do actually makes sense and is correct.

Did you mean to say __vmalloc? If so, yes, I should use that.

yeah

Quoted 2 lines

By the way, I now noticed the might_sleep() in alloc_vmap_area() which makes it unsafe to call vmalloc* in GFP_ATOMIC contexts. It was added recently:

Do we call alloc_vmap_area from true atomic contexts (aka from under spinlocks etc)? I thought this was a nogo and GFP_NOWAIT resp. GFP_ATOMIC was more about optimistic request resp. access to memory reserves rather than true atomicity requirements.

Quoted 9 lines

commit 5803ed292e63 ("mm: mark all calls into the vmalloc subsystem as potentially sleeping")

Any suggestions on how to deal with that? For instance, would it be safe to replace it with:

might_sleep_if(gfpflags_allow_blocking(gfp_mask));

and then skip purge_vmap_area_lazy() if blocking is not allowed?

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#7 lore raw
Quoted 9 lines · Michal Hocko

/*
* No space left. Create a new chunk. We don't want multiple
* tasks to create chunks simultaneously. Serialize and create iff
* there's still no empty chunk after grabbing the mutex. */
if (is_atomic)
goto fail;

right before pcpu_populate_chunk so is this actually a problem?

Yes, this prevents adding more pcpu chunks and so cause "atomic" allocations to fail more easily.

Quoted 7 lines
Quoted 2 lines

By the way, I now noticed the might_sleep() in alloc_vmap_area() which makes it unsafe to call vmalloc* in GFP_ATOMIC contexts. It was added recently:

Do we call alloc_vmap_area from true atomic contexts (aka from under spinlocks etc)? I thought this was a nogo and GFP_NOWAIT resp. GFP_ATOMIC was more about optimistic request resp. access to memory reserves rather than true atomicity requirements.

In the call path that I am trying to fix, the caller uses GFP_NOWAIT mask. The caller is holding a spinlock (request_queue->queue_lock) so we can't afford to sleep.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#8 lore raw
Quoted 9 lines · Tahsin Erdogan
Quoted 8 lines · Michal Hocko

/*
* No space left. Create a new chunk. We don't want multiple
* tasks to create chunks simultaneously. Serialize and create iff
* there's still no empty chunk after grabbing the mutex. */
if (is_atomic)
goto fail;

Quoted 4 lines
Quoted 1 line

right before pcpu_populate_chunk so is this actually a problem?

Yes, this prevents adding more pcpu chunks and so cause "atomic" allocations to fail more easily.

Then I fail to see what is the problem you are trying to fix.

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#9 lore raw
Quoted 10 lines · Michal Hocko
Quoted 9 lines · Tahsin Erdogan
Quoted 8 lines · Michal Hocko

/*
* No space left. Create a new chunk. We don't want multiple
* tasks to create chunks simultaneously. Serialize and create iff
* there's still no empty chunk after grabbing the mutex. */
if (is_atomic)
goto fail;

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

right before pcpu_populate_chunk so is this actually a problem?

Yes, this prevents adding more pcpu chunks and so cause "atomic" allocations to fail more easily.

Then I fail to see what is the problem you are trying to fix.

To be more specific. Could you describe what more can we do in the vmalloc layer for GFP_NOWAIT allocations? They certainly cannot sleep and cannot perform the reclaim so you have to rely on the background work.

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#10 lore raw
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

Yes, this prevents adding more pcpu chunks and so cause "atomic" allocations to fail more easily.

Then I fail to see what is the problem you are trying to fix.

To be more specific. Could you describe what more can we do in the vmalloc layer for GFP_NOWAIT allocations? They certainly cannot sleep and cannot perform the reclaim so you have to rely on the background work.

The main problem that I am trying to fix is in percpu.c code. It currently doesn't
even attempt to call vmalloc() for GFP_NOWAIT case. It solely relies on the background allocator to replenish the reserves. I would like percpu.c to call __vmalloc(GFP_NOWAIT) inline and see whether that succeeds. If that fails, it is fair to fail the call.

For this to work, __vmalloc() should be ready to serve a caller that is holding a
spinlock. The might_sleep() in alloc_vmap_area() basically prevents us calling vmalloc in this context.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#11 lore raw
Quoted 11 lines · Tahsin Erdogan
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

Yes, this prevents adding more pcpu chunks and so cause "atomic" allocations to fail more easily.

Then I fail to see what is the problem you are trying to fix.

To be more specific. Could you describe what more can we do in the vmalloc layer for GFP_NOWAIT allocations? They certainly cannot sleep and cannot perform the reclaim so you have to rely on the background work.

Quoted 6 lines

The main problem that I am trying to fix is in percpu.c code. It currently doesn't even attempt to call vmalloc() for GFP_NOWAIT case. It solely relies on the background allocator to replenish the reserves. I would like percpu.c to call __vmalloc(GFP_NOWAIT) inline and see whether that succeeds. If that fails, it is fair to fail the call.

OK, that wasn't really clean from the patch to me. I guess it would be much more easier if a preparatory patch did the gfp mask propagation and then have patch that changes the pcpu allocator the way you need.

Quoted 3 lines

For this to work, __vmalloc() should be ready to serve a caller that is holding a spinlock. The might_sleep() in alloc_vmap_area() basically prevents us calling vmalloc in this context.

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer
#12 lore raw
In reply to Tahsin Erdogan

Hello,

Quoted 14 lines · Tahsin Erdogan
Quoted 7 lines · Michal Hocko
Quoted 4 lines · Tahsin Erdogan

When pcpu_alloc() is called with gfp != GFP_KERNEL, the likelihood of a failure is higher than GFP_KERNEL case. This is mainly because pcpu_alloc() relies on previously allocated reserves and does not make an effort to add memory to its pools for non-GFP_KERNEL case.

Who is going to use a different mask?

blkg_create() makes a call with a non-GFP_KERNEL mask:

   new_blkg = blkg_alloc(blkcg, q, GFP_NOWAIT | __GFP_NOWARN);

which turns into a call stack like below:

Quoted 12 lines
__vmalloc+0x45/0x50
pcpu_mem_zalloc+0x50/0x80
pcpu_populate_chunk+0x3b/0x380
pcpu_alloc+0x588/0x6e0
__alloc_percpu_gfp+0xd/0x10
__percpu_counter_init+0x55/0xc0
blkg_alloc+0x76/0x230
blkg_create+0x489/0x670
blkg_lookup_create+0x9a/0x230
generic_make_request_checks+0x7dd/0x890
generic_make_request+0x1f/0x180
submit_bio+0x61/0x120

As indicated by GFP_NOWAIT | __GFP_NOWARN, it's okay to fail there. It's not okay to fail consistently for a long time but it's not a big issue to fail occassionally even if somewhat bunched up. The only bad side effect of that is temporary misaccounting of some IOs, which shouldn't be noticeable outside of pathological cases. If you're actually seeing adverse effects of this, I'd love to learn about it.

Quoted 7 lines
Quoted 5 lines

We already have __vmalloc_gfp, why this cannot be used? Also note that vmalloc dosn't really support arbitrary gfp flags. One have to be really careful because there are some internal allocations which are hardcoded GFP_KERNEL. Also this patch doesn't really add any new callers so it is hard to tell whether what you do actually makes sense and is correct.

Did you mean to say __vmalloc? If so, yes, I should use that.

So, the last time I looked at it the thorny ones in that path are the page table (pgd, pud...) allocation functions. There are several layers of indirection there but they end up in arch-specific implemntations which hard code GFP_KERNEL. Without fixing them up, we can't guarantee mapping the allocated pages making things kinda moot.

The only reason percpu allocator has the background allocator stuff is vmalloc path can't do non-blocking allocations. If we can properly fix that up, we can get rid of all those code from percpu allocator and simply path the gfp flag to vmap functions. Please take a look at __pcpu_map_pages() in mm/percpu-vm.c. map_kernel_range_noflush() is the function which has implicit GFP_KERNEL allocation in it and what's requiring the reserve.

If you can get rid of that, awesome, but given that your patch doesn't touch that at all, I can't see how it's supposed to work.

Thanks.

Signature · 7 lines
tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#13 lore raw

Hi Tejun,

Quoted 13 lines · Tejun Heo
Quoted 4 lines
__vmalloc+0x45/0x50
pcpu_mem_zalloc+0x50/0x80
pcpu_populate_chunk+0x3b/0x380
pcpu_alloc+0x588/0x6e0
Quoted 8 lines
__alloc_percpu_gfp+0xd/0x10
__percpu_counter_init+0x55/0xc0
blkg_alloc+0x76/0x230
blkg_create+0x489/0x670
blkg_lookup_create+0x9a/0x230
generic_make_request_checks+0x7dd/0x890
generic_make_request+0x1f/0x180
submit_bio+0x61/0x120
Quoted 6 lines

As indicated by GFP_NOWAIT | __GFP_NOWARN, it's okay to fail there. It's not okay to fail consistently for a long time but it's not a big issue to fail occassionally even if somewhat bunched up. The only bad side effect of that is temporary misaccounting of some IOs, which shouldn't be noticeable outside of pathological cases. If you're actually seeing adverse effects of this, I'd love to learn about it.

A better example is the call path below:

pcpu_alloc+0x68f/0x710
__alloc_percpu_gfp+0xd/0x10
__percpu_counter_init+0x55/0xc0
cfq_pd_alloc+0x3b2/0x4e0
blkg_alloc+0x187/0x230
blkg_create+0x489/0x670
blkg_lookup_create+0x9a/0x230
blkg_conf_prep+0x1fb/0x240
__cfqg_set_weight_device.isra.105+0x5c/0x180
cfq_set_weight_on_dfl+0x69/0xc0
cgroup_file_write+0x39/0x1c0
kernfs_fop_write+0x13f/0x1d0
__vfs_write+0x23/0x120
vfs_write+0xc2/0x1f0
SyS_write+0x44/0xb0
entry_SYSCALL_64_fastpath+0x18/0xad

A failure in this call path gives grief to tools which are trying to configure io
weights. We see occasional failures happen here shortly after reboots even when system is not under any memory pressure. Machines with a lot of cpus are obviously more vulnerable.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

maintainer
#14 lore raw

Hello,

Quoted 19 lines · Tahsin Erdogan

A better example is the call path below:

pcpu_alloc+0x68f/0x710
__alloc_percpu_gfp+0xd/0x10
__percpu_counter_init+0x55/0xc0
cfq_pd_alloc+0x3b2/0x4e0
blkg_alloc+0x187/0x230
blkg_create+0x489/0x670
blkg_lookup_create+0x9a/0x230
blkg_conf_prep+0x1fb/0x240
__cfqg_set_weight_device.isra.105+0x5c/0x180
cfq_set_weight_on_dfl+0x69/0xc0
cgroup_file_write+0x39/0x1c0
kernfs_fop_write+0x13f/0x1d0
__vfs_write+0x23/0x120
vfs_write+0xc2/0x1f0
SyS_write+0x44/0xb0
entry_SYSCALL_64_fastpath+0x18/0xad
Quoted 5 lines

A failure in this call path gives grief to tools which are trying to configure io
weights. We see occasional failures happen here shortly after reboots even when system is not under any memory pressure. Machines with a lot of cpus are obviously more vulnerable.

Ah, absolutely, that's a stupid failure but we should be able to fix that by making the blkg functions take gfp mask and allocate accordingly, right? It'll probably take preallocation tricks because of locking but should be doable.

Thanks.

Signature · 7 lines
tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#15 lore raw
Quoted 28 lines · Tejun Heo

Hello,

Quoted 19 lines · Tahsin Erdogan

A better example is the call path below:

pcpu_alloc+0x68f/0x710
__alloc_percpu_gfp+0xd/0x10
__percpu_counter_init+0x55/0xc0
cfq_pd_alloc+0x3b2/0x4e0
blkg_alloc+0x187/0x230
blkg_create+0x489/0x670
blkg_lookup_create+0x9a/0x230
blkg_conf_prep+0x1fb/0x240
__cfqg_set_weight_device.isra.105+0x5c/0x180
cfq_set_weight_on_dfl+0x69/0xc0
cgroup_file_write+0x39/0x1c0
kernfs_fop_write+0x13f/0x1d0
__vfs_write+0x23/0x120
vfs_write+0xc2/0x1f0
SyS_write+0x44/0xb0
entry_SYSCALL_64_fastpath+0x18/0xad
Quoted 5 lines

A failure in this call path gives grief to tools which are trying to configure io
weights. We see occasional failures happen here shortly after reboots even when system is not under any memory pressure. Machines with a lot of cpus are obviously more vulnerable.

Quoted 4 lines

Ah, absolutely, that's a stupid failure but we should be able to fix that by making the blkg functions take gfp mask and allocate accordingly, right? It'll probably take preallocation tricks because of locking but should be doable.

My initial goal was to allow calls to vmalloc(), but I now see the challenges in that
approach.

Doing preallocations would probably work but not sure if that can be done without
complicating code too much. Could you describe what you have in mind?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

maintainer
#16 lore raw

Hello, Tahsin.

Quoted 8 lines · Tahsin Erdogan
Quoted 4 lines

Ah, absolutely, that's a stupid failure but we should be able to fix that by making the blkg functions take gfp mask and allocate accordingly, right? It'll probably take preallocation tricks because of locking but should be doable.

My initial goal was to allow calls to vmalloc(), but I now see the challenges in that
approach.

I'd love to see that working too but this is a different issue. Even GFP_ATOMIC can fail under pressure and it's kinda wrong to depend on that for userspace interactions.

Quoted 3 lines

Doing preallocations would probably work but not sure if that can be done without
complicating code too much. Could you describe what you have in mind?

So, blkg_create() already takes @new_blkg argument which is the preallocated blkg and used during q init. Wouldn't it work to make blkg_lookup_create() take @new_blkg too and pass it down to blkg_create() (and also free it if it doesn't get used)? Then, blkg_conf_prep() can always (or after a failure with -ENOMEM) allocate a new blkg before calling into blkg_lookup_create(). I don't think it'll complicate the code path that much.

Thanks.

Signature · 7 lines
tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#17 lore raw
Quoted 11 lines
Quoted 3 lines

Doing preallocations would probably work but not sure if that can be done without
complicating code too much. Could you describe what you have in mind?

So, blkg_create() already takes @new_blkg argument which is the preallocated blkg and used during q init. Wouldn't it work to make blkg_lookup_create() take @new_blkg too and pass it down to blkg_create() (and also free it if it doesn't get used)? Then, blkg_conf_prep() can always (or after a failure with -ENOMEM) allocate a new blkg before calling into blkg_lookup_create(). I don't think it'll complicate the code path that much.

That makes sense. I will work a patch that does that (unless you are interested in implementing it yourself).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

maintainer
#18 lore raw
Quoted 2 lines · Tahsin Erdogan

That makes sense. I will work a patch that does that (unless you are interested in implementing it yourself).

I'd really appreciate if you can work on it. Thanks a lot!

Signature · 7 lines
tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>