percpu: improve allocation success rate for non-GFP_KERNEL callers
Subsystems (4)
PATCH 3/3 percpu: improve allocation success rate for non-GFP_KERNEL callers
When pcpu_alloc() is called with gfp != GFP_KERNEL, the likelihood of a failure is higher than GFP_KERNEL case. This is mainly because pcpu_alloc() relies on previously allocated reserves and does not make an effort to add memory to its pools for non-GFP_KERNEL case.
This issue is somewhat mitigated by kicking off a background work when a memory allocation failure occurs. But this doesn't really help the original victim of allocation failure.
This problem affects blkg_lookup_create() callers on machines with a lot of cpus.
This patch reduces failure cases by trying to expand the memory pools. It passes along gfp flag so it is safe to allocate memory this way.
To make this work, a gfp flag aware vmalloc_gfp() function is added. Also, locking around vmap_area_lock has been updated to save/restore irq flags. This was needed to avoid a lockdep problem between request_queue->queue_lock and vmap_area_lock.
--- include/linux/vmalloc.h | 5 +- mm/percpu-km.c | 8 +-- mm/percpu-vm.c | 119 +++++++++++------------------------- mm/percpu.c | 156 ++++++++++++++++++++++++++++-------------------- mm/vmalloc.c | 74 ++++++++++++++--------- 5 files changed, 179 insertions(+), 183 deletions(-)
include/linux/vmalloc.h +3 -2
@@ -72,6 +72,7 @@ extern void *vzalloc(unsigned long size);extern void *vmalloc_user (unsigned long size);extern void *vmalloc_node (unsigned long size,int node);extern void *vzalloc_node (unsigned long size,int node); +extern void *vmalloc_gfp (unsigned long size,gfp_t gfp_mask);extern void *vmalloc_exec (unsigned long size);extern void *vmalloc_32 (unsigned long size);extern void *vmalloc_32_user (unsigned long size);
@@ -165,14 +166,14 @@ extern __init void vm_area_register_early(struct vm_struct *vm, size_t align);# ifdef CONFIG_MMU struct vm_struct **pcpu_get_vm_areas (const unsigned long *offsets,const size_t *sizes,int nr_vms, -size_t align); +size_t align,gfp_t gfp_mask);void pcpu_free_vm_areas (struct vm_struct **vms,int nr_vms);# else static inline struct vm_struct **pcpu_get_vm_areas (const unsigned long *offsets,const size_t *sizes,int nr_vms, -size_t align) +size_t align,gfp_t gfp_mask) {return NULL ; }
mm/percpu-km.c +4 -4
@@ -34,7 +34,7 @@#include <linux/log2.h> static int pcpu_populate_chunk (struct pcpu_chunk *chunk, -int page_start,int page_end) +int page_start,int page_end,gfp_t gfp) {return 0 ; }
@@ -45,18 +45,18 @@ static void pcpu_depopulate_chunk(struct pcpu_chunk *chunk,/* nada */ } -static struct pcpu_chunk *pcpu_create_chunk (void ) +static struct pcpu_chunk *pcpu_create_chunk (gfp_t gfp) {const int nr_pages = pcpu_group_sizes[0 ] >>PAGE_SHIFT ;struct pcpu_chunk *chunk;struct page *pages;int i; - chunk =pcpu_alloc_chunk (); + chunk =pcpu_alloc_chunk (gfp);if (!chunk)return NULL ; - pages =alloc_pages (GFP_KERNEL ,order_base_2 (nr_pages)); + pages =alloc_pages (gfp,order_base_2 (nr_pages));if (!pages) {pcpu_free_chunk (chunk);return NULL ;
mm/percpu-vm.c +34 -85
191 lines not on this page · raw message
mm/percpu.c +92 -64
365 lines not on this page · raw message
mm/vmalloc.c +46 -28
243 lines not on this page · raw message
Signature · 7 lines
2.11.0.483.g087da7b7c-goog -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
kbuild test robot
build botreproduce: wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # save the attached .config to linux build tree…
Hi Tahsin,
[auto build test ERROR on mmotm/master]
[also build test ERROR on v4.10 next-20170224]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Tahsin-Erdogan/percpu-remove-unused-chunk_alloc-parameter-from-pcpu_get_pages/20170226-052515 base: git://git.cmpxchg.org/linux-mmotm.git master config: sh-rsk7269_defconfig (attached as .config) compiler: sh4-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=sh
All errors (new ones prefixed by >>):
mm/built-in.o: In function `pcpu_mem_zalloc':
>> percpu.c:(.text+0x12670): undefined reference to `vmalloc_gfp'
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
kbuild test robot
build botreproduce: wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # save the attached .config to linux build tree…
Hi Tahsin,
[auto build test ERROR on mmotm/master]
[also build test ERROR on v4.10 next-20170224]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Tahsin-Erdogan/percpu-remove-unused-chunk_alloc-parameter-from-pcpu_get_pages/20170226-052515 base: git://git.cmpxchg.org/linux-mmotm.git master config: blackfin-BF561-EZKIT-SMP_defconfig (attached as .config) compiler: bfin-uclinux-gcc (GCC) 6.2.0
reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=blackfin
All errors (new ones prefixed by >>):
mm/built-in.o: In function `pcpu_mem_zalloc':
>> mm/percpu.c:314: undefined reference to `vmalloc_gfp'
vim +314 mm/percpu.c
308 if (WARN_ON_ONCE(!slab_is_available()))
309 return NULL;
310
311 if (size <= PAGE_SIZE)
312 return kzalloc(size, gfp);
313 else
> 314 return vmalloc_gfp(size, gfp | __GFP_HIGHMEM | __GFP_ZERO);
315 }
316
317 /**
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
PATCH v2 3/3 percpu: improve allocation success rate for non-GFP_KERNEL callers
When pcpu_alloc() is called with gfp != GFP_KERNEL, the likelihood of a failure is higher than GFP_KERNEL case. This is mainly because pcpu_alloc() relies on previously allocated reserves and does not make an effort to add memory to its pools for non-GFP_KERNEL case.
This issue is somewhat mitigated by kicking off a background work when a memory allocation failure occurs. But this doesn't really help the original victim of allocation failure.
This problem affects blkg_lookup_create() callers on machines with a lot of cpus.
This patch reduces failure cases by trying to expand the memory pools. It passes along gfp flag so it is safe to allocate memory this way.
To make this work, a gfp flag aware vmalloc_gfp() function is added. Also, locking around vmap_area_lock has been updated to save/restore irq flags. This was needed to avoid a lockdep problem between request_queue->queue_lock and vmap_area_lock.
---
v2:
added vmalloc_gfp() to mm/nommu.c as well
include/linux/vmalloc.h | 5 +- mm/nommu.c | 5 ++ mm/percpu-km.c | 8 +-- mm/percpu-vm.c | 119 +++++++++++------------------------- mm/percpu.c | 156 ++++++++++++++++++++++++++++-------------------- mm/vmalloc.c | 74 ++++++++++++++--------- 6 files changed, 184 insertions(+), 183 deletions(-)
include/linux/vmalloc.h +3 -2
@@ -72,6 +72,7 @@ extern void *vzalloc(unsigned long size);extern void *vmalloc_user (unsigned long size);extern void *vmalloc_node (unsigned long size,int node);extern void *vzalloc_node (unsigned long size,int node); +extern void *vmalloc_gfp (unsigned long size,gfp_t gfp_mask);extern void *vmalloc_exec (unsigned long size);extern void *vmalloc_32 (unsigned long size);extern void *vmalloc_32_user (unsigned long size);
@@ -165,14 +166,14 @@ extern __init void vm_area_register_early(struct vm_struct *vm, size_t align);# ifdef CONFIG_MMU struct vm_struct **pcpu_get_vm_areas (const unsigned long *offsets,const size_t *sizes,int nr_vms, -size_t align); +size_t align,gfp_t gfp_mask);void pcpu_free_vm_areas (struct vm_struct **vms,int nr_vms);# else static inline struct vm_struct **pcpu_get_vm_areas (const unsigned long *offsets,const size_t *sizes,int nr_vms, -size_t align) +size_t align,gfp_t gfp_mask) {return NULL ; }
mm/nommu.c +5 -0
@@ -359,6 +359,11 @@ void *vzalloc_node(unsigned long size, int node) }EXPORT_SYMBOL (vzalloc_node); +void *vmalloc_gfp (unsigned long size,gfp_t gfp_mask) +{ +return __vmalloc (size, gfp_mask,PAGE_KERNEL ); +} +#ifndef PAGE_KERNEL_EXEC # define PAGE_KERNEL_EXEC PAGE_KERNEL#endif
mm/percpu-km.c +4 -4
@@ -34,7 +34,7 @@#include <linux/log2.h> static int pcpu_populate_chunk (struct pcpu_chunk *chunk, -int page_start,int page_end) +int page_start,int page_end,gfp_t gfp) {return 0 ; }
@@ -45,18 +45,18 @@ static void pcpu_depopulate_chunk(struct pcpu_chunk *chunk,/* nada */ } -static struct pcpu_chunk *pcpu_create_chunk (void ) +static struct pcpu_chunk *pcpu_create_chunk (gfp_t gfp) {const int nr_pages = pcpu_group_sizes[0 ] >>PAGE_SHIFT ;struct pcpu_chunk *chunk;struct page *pages;int i; - chunk =pcpu_alloc_chunk (); + chunk =pcpu_alloc_chunk (gfp);if (!chunk)return NULL ; - pages =alloc_pages (GFP_KERNEL ,order_base_2 (nr_pages)); + pages =alloc_pages (gfp,order_base_2 (nr_pages));if (!pages) {pcpu_free_chunk (chunk);return NULL ;
mm/percpu-vm.c +34 -85
191 lines not on this page · raw message
mm/percpu.c +92 -64
365 lines not on this page · raw message
mm/vmalloc.c +46 -28
243 lines not on this page · raw message
Signature · 7 lines
2.11.0.483.g087da7b7c-goog -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 4 lines · Tahsin Erdogan
When
pcpu_alloc()is called with gfp !=GFP_KERNEL, the likelihood of a failure is higher thanGFP_KERNELcase. This is mainly becausepcpu_alloc()relies on previously allocated reserves and does not make an effort to add memory to its pools for non-GFP_KERNEL case.
Who is going to use a different mask?
Quoted 7 lines
This issue is somewhat mitigated by kicking off a background work when a memory allocation failure occurs. But this doesn't really help the original victim of allocation failure.
This problem affects
blkg_lookup_create()callers on machines with a lot of cpus.
Quoted 7 lines
This patch reduces failure cases by trying to expand the memory pools. It passes along gfp flag so it is safe to allocate memory this way.
To make this work, a gfp flag aware
vmalloc_gfp()function is added. Also, locking aroundvmap_area_lockhas been updated to save/restore irq flags. This was needed to avoid a lockdep problem betweenrequest_queue->queue_lockandvmap_area_lock.
We already have __vmalloc_gfp, why this cannot be used? Also note that vmalloc dosn't really support arbitrary gfp flags. One have to be really careful because there are some internal allocations which are hardcoded GFP_KERNEL. Also this patch doesn't really add any new callers so it is hard to tell whether what you do actually makes sense and is correct.
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · Michal Hocko
Quoted 4 lines · Tahsin Erdogan
When
pcpu_alloc()is called with gfp !=GFP_KERNEL, the likelihood of a failure is higher thanGFP_KERNELcase. This is mainly becausepcpu_alloc()relies on previously allocated reserves and does not make an effort to add memory to its pools for non-GFP_KERNEL case.Who is going to use a different mask?
blkg_create() makes a call with a non-GFP_KERNEL mask:
new_blkg = blkg_alloc(blkcg, q, GFP_NOWAIT | __GFP_NOWARN);
which turns into a call stack like below:
__vmalloc+0x45/0x50 pcpu_mem_zalloc+0x50/0x80 pcpu_populate_chunk+0x3b/0x380 pcpu_alloc+0x588/0x6e0 __alloc_percpu_gfp+0xd/0x10 __percpu_counter_init+0x55/0xc0 blkg_alloc+0x76/0x230 blkg_create+0x489/0x670 blkg_lookup_create+0x9a/0x230 generic_make_request_checks+0x7dd/0x890 generic_make_request+0x1f/0x180 submit_bio+0x61/0x120
Quoted 5 lines
We already have
__vmalloc_gfp, why this cannot be used? Also note that vmalloc dosn't really support arbitrary gfp flags. One have to be really careful because there are some internal allocations which are hardcodedGFP_KERNEL. Also this patch doesn't really add any new callers so it is hard to tell whether what you do actually makes sense and is correct.
Did you mean to say __vmalloc? If so, yes, I should use that.
By the way, I now noticed the might_sleep() in alloc_vmap_area() which makes it unsafe to call vmalloc* in GFP_ATOMIC contexts. It was added recently:
commit 5803ed292e63 ("mm: mark all calls into the vmalloc subsystem as potentially sleeping")
Any suggestions on how to deal with that? For instance, would it be safe to replace it with:
might_sleep_if(gfpflags_allow_blocking(gfp_mask));
and then skip purge_vmap_area_lazy() if blocking is not allowed?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 14 lines · Tahsin Erdogan
Quoted 7 lines · Michal Hocko
Quoted 4 lines · Tahsin Erdogan
When
pcpu_alloc()is called with gfp !=GFP_KERNEL, the likelihood of a failure is higher thanGFP_KERNELcase. This is mainly becausepcpu_alloc()relies on previously allocated reserves and does not make an effort to add memory to its pools for non-GFP_KERNEL case.Who is going to use a different mask?
blkg_create()makes a call with a non-GFP_KERNEL mask:new_blkg = blkg_alloc(blkcg, q, GFP_NOWAIT | __GFP_NOWARN);which turns into a call stack like below:
Quoted 12 lines
__vmalloc+0x45/0x50 pcpu_mem_zalloc+0x50/0x80 pcpu_populate_chunk+0x3b/0x380 pcpu_alloc+0x588/0x6e0 __alloc_percpu_gfp+0xd/0x10 __percpu_counter_init+0x55/0xc0 blkg_alloc+0x76/0x230 blkg_create+0x489/0x670 blkg_lookup_create+0x9a/0x230 generic_make_request_checks+0x7dd/0x890 generic_make_request+0x1f/0x180 submit_bio+0x61/0x120
OK, I see. Thanks for the clarification. I am not familiar with the pcp allocator much, but we have
/* * No space left. Create a new chunk. We don't want multiple * tasks to create chunks simultaneously. Serialize and create iff * there's still no empty chunk after grabbing the mutex. */ if (is_atomic) goto fail;
right before pcpu_populate_chunk so is this actually a problem?
Quoted 7 lines
Quoted 5 lines
We already have
__vmalloc_gfp, why this cannot be used? Also note that vmalloc dosn't really support arbitrary gfp flags. One have to be really careful because there are some internal allocations which are hardcodedGFP_KERNEL. Also this patch doesn't really add any new callers so it is hard to tell whether what you do actually makes sense and is correct.Did you mean to say
__vmalloc? If so, yes, I should use that.
yeah
Quoted 2 lines
By the way, I now noticed the
might_sleep()inalloc_vmap_area()which makes it unsafe to call vmalloc* inGFP_ATOMICcontexts. It was added recently:
Do we call alloc_vmap_area from true atomic contexts (aka from under spinlocks etc)? I thought this was a nogo and GFP_NOWAIT resp. GFP_ATOMIC was more about optimistic request resp. access to memory reserves rather than true atomicity requirements.
Quoted 9 lines
commit
5803ed292e63("mm: mark all calls into the vmalloc subsystem as potentially sleeping")Any suggestions on how to deal with that? For instance, would it be safe to replace it with:
might_sleep_if(gfpflags_allow_blocking(gfp_mask));
and then skip
purge_vmap_area_lazy()if blocking is not allowed?
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 9 lines · Michal Hocko
/*
* No space left. Create a new chunk. We don't want multiple
* tasks to create chunks simultaneously. Serialize and create iff
* there's still no empty chunk after grabbing the mutex. */
if (is_atomic)
goto fail;right before
pcpu_populate_chunkso is this actually a problem?
Yes, this prevents adding more pcpu chunks and so cause "atomic" allocations to fail more easily.
Quoted 7 lines
Quoted 2 lines
By the way, I now noticed the
might_sleep()inalloc_vmap_area()which makes it unsafe to call vmalloc* inGFP_ATOMICcontexts. It was added recently:Do we call
alloc_vmap_areafrom true atomic contexts (aka from under spinlocks etc)? I thought this was a nogo andGFP_NOWAITresp.GFP_ATOMICwas more about optimistic request resp. access to memory reserves rather than true atomicity requirements.
In the call path that I am trying to fix, the caller uses GFP_NOWAIT mask. The caller is holding a spinlock (request_queue->queue_lock) so we can't afford to sleep.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 9 lines · Tahsin Erdogan
Quoted 8 lines · Michal Hocko
/*
* No space left. Create a new chunk. We don't want multiple
* tasks to create chunks simultaneously. Serialize and create iff
* there's still no empty chunk after grabbing the mutex. */
if (is_atomic)
goto fail;
Quoted 4 lines
Quoted 1 line
right before
pcpu_populate_chunkso is this actually a problem?Yes, this prevents adding more pcpu chunks and so cause "atomic" allocations to fail more easily.
Then I fail to see what is the problem you are trying to fix.
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines · Michal Hocko
Quoted 9 lines · Tahsin Erdogan
Quoted 8 lines · Michal Hocko
/*
* No space left. Create a new chunk. We don't want multiple
* tasks to create chunks simultaneously. Serialize and create iff
* there's still no empty chunk after grabbing the mutex. */
if (is_atomic)
goto fail;
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
right before
pcpu_populate_chunkso is this actually a problem?Yes, this prevents adding more pcpu chunks and so cause "atomic" allocations to fail more easily.
Then I fail to see what is the problem you are trying to fix.
To be more specific. Could you describe what more can we do in the vmalloc layer for GFP_NOWAIT allocations? They certainly cannot sleep and cannot perform the reclaim so you have to rely on the background work.
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
Yes, this prevents adding more pcpu chunks and so cause "atomic" allocations to fail more easily.
Then I fail to see what is the problem you are trying to fix.
To be more specific. Could you describe what more can we do in the vmalloc layer for
GFP_NOWAITallocations? They certainly cannot sleep and cannot perform the reclaim so you have to rely on the background work.
The main problem that I am trying to fix is in percpu.c code. It currently doesn't
even attempt to call vmalloc() for GFP_NOWAIT case. It solely relies on the background allocator to replenish the reserves. I would like percpu.c to call __vmalloc(GFP_NOWAIT) inline and see whether that succeeds. If that fails, it is fair to fail the call.
For this to work, __vmalloc() should be ready to serve a caller that is holding a
spinlock. The might_sleep() in alloc_vmap_area() basically prevents us calling vmalloc in this context.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Tahsin Erdogan
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
Yes, this prevents adding more pcpu chunks and so cause "atomic" allocations to fail more easily.
Then I fail to see what is the problem you are trying to fix.
To be more specific. Could you describe what more can we do in the vmalloc layer for
GFP_NOWAITallocations? They certainly cannot sleep and cannot perform the reclaim so you have to rely on the background work.
Quoted 6 lines
The main problem that I am trying to fix is in
percpu.ccode. It currently doesn't even attempt to callvmalloc()forGFP_NOWAITcase. It solely relies on the background allocator to replenish the reserves. I would likepercpu.cto call__vmalloc(GFP_NOWAIT)inline and see whether that succeeds. If that fails, it is fair to fail the call.
OK, that wasn't really clean from the patch to me. I guess it would be much more easier if a preparatory patch did the gfp mask propagation and then have patch that changes the pcpu allocator the way you need.
Quoted 3 lines
For this to work,
__vmalloc()should be ready to serve a caller that is holding a spinlock. Themight_sleep()inalloc_vmap_area()basically prevents us calling vmalloc in this context.
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Tejun Heo
maintainerHello,
Quoted 14 lines · Tahsin Erdogan
Quoted 7 lines · Michal Hocko
Quoted 4 lines · Tahsin Erdogan
When
pcpu_alloc()is called with gfp !=GFP_KERNEL, the likelihood of a failure is higher thanGFP_KERNELcase. This is mainly becausepcpu_alloc()relies on previously allocated reserves and does not make an effort to add memory to its pools for non-GFP_KERNEL case.Who is going to use a different mask?
blkg_create()makes a call with a non-GFP_KERNEL mask:new_blkg = blkg_alloc(blkcg, q, GFP_NOWAIT | __GFP_NOWARN);which turns into a call stack like below:
Quoted 12 lines
__vmalloc+0x45/0x50 pcpu_mem_zalloc+0x50/0x80 pcpu_populate_chunk+0x3b/0x380 pcpu_alloc+0x588/0x6e0 __alloc_percpu_gfp+0xd/0x10 __percpu_counter_init+0x55/0xc0 blkg_alloc+0x76/0x230 blkg_create+0x489/0x670 blkg_lookup_create+0x9a/0x230 generic_make_request_checks+0x7dd/0x890 generic_make_request+0x1f/0x180 submit_bio+0x61/0x120
As indicated by GFP_NOWAIT | __GFP_NOWARN, it's okay to fail there. It's not okay to fail consistently for a long time but it's not a big issue to fail occassionally even if somewhat bunched up. The only bad side effect of that is temporary misaccounting of some IOs, which shouldn't be noticeable outside of pathological cases. If you're actually seeing adverse effects of this, I'd love to learn about it.
Quoted 7 lines
Quoted 5 lines
We already have
__vmalloc_gfp, why this cannot be used? Also note that vmalloc dosn't really support arbitrary gfp flags. One have to be really careful because there are some internal allocations which are hardcodedGFP_KERNEL. Also this patch doesn't really add any new callers so it is hard to tell whether what you do actually makes sense and is correct.Did you mean to say
__vmalloc? If so, yes, I should use that.
So, the last time I looked at it the thorny ones in that path are the page table (pgd, pud...) allocation functions. There are several layers of indirection there but they end up in arch-specific implemntations which hard code GFP_KERNEL. Without fixing them up, we can't guarantee mapping the allocated pages making things kinda moot.
The only reason percpu allocator has the background allocator stuff is vmalloc path can't do non-blocking allocations. If we can properly fix that up, we can get rid of all those code from percpu allocator and simply path the gfp flag to vmap functions. Please take a look at __pcpu_map_pages() in mm/percpu-vm.c. map_kernel_range_noflush() is the function which has implicit GFP_KERNEL allocation in it and what's requiring the reserve.
If you can get rid of that, awesome, but given that your patch doesn't touch that at all, I can't see how it's supposed to work.
Thanks.
Signature · 7 lines
tejun -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hi Tejun,
Quoted 13 lines · Tejun Heo
Quoted 4 lines
__vmalloc+0x45/0x50 pcpu_mem_zalloc+0x50/0x80 pcpu_populate_chunk+0x3b/0x380 pcpu_alloc+0x588/0x6e0Quoted 8 lines
__alloc_percpu_gfp+0xd/0x10 __percpu_counter_init+0x55/0xc0 blkg_alloc+0x76/0x230 blkg_create+0x489/0x670 blkg_lookup_create+0x9a/0x230 generic_make_request_checks+0x7dd/0x890 generic_make_request+0x1f/0x180 submit_bio+0x61/0x120
Quoted 6 lines
As indicated by
GFP_NOWAIT|__GFP_NOWARN, it's okay to fail there. It's not okay to fail consistently for a long time but it's not a big issue to fail occassionally even if somewhat bunched up. The only bad side effect of that is temporary misaccounting of some IOs, which shouldn't be noticeable outside of pathological cases. If you're actually seeing adverse effects of this, I'd love to learn about it.
A better example is the call path below:
pcpu_alloc+0x68f/0x710 __alloc_percpu_gfp+0xd/0x10 __percpu_counter_init+0x55/0xc0 cfq_pd_alloc+0x3b2/0x4e0 blkg_alloc+0x187/0x230 blkg_create+0x489/0x670 blkg_lookup_create+0x9a/0x230 blkg_conf_prep+0x1fb/0x240 __cfqg_set_weight_device.isra.105+0x5c/0x180 cfq_set_weight_on_dfl+0x69/0xc0 cgroup_file_write+0x39/0x1c0 kernfs_fop_write+0x13f/0x1d0 __vfs_write+0x23/0x120 vfs_write+0xc2/0x1f0 SyS_write+0x44/0xb0 entry_SYSCALL_64_fastpath+0x18/0xad
A failure in this call path gives grief to tools which are trying to configure io
weights. We see occasional failures happen here shortly after reboots even when system is not under any memory pressure. Machines with a lot of cpus are obviously more vulnerable.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Tejun Heo
maintainerHello,
Quoted 19 lines · Tahsin Erdogan
A better example is the call path below:
pcpu_alloc+0x68f/0x710 __alloc_percpu_gfp+0xd/0x10 __percpu_counter_init+0x55/0xc0 cfq_pd_alloc+0x3b2/0x4e0 blkg_alloc+0x187/0x230 blkg_create+0x489/0x670 blkg_lookup_create+0x9a/0x230 blkg_conf_prep+0x1fb/0x240 __cfqg_set_weight_device.isra.105+0x5c/0x180 cfq_set_weight_on_dfl+0x69/0xc0 cgroup_file_write+0x39/0x1c0 kernfs_fop_write+0x13f/0x1d0 __vfs_write+0x23/0x120 vfs_write+0xc2/0x1f0 SyS_write+0x44/0xb0 entry_SYSCALL_64_fastpath+0x18/0xad
Quoted 5 lines
A failure in this call path gives grief to tools which are trying to configure io
weights. We see occasional failures happen here shortly after reboots even when system is not under any memory pressure. Machines with a lot of cpus are obviously more vulnerable.
Ah, absolutely, that's a stupid failure but we should be able to fix that by making the blkg functions take gfp mask and allocate accordingly, right? It'll probably take preallocation tricks because of locking but should be doable.
Thanks.
Signature · 7 lines
tejun -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 28 lines · Tejun Heo
Hello,
Quoted 19 lines · Tahsin Erdogan
A better example is the call path below:
pcpu_alloc+0x68f/0x710 __alloc_percpu_gfp+0xd/0x10 __percpu_counter_init+0x55/0xc0 cfq_pd_alloc+0x3b2/0x4e0 blkg_alloc+0x187/0x230 blkg_create+0x489/0x670 blkg_lookup_create+0x9a/0x230 blkg_conf_prep+0x1fb/0x240 __cfqg_set_weight_device.isra.105+0x5c/0x180 cfq_set_weight_on_dfl+0x69/0xc0 cgroup_file_write+0x39/0x1c0 kernfs_fop_write+0x13f/0x1d0 __vfs_write+0x23/0x120 vfs_write+0xc2/0x1f0 SyS_write+0x44/0xb0 entry_SYSCALL_64_fastpath+0x18/0xadQuoted 5 lines
A failure in this call path gives grief to tools which are trying to configure io
weights. We see occasional failures happen here shortly after reboots even when system is not under any memory pressure. Machines with a lot of cpus are obviously more vulnerable.
Quoted 4 lines
Ah, absolutely, that's a stupid failure but we should be able to fix that by making the blkg functions take gfp mask and allocate accordingly, right? It'll probably take preallocation tricks because of locking but should be doable.
My initial goal was to allow calls to vmalloc(), but I now see the challenges in that
approach.
Doing preallocations would probably work but not sure if that can be done without
complicating code too much. Could you describe what you have in mind?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Tejun Heo
maintainerHello, Tahsin.
Quoted 8 lines · Tahsin Erdogan
Quoted 4 lines
Ah, absolutely, that's a stupid failure but we should be able to fix that by making the blkg functions take gfp mask and allocate accordingly, right? It'll probably take preallocation tricks because of locking but should be doable.
My initial goal was to allow calls to
vmalloc(), but I now see the challenges in that
approach.
I'd love to see that working too but this is a different issue. Even GFP_ATOMIC can fail under pressure and it's kinda wrong to depend on that for userspace interactions.
Quoted 3 lines
Doing preallocations would probably work but not sure if that can be done without
complicating code too much. Could you describe what you have in mind?
So, blkg_create() already takes @new_blkg argument which is the preallocated blkg and used during q init. Wouldn't it work to make blkg_lookup_create() take @new_blkg too and pass it down to blkg_create() (and also free it if it doesn't get used)? Then, blkg_conf_prep() can always (or after a failure with -ENOMEM) allocate a new blkg before calling into blkg_lookup_create(). I don't think it'll complicate the code path that much.
Thanks.
Signature · 7 lines
tejun -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines
Quoted 3 lines
Doing preallocations would probably work but not sure if that can be done without
complicating code too much. Could you describe what you have in mind?So,
blkg_create()already takes @new_blkg argument which is the preallocated blkg and used during q init. Wouldn't it work to makeblkg_lookup_create()take @new_blkg too and pass it down toblkg_create()(and also free it if it doesn't get used)? Then,blkg_conf_prep()can always (or after a failure with-ENOMEM) allocate a new blkg before calling intoblkg_lookup_create(). I don't think it'll complicate the code path that much.
That makes sense. I will work a patch that does that (unless you are interested in implementing it yourself).
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Tejun Heo
maintainerQuoted 2 lines · Tahsin Erdogan
That makes sense. I will work a patch that does that (unless you are interested in implementing it yourself).
I'd really appreciate if you can work on it. Thanks a lot!
Signature · 7 lines
tejun -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>