raid5: use memalloc_noio_save()/restore in resize_chunks()
PATCH 1/1 raid5: use memalloc_noio_save()/restore in resize_chunks()
Commit b330e6a49dc3 ("md: convert to kvmalloc") uses kvmalloc_array() to allocate memory with GFP_NOIO flag in resize_chunks() via function scribble_alloc(),
2269 err = scribble_alloc(percpu, new_disks,
2270 new_sectors / STRIPE_SECTORS,
2271 GFP_NOIO);
The purpose of GFP_NOIO flag to kvmalloc_array() is to allocate non-physically continuous pages and avoid extra I/Os of page reclaim which triggered by memory allocation. When system memory is under heavy pressure, non-physically continuous pages allocation is more probably to success than allocating physically continuous pages.
But as a non GFP_KERNEL compatible flag, GFP_NOIO is not acceptible by kvmalloc_node() and the memory allocation indeed is handled with kmalloc_node() to allocate physically continuous pages. This is not the expected behavior of the original purpose when mistakenly using GFP_NOIO flag.
In this patch, the memalloc scope APIs memalloc_noio_save() and memalloc_noio_restore() are used when calling scribble_alloc(). Then when calling kvmalloc_array() with GFP_KERNEL mask, the scope APIs may indicatet the allocating context to avoid memory reclaim related I/Os, to avoid recursive I/O deadlock on the md raid array itself which is calling scribble_alloc() to allocate non-physically continuous pages.
This patch also removes gfp_t flags from scribble_alloc() parameters list, because the invalid GFP_NOIO is replaced by memalloc scope APIs.
--- drivers/md/raid5.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-)
drivers/md/raid5.c +16 -6
@@ -2228,14 +2228,15 @@ static int grow_stripes(struct r5conf *conf, int num)* of the P and Q blocks. */ static int scribble_alloc (struct raid5_percpu *percpu, -int num,int cnt,gfp_t flags) +int num,int cnt) {size_t obj_size =sizeof (struct page *) * (num+2 ) +sizeof (addr_conv_t) * (num+2 );void *scribble; +unsigned int noio_flag; - scribble =kvmalloc_array (cnt, obj_size, flags); + scribble =kvmalloc_array (cnt, obj_size,GFP_KERNEL );if (!scribble)return -ENOMEM ;
@@ -2250,6 +2251,7 @@ static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors) {unsigned long cpu;int err =0 ; +unsigned int noio_flag;/* * Never shrink. And mddev_suspend() could deadlock if this is called
@@ -2262,16 +2264,25 @@ static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors)mddev_suspend (conf->mddev);get_online_cpus (); +/* +* scribble_alloc() allocates memory by kvmalloc_array(), if +* the memory allocation triggers memory reclaim I/Os onto +* this raid array, there might be potential deadlock if this +* raid array happens to be suspended during memory allocation. +* Here the scope APIs are used to disable such recursive memory +* reclaim I/Os. +*/ + noio_flag =memalloc_noio_save ();for_each_present_cpu (cpu) {struct raid5_percpu *percpu; percpu =per_cpu_ptr (conf->percpu, cpu); err =scribble_alloc (percpu, new_disks, - new_sectors /STRIPE_SECTORS , -GFP_NOIO ); + new_sectors /STRIPE_SECTORS );if (err)break ; } +memalloc_noio_restore (noio_flag);put_online_cpus ();mddev_resume (conf->mddev);
@@ -6759,8 +6770,7 @@ static int alloc_scratch_buffer(struct r5conf *conf, struct raid5_percpu *percpu conf->previous_raid_disks), max(conf->chunk_sectors, conf->prev_chunk_sectors) - / STRIPE_SECTORS, - GFP_KERNEL)) { + / STRIPE_SECTORS)) {free_scratch_buffer (conf, percpu);return -ENOMEM ; }
Signature · 1 line
2.25.0
kbuild test robot
build botI love your patch! Yet something to improve:
Hi Coly,
I love your patch! Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v5.6 next-20200403]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Coly-Li/raid5-use-memalloc_noio_save-restore-in-resize_chunks/20200402-215014 base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 919dce24701f7b34681a6a1d3ef95c9f6c4fb1cc config: x86_64-rhel (attached as .config) compiler: gcc-7 (Ubuntu 7.5.0-6ubuntu2) 7.5.0 reproduce: # save the attached .config to linux build tree make ARCH=x86_64
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
drivers/md/raid5.c: In function 'scribble_alloc':
drivers/md/raid5.c:2237:15: warning: unused variable 'noio_flag' [-Wunused-variable]
unsigned int noio_flag;
^~~~~~~~~
drivers/md/raid5.c: In function 'resize_chunks':
>> drivers/md/raid5.c:2275:14: error: implicit declaration of function 'memalloc_noio_save'; did you mean 'vmalloc_to_page'? [-Werror=implicit-function-declaration]
noio_flag = memalloc_noio_save();
^~~~~~~~~~~~~~~~~~
vmalloc_to_page
>> drivers/md/raid5.c:2285:2: error: implicit declaration of function 'memalloc_noio_restore'; did you mean '__vmalloc_node_range'? [-Werror=implicit-function-declaration]
memalloc_noio_restore(noio_flag);
^~~~~~~~~~~~~~~~~~~~~
__vmalloc_node_rangecc1: some warnings being treated as errors
vim +2275 drivers/md/raid5.c
2249
2250 static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors)
2251 {
2252 unsigned long cpu;
2253 int err = 0;
2254 unsigned int noio_flag;
2255
2256 /*
2257 * Never shrink. And mddev_suspend() could deadlock if this is called
2258 * from raid5d. In that case, scribble_disks and scribble_sectors
2259 * should equal to new_disks and new_sectors
2260 */
2261 if (conf->scribble_disks >= new_disks &&
2262 conf->scribble_sectors >= new_sectors)
2263 return 0;
2264 mddev_suspend(conf->mddev);
2265 get_online_cpus();
2266
2267 /*
2268 * scribble_alloc() allocates memory by kvmalloc_array(), if
2269 * the memory allocation triggers memory reclaim I/Os onto
2270 * this raid array, there might be potential deadlock if this
2271 * raid array happens to be suspended during memory allocation.
2272 * Here the scope APIs are used to disable such recursive memory
2273 * reclaim I/Os.
2274 */
> 2275 noio_flag = memalloc_noio_save();
2276 for_each_present_cpu(cpu) {
2277 struct raid5_percpu *percpu;
2278
2279 percpu = per_cpu_ptr(conf->percpu, cpu);
2280 err = scribble_alloc(percpu, new_disks,
2281 new_sectors / STRIPE_SECTORS);
2282 if (err)
2283 break;
2284 }
> 2285 memalloc_noio_restore(noio_flag); 2286
2287 put_online_cpus();
2288 mddev_resume(conf->mddev);
2289 if (!err) {
2290 conf->scribble_disks = new_disks;
2291 conf->scribble_sectors = new_sectors;
2292 }
2293 return err;
2294 }
2295
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/[email protected]
drivers/md/raid5.c static int grow_stripes(struct r5conf *conf, int num) · line 2239
Quoted 2 lines · Coly Li
drivers/md/raid5.c +1
-1- scribble =kvmalloc_array (cnt, obj_size, flags); + scribble =kvmalloc_array (cnt, obj_size,GFP_KERNEL );
Maybe it is simpler to call kvmalloc_array between memalloc_noio_save and memalloc_noio_restore. And seems sched/mm.h need to be included per the report from LKP.
Thanks,
Guoqing
Quoted 7 lines · Guoqing Jiang
Quoted 2 lines · Coly Li
- scribble =
kvmalloc_array(cnt, obj_size, flags); + scribble =kvmalloc_array(cnt, obj_size, GFP_KERNEL);Maybe it is simpler to call
kvmalloc_arraybetweenmemalloc_noio_saveandmemalloc_noio_restore.
And seemssched/mm.hneed to be included per the report from LKP.
The falgs can be,
- GFP_KERNEL: when called from alloc_scratch_buffer()
- GFP_NOIO: when called from resize_chunks().
If the scope APIs are used inside scribble_alloc(), the first call path is restricted as noio, which is not expected. So I only use the scope APIs around the location where GFP_NOIO is used.
Anyway, Michal Hocko suggests to add the scope APIs in mddev_suspend()/mddev_resume(). Then in the whole code path where md raid array is suspend, we don't need to worry the recursive memory reclaim I/Os onto the array. After checking the complicated raid5 code, I come to realize this suggestion makes sense.
I will try to compose a v2 patch following Michal's suggestion.
Signature · 1 line
Coly Li
Quoted 7 lines · Coly Li
Quoted 6 lines · Guoqing Jiang
Quoted 2 lines · Coly Li
- scribble =
kvmalloc_array(cnt, obj_size, flags); + scribble =kvmalloc_array(cnt, obj_size, GFP_KERNEL);Maybe it is simpler to call
kvmalloc_arraybetweenmemalloc_noio_saveandmemalloc_noio_restore.
And seemssched/mm.hneed to be included per the report from LKP.
Quoted 7 lines
The falgs can be,
-GFP_KERNEL: when called fromalloc_scratch_buffer()
-GFP_NOIO: when called fromresize_chunks().If the scope APIs are used inside
scribble_alloc(), the first call path is restricted as noio, which is not expected. So I only use the scope APIs around the location whereGFP_NOIOis used.
Thanks for the explanation. I assume it can be distinguished by check the flag, eg, FYI.
if (flag == GFP_NOIO)
memalloc_noio_save();kvmalloc_array();
if (flag == GFP_NOIO)
memalloc_noio_restore();
Quoted 5 lines
Anyway, Michal Hocko suggests to add the scope APIs in mddev_suspend()/mddev_resume(). Then in the whole code path where md raid array is suspend, we don't need to worry the recursive memory reclaim I/Os onto the array. After checking the complicated raid5 code, I come to realize this suggestion makes sense.
Hmm, mddev_suspend/resume are called at lots of places, then it's better to check if all the places don't allocate memory with GFP_KERNEL.
And seems In level_store(), sysfs_create_group could be called between suspend and resume, then the two functions (kstrdup_const(name, GFP_KERNEL) and kmem_cache_zalloc(kernfs_node_cache, GFP_KERNEL)) could be triggered by the path:
sysfs_create_group ->internal_create_group -> kernfs_create_dir_ns -> kernfs_new_node -> __kernfs_new_node
Not know memalloc_noio_{save,restore} well, but I guess it is better to use them to mark a small scope, just my two cents.
Thanks,
Guoqing
Quoted 20 lines · Guoqing Jiang
Quoted 7 lines · Coly Li
Quoted 6 lines · Guoqing Jiang
Quoted 2 lines · Coly Li
- scribble =
kvmalloc_array(cnt, obj_size, flags); + scribble =kvmalloc_array(cnt, obj_size, GFP_KERNEL);Maybe it is simpler to call
kvmalloc_arraybetweenmemalloc_noio_saveandmemalloc_noio_restore.
And seemssched/mm.hneed to be included per the report from LKP.Quoted 7 lines
The falgs can be,
-GFP_KERNEL: when called fromalloc_scratch_buffer()
-GFP_NOIO: when called fromresize_chunks().If the scope APIs are used inside
scribble_alloc(), the first call path is restricted as noio, which is not expected. So I only use the scope APIs around the location whereGFP_NOIOis used.Thanks for the explanation. I assume it can be distinguished by check the flag,
eg, FYI.
Quoted 5 lines
if (flag ==
GFP_NOIO)
memalloc_noio_save();kvmalloc_array();
if (flag ==GFP_NOIO)
memalloc_noio_restore();
This was similar to my original idea, but finally I decide to accept Michal's suggestion to add them into mddev_suspend()/mddev_resume. Let me explain.
Quoted 9 lines
Quoted 5 lines
Anyway, Michal Hocko suggests to add the scope APIs in mddev_suspend()/mddev_resume(). Then in the whole code path where md raid array is suspend, we don't need to worry the recursive memory reclaim I/Os onto the array. After checking the complicated raid5 code, I come to realize this suggestion makes sense.
Hmm, mddev_suspend/resume are called at lots of places, then it's better to check if all the places don't allocate memory with
GFP_KERNEL.
When mddev_suspend() is called, then all I/Os on the suspending raid array have to wait until mddev_resume() is called. Inside the suspending region, all memory allocation should be careful to avoid memory reclaim I/Os going back to the suspending raid array. If such recursive I/Os happen, we will experience deadlock.
Therefore we should be very careful to use GFP_KERNEL to allocate memory during the period when the raid array is suspending. No matter physically continuous or virtually continuous pages.
Quoted 9 lines
And seems In
level_store(),sysfs_create_groupcould be called between suspend
and resume, then the two functions (kstrdup_const(name, GFP_KERNEL)andkmem_cache_zalloc(kernfs_node_cache, GFP_KERNEL)) could be triggered by the path:
sysfs_create_group->internal_create_group ->kernfs_create_dir_ns->kernfs_new_node->__kernfs_new_node
From your information, the above code path is suspicious. Although the deadlock might not happen in practice, unless the raid array is used as the only fs volume, and all memory writeback or swap I/Os will only go into this array.
Checking all the location of memory allocation is trivial but possible, how to prevent new code with memory allocation to avoid the potential deadlock risk during the suspending period is still a problem.
Quoted 3 lines
Not know memalloc_noio_{save,restore} well, but I guess it is better to use them
to mark a small scope, just my two cents.
Since mddev_suspend() is the entry point to prevent I/Os on the array, it is the good location to restrict memory reclaim I/Os of memory allocation. Finally I realize this is a brilliant idea after Michal Hocko suggests me for a while.
Thanks.
Signature · 1 line
Coly Li
On Thu 09-04-20 23:38:13, Guoqing Jiang wrote:
[...]
Quoted 2 lines
Not know memalloc_noio_{save,restore} well, but I guess it is better to use them to mark a small scope, just my two cents.
This would go against the intentio of the api. It is really meant to define reclaim recursion problematic scope. If there is a clear entry point where any further allocation recursing to FS/IO could deadlock then it should be used at that level. This might be a lock which is taken from the reclaim or like this case a device is suspended and no IO is processed so anything that would wait for an IO or rely on IO making progress in the reclaim path would deadlock.
Please have a look at Documentation/core-api/gfp_mask-from-fs-io.rst and let me know is something could be made more clear or explicit. I am more than happy to improve the documentation.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 6 lines · Michal Hocko
On Thu 09-04-20 23:38:13, Guoqing Jiang wrote:
[...]Quoted 2 lines
Not know memalloc_noio_{save,restore} well, but I guess it is better to use them to mark a small scope, just my two cents.
This would go against the intentio of the api. It is really meant to define reclaim recursion problematic scope.
Well, in current proposal, the scope is just when scribble_allo/kvmalloc_array is called.
memalloc_noio_save
scribble_allo/kvmalloc_arraymemalloc_noio_restore
With the new proposal, the marked scope would be bigger than current one since there are lots of places call mddev_suspend/resume.
mddev_suspendmemalloc_noio_save
...memalloc_noio_restoremddev_resume
IMHO, if the current proposal works then what is the advantage to increase the scope. If all the callers of mddev_suspend/resume could suffer from the deadlock issue due to recursing fs io, then it is definitely need to use the new proposal.
Quoted 2 lines
If there is a clear entry point where any further allocation recursing to FS/IO could deadlock then it should be used at that level.
Agree.
At the end of mddev_suspend, I guess there is no FS/IO could happen to the array, because mddev->suspended++ and quiesce(mddev, 1) are called previously in mddev_suspend. And it makes me curious to go through to the call chain:
layout_store / chunk_size_store / update_raid_disks / update_array_info / md_check_recovery => pers->check_reshape => raid{5,6}_check_reshape => check_reshape => resize_chunks => scribble_alloc(..., GFP_NOIO)
Perhaps I missed something, but seems all the 5 original callers are not called between mddev_suspend and mddev_resume, if so, then with the new proposal, scribble_alloc() could not be protected by the memalloc_noio_{save,restore}.
Quoted 4 lines
This might be a lock which is
taken from the reclaim or like this case a device is suspended and no IO is processed so anything that would wait for an IO or rely on IO making progress in the reclaim path would deadlock.
Thanks for teaching.
Quoted 3 lines
Please have a look at
Documentation/core-api/gfp_mask-from-fs-io.rstand let me know is something could be made more clear or explicit. I am more than happy to improve the documentation.
Thanks again for your lighting, will read it.
Regards,
Guoqing
Quoted 19 lines · Guoqing Jiang
Quoted 6 lines · Michal Hocko
On Thu 09-04-20 23:38:13, Guoqing Jiang wrote:
[...]Quoted 2 lines
Not know memalloc_noio_{save,restore} well, but I guess it is better to use them to mark a small scope, just my two cents.
This would go against the intentio of the api. It is really meant to define reclaim recursion problematic scope.
Well, in current proposal, the scope is just when
scribble_allo/kvmalloc_array is called.
memalloc_noio_save
scribble_allo/kvmalloc_arraymemalloc_noio_restoreWith the new proposal, the marked scope would be bigger than current one since there
are lots of places call mddev_suspend/resume.
Quoted 8 lines
mddev_suspendmemalloc_noio_save
...memalloc_noio_restoremddev_resumeIMHO, if the current proposal works then what is the advantage to increase the scope.
The advantage is twofold. It serves the documentation purpose because it is clear _what_ and _why_ is the actual allocation restricted context. In this case mddev_{suspend,resume} because XYZ and you do not have to care about __GFP_IO for _any_ allocation inside the scope.
Quoted 3 lines
If all the callers of mddev_suspend/resume could suffer from the deadlock issue due to recursing fs io, then it is definitely need to use the new proposal.
Why some of them wouldn't? Isn't the mddev_suspend going to block any IO on the device? The thing is that you cannot tell which devices the allocator can issue IO for therefore GFP_NOIO is a global flag. Please also note that the scope API is bound to a process context so it only affects the current execution.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 25 lines · Michal Hocko
Quoted 24 lines · Guoqing Jiang
Quoted 6 lines · Michal Hocko
On Thu 09-04-20 23:38:13, Guoqing Jiang wrote:
[...]Quoted 2 lines
Not know memalloc_noio_{save,restore} well, but I guess it is better to use them to mark a small scope, just my two cents.
This would go against the intentio of the api. It is really meant to define reclaim recursion problematic scope.
Well, in current proposal, the scope is just when
scribble_allo/kvmalloc_array is called.
memalloc_noio_save
scribble_allo/kvmalloc_arraymemalloc_noio_restoreWith the new proposal, the marked scope would be bigger than current one since there
are lots of places call mddev_suspend/resume.
mddev_suspendmemalloc_noio_save
...memalloc_noio_restoremddev_resume
Quoted 6 lines
Quoted 2 lines
IMHO, if the current proposal works then what is the advantage to increase the scope.
The advantage is twofold. It serves the documentation purpose because it is clear _what_ and _why_ is the actual allocation restricted context. In this case mddev_{suspend,resume} because XYZ and you do not have to care about
__GFP_IOfor _any_ allocation inside the scope.
Personally, I'd prefer fine grained protection scope, anyway just my own flavor. And I think Song has his opinion about the proposal, I will respect his decision.
Thanks,
Guoqing
Quoted 32 lines · Guoqing Jiang
Quoted 27 lines · Michal Hocko
Quoted 26 lines · Guoqing Jiang
Quoted 6 lines · Michal Hocko
On Thu 09-04-20 23:38:13, Guoqing Jiang wrote:
[...]Quoted 2 lines
Not know memalloc_noio_{save,restore} well, but I guess it is better to use them to mark a small scope, just my two cents.
This would go against the intentio of the api. It is really meant to define reclaim recursion problematic scope.
Well, in current proposal, the scope is just when
scribble_allo/kvmalloc_array is called.
memalloc_noio_save
scribble_allo/kvmalloc_arraymemalloc_noio_restoreWith the new proposal, the marked scope would be bigger than current one since there
are lots of places call mddev_suspend/resume.
mddev_suspendmemalloc_noio_save
...memalloc_noio_restoremddev_resumeIMHO, if the current proposal works then what is the advantage to increase the scope.
Quoted 8 lines
Quoted 4 lines
The advantage is twofold. It serves the documentation purpose because it is clear _what_ and _why_ is the actual allocation restricted context. In this case mddev_{suspend,resume} because XYZ and you do not have to care about
__GFP_IOfor _any_ allocation inside the scope.Personally, I'd prefer fine grained protection scope, anyway just my own flavor. And I think Song has his opinion about the proposal, I will respect his decision.
Thanks Coly, Guoqing, and Michal for these great inputs.
Coly's v3 set looks good to me. I will go ahead apply that version to md-next branch.
Thanks again,
Song
Hi Coly,
Thanks for the patch!
Quoted 45 lines · Coly Li
Commit
b330e6a49dc3("md: convert to kvmalloc") useskvmalloc_array()to allocate memory withGFP_NOIOflag inresize_chunks()via functionscribble_alloc(),
2269 err = scribble_alloc(percpu,new_disks,
2270new_sectors/STRIPE_SECTORS,
2271GFP_NOIO);The purpose of
GFP_NOIOflag tokvmalloc_array()is to allocate non-physically continuous pages and avoid extra I/Os of page reclaim which triggered by memory allocation. When system memory is under heavy pressure, non-physically continuous pages allocation is more probably to success than allocating physically continuous pages.But as a non
GFP_KERNELcompatible flag,GFP_NOIOis not acceptible bykvmalloc_node()and the memory allocation indeed is handled withkmalloc_node()to allocate physically continuous pages. This is not the expected behavior of the original purpose when mistakenly usingGFP_NOIOflag.In this patch, the memalloc scope APIs
memalloc_noio_save()andmemalloc_noio_restore()are used when callingscribble_alloc(). Then when callingkvmalloc_array()withGFP_KERNELmask, the scope APIs may indicatet the allocating context to avoid memory reclaim related I/Os, to avoid recursive I/O deadlock on the md raid array itself which is callingscribble_alloc()to allocate non-physically continuous pages.This patch also removes
gfp_tflags fromscribble_alloc()parameters list, because the invalidGFP_NOIOis replaced by memalloc scope APIs.Fixes b330e6a49dc3 ("md: convert to kvmalloc")Signed-off-by Coly Li <[email protected]>Cc Kent Overstreet <[email protected]>Cc Michal Hocko <[email protected]>---
drivers/md/raid5.c| 22 ++++++++++++++++------1 file changed, 16 insertions(+), 6 deletions(-)drivers/md/raid5.c +0
-0
drivers/md/raid5.c static int grow_stripes(struct r5conf *conf, int num) · line 2229
Quoted 2 lines
drivers/md/raid5.c +0
-0@@ -2228,14 +2228,15 @@ static int grow_stripes(struct r5conf *conf, int num)* of the P and Q blocks. */
I noticed the comment before scribble_alloc() is outdated. Maybe fix also that as we are on it?
drivers/md/raid5.c static int grow_stripes(struct r5conf *conf, int num) · line 2237
Quoted 9 lines
drivers/md/raid5.c +2
-1static int scribble_alloc (struct raid5_percpu *percpu, -int num,int cnt,gfp_t flags) +int num,int cnt) {size_t obj_size =sizeof (struct page *) * (num+2 ) +sizeof (addr_conv_t) * (num+2 );void *scribble; +unsigned int noio_flag;
I think we don't use noio_flag in scribble_alloc()?
Thanks,
Song
Quoted 44 lines · Song Liu
Hi Coly,
Thanks for the patch!
Quoted 40 lines · Coly Li
Commit
b330e6a49dc3("md: convert to kvmalloc") useskvmalloc_array()to allocate memory withGFP_NOIOflag inresize_chunks()via functionscribble_alloc(),
2269 err = scribble_alloc(percpu,new_disks,
2270new_sectors/STRIPE_SECTORS,
2271GFP_NOIO);The purpose of
GFP_NOIOflag tokvmalloc_array()is to allocate non-physically continuous pages and avoid extra I/Os of page reclaim which triggered by memory allocation. When system memory is under heavy pressure, non-physically continuous pages allocation is more probably to success than allocating physically continuous pages.But as a non
GFP_KERNELcompatible flag,GFP_NOIOis not acceptible bykvmalloc_node()and the memory allocation indeed is handled withkmalloc_node()to allocate physically continuous pages. This is not the expected behavior of the original purpose when mistakenly usingGFP_NOIOflag.In this patch, the memalloc scope APIs
memalloc_noio_save()andmemalloc_noio_restore()are used when callingscribble_alloc(). Then when callingkvmalloc_array()withGFP_KERNELmask, the scope APIs may indicatet the allocating context to avoid memory reclaim related I/Os, to avoid recursive I/O deadlock on the md raid array itself which is callingscribble_alloc()to allocate non-physically continuous pages.This patch also removes
gfp_tflags fromscribble_alloc()parameters list, because the invalidGFP_NOIOis replaced by memalloc scope APIs.Fixes b330e6a49dc3 ("md: convert to kvmalloc")Signed-off-by Coly Li <[email protected]>Cc Kent Overstreet <[email protected]>Cc Michal Hocko <[email protected]>---
drivers/md/raid5.c| 22 ++++++++++++++++------1 file changed, 16 insertions(+), 6 deletions(-)
Quoted 11 lines
drivers/md/raid5.c static int grow_stripes(struct r5conf *conf, int num) · line 2229
Quoted 7 lines
drivers/md/raid5.c +0
-0@@ -2228,14 +2228,15 @@ static int grow_stripes(struct r5conf *conf, int num)* of the P and Q blocks. */ I noticed the comment before
scribble_alloc()is outdated. Maybe fix also that as we are on it?
OK, I will do that.
Quoted 10 lines
drivers/md/raid5.c static int grow_stripes(struct r5conf *conf, int num) · line 2237
Quoted 9 lines
drivers/md/raid5.c +2
-1static int scribble_alloc (struct raid5_percpu *percpu, -int num,int cnt,gfp_t flags) +int num,int cnt) {size_t obj_size =sizeof (struct page *) * (num+2 ) +sizeof (addr_conv_t) * (num+2 );void *scribble; +unsigned int noio_flag;I think we don't use
noio_flaginscribble_alloc()?
You are right of cause. It should not be here :-)
Thanks.
Signature · 1 line
Coly Li