Skip to contentSkip to conversation

raid5: use memalloc_noio_save()/restore in resize_chunks()

PATCH In review Coly Li 5 participants 13 messages Apr 2, 2020 – Apr 30, 2020 MD RAID

lore
Share & tools
lore

Coly Li

author
#0 lore raw

PATCH 1/1 raid5: use memalloc_noio_save()/restore in resize_chunks()

Commit b330e6a49dc3 ("md: convert to kvmalloc") uses kvmalloc_array() to allocate memory with GFP_NOIO flag in resize_chunks() via function scribble_alloc(),
2269 err = scribble_alloc(percpu, new_disks,
2270 new_sectors / STRIPE_SECTORS,
2271 GFP_NOIO);

The purpose of GFP_NOIO flag to kvmalloc_array() is to allocate non-physically continuous pages and avoid extra I/Os of page reclaim which triggered by memory allocation. When system memory is under heavy pressure, non-physically continuous pages allocation is more probably to success than allocating physically continuous pages.

But as a non GFP_KERNEL compatible flag, GFP_NOIO is not acceptible by kvmalloc_node() and the memory allocation indeed is handled with kmalloc_node() to allocate physically continuous pages. This is not the expected behavior of the original purpose when mistakenly using GFP_NOIO flag.

In this patch, the memalloc scope APIs memalloc_noio_save() and memalloc_noio_restore() are used when calling scribble_alloc(). Then when calling kvmalloc_array() with GFP_KERNEL mask, the scope APIs may indicatet the allocating context to avoid memory reclaim related I/Os, to avoid recursive I/O deadlock on the md raid array itself which is calling scribble_alloc() to allocate non-physically continuous pages.

This patch also removes gfp_t flags from scribble_alloc() parameters list, because the invalid GFP_NOIO is replaced by memalloc scope APIs.

Fixes b330e6a49dc3 ("md: convert to kvmalloc")
Cc Kent Overstreet <[email protected]>
Cc Michal Hocko <[email protected]>
---
 drivers/md/raid5.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)
drivers/md/raid5.c +16 -6
index ba00e9877f02..6b23f8aba169 100644
@@ -2228,14 +2228,15 @@ static int grow_stripes(struct r5conf *conf, int num)
  * of the P and Q blocks.
  */
 static int scribble_alloc(struct raid5_percpu *percpu,
-			  int num, int cnt, gfp_t flags)
+			  int num, int cnt)
 {
 	size_t obj_size =
 		sizeof(struct page *) * (num+2) +
 		sizeof(addr_conv_t) * (num+2);
 	void *scribble;
+	unsigned int noio_flag;
 
-	scribble = kvmalloc_array(cnt, obj_size, flags);
+	scribble = kvmalloc_array(cnt, obj_size, GFP_KERNEL);
 	if (!scribble)
 		return -ENOMEM;
 
@@ -2250,6 +2251,7 @@ static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors)
 {
 	unsigned long cpu;
 	int err = 0;
+	unsigned int noio_flag;
 
 	/*
 	 * Never shrink. And mddev_suspend() could deadlock if this is called
@@ -2262,16 +2264,25 @@ static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors)
 	mddev_suspend(conf->mddev);
 	get_online_cpus();
 
+	/*
+	 * scribble_alloc() allocates memory by kvmalloc_array(), if
+	 * the memory allocation triggers memory reclaim I/Os onto
+	 * this raid array, there might be potential deadlock if this
+	 * raid array happens to be suspended during memory allocation.
+	 * Here the scope APIs are used to disable such recursive memory
+	 * reclaim I/Os.
+	 */
+	noio_flag = memalloc_noio_save();
 	for_each_present_cpu(cpu) {
 		struct raid5_percpu *percpu;
 
 		percpu = per_cpu_ptr(conf->percpu, cpu);
 		err = scribble_alloc(percpu, new_disks,
-				     new_sectors / STRIPE_SECTORS,
-				     GFP_NOIO);
+				     new_sectors / STRIPE_SECTORS);
 		if (err)
 			break;
 	}
+	memalloc_noio_restore(noio_flag);
 
 	put_online_cpus();
 	mddev_resume(conf->mddev);
@@ -6759,8 +6770,7 @@ static int alloc_scratch_buffer(struct r5conf *conf, struct raid5_percpu *percpu
 			       conf->previous_raid_disks),
 			   max(conf->chunk_sectors,
 			       conf->prev_chunk_sectors)
-			   / STRIPE_SECTORS,
-			   GFP_KERNEL)) {
+			   / STRIPE_SECTORS)) {
 		free_scratch_buffer(conf, percpu);
 		return -ENOMEM;
 	}
Signature · 1 line
2.25.0
#1 lore raw
I love your patch! Yet something to improve:

Hi Coly,

I love your patch! Yet something to improve:

[auto build test ERROR on linus/master]
[also build test ERROR on v5.6 next-20200403]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Coly-Li/raid5-use-memalloc_noio_save-restore-in-resize_chunks/20200402-215014
base:   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 919dce24701f7b34681a6a1d3ef95c9f6c4fb1cc
config: x86_64-rhel (attached as .config)
compiler: gcc-7 (Ubuntu 7.5.0-6ubuntu2) 7.5.0
reproduce:
        # save the attached .config to linux build tree
        make ARCH=x86_64 

If you fix the issue, kindly add following tag

All errors (new ones prefixed by >>):

drivers/md/raid5.c: In function 'scribble_alloc':
drivers/md/raid5.c:2237:15: warning: unused variable 'noio_flag' [-Wunused-variable]

     unsigned int noio_flag;
                  ^~~~~~~~~
   drivers/md/raid5.c: In function 'resize_chunks':
>> drivers/md/raid5.c:2275:14: error: implicit declaration of function 'memalloc_noio_save'; did you mean 'vmalloc_to_page'? [-Werror=implicit-function-declaration]
     noio_flag = memalloc_noio_save();
                 ^~~~~~~~~~~~~~~~~~
                 vmalloc_to_page
>> drivers/md/raid5.c:2285:2: error: implicit declaration of function 'memalloc_noio_restore'; did you mean '__vmalloc_node_range'? [-Werror=implicit-function-declaration]
     memalloc_noio_restore(noio_flag);
     ^~~~~~~~~~~~~~~~~~~~~
     __vmalloc_node_range

cc1: some warnings being treated as errors

vim +2275 drivers/md/raid5.c

2249
2250 static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors)
2251 {
2252 unsigned long cpu;
2253 int err = 0;
2254 unsigned int noio_flag;
2255
2256 /*
2257 * Never shrink. And mddev_suspend() could deadlock if this is called
2258 * from raid5d. In that case, scribble_disks and scribble_sectors
2259 * should equal to new_disks and new_sectors
2260 */
2261 if (conf->scribble_disks >= new_disks &&
2262 conf->scribble_sectors >= new_sectors)
2263 return 0;
2264 mddev_suspend(conf->mddev);
2265 get_online_cpus();
2266
2267 /*
2268 * scribble_alloc() allocates memory by kvmalloc_array(), if
2269 * the memory allocation triggers memory reclaim I/Os onto
2270 * this raid array, there might be potential deadlock if this
2271 * raid array happens to be suspended during memory allocation.
2272 * Here the scope APIs are used to disable such recursive memory
2273 * reclaim I/Os.
2274 */

> 2275		noio_flag = memalloc_noio_save();
  2276		for_each_present_cpu(cpu) {
  2277			struct raid5_percpu *percpu;
  2278	
  2279			percpu = per_cpu_ptr(conf->percpu, cpu);
  2280			err = scribble_alloc(percpu, new_disks,
  2281					     new_sectors / STRIPE_SECTORS);
  2282			if (err)
  2283				break;
  2284		}
> 2285		memalloc_noio_restore(noio_flag);

2286
2287 put_online_cpus();
2288 mddev_resume(conf->mddev);
2289 if (!err) {
2290 conf->scribble_disks = new_disks;
2291 conf->scribble_sectors = new_sectors;
2292 }
2293 return err;
2294 }
2295

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/[email protected]

#2 lore raw
In reply to Coly Li

drivers/md/raid5.c static int grow_stripes(struct r5conf *conf, int num) · line 2239

Quoted 2 lines · Coly Li
drivers/md/raid5.c +1 -1
-	scribble = kvmalloc_array(cnt, obj_size, flags);
+	scribble = kvmalloc_array(cnt, obj_size, GFP_KERNEL);

Maybe it is simpler to call kvmalloc_array between memalloc_noio_save and memalloc_noio_restore. And seems sched/mm.h need to be included per the report from LKP.

Thanks,
Guoqing

author
#3 lore raw
Quoted 7 lines · Guoqing Jiang
Quoted 2 lines · Coly Li

-    scribble = kvmalloc_array(cnt, obj_size, flags); +    scribble = kvmalloc_array(cnt, obj_size, GFP_KERNEL);

Maybe it is simpler to call kvmalloc_array between memalloc_noio_save and memalloc_noio_restore.
And seems sched/mm.h need to be included per the report from LKP.

The falgs can be,
- GFP_KERNEL: when called from alloc_scratch_buffer()
- GFP_NOIO: when called from resize_chunks().

If the scope APIs are used inside scribble_alloc(), the first call path is restricted as noio, which is not expected. So I only use the scope APIs around the location where GFP_NOIO is used.

Anyway, Michal Hocko suggests to add the scope APIs in mddev_suspend()/mddev_resume(). Then in the whole code path where md raid array is suspend, we don't need to worry the recursive memory reclaim I/Os onto the array. After checking the complicated raid5 code, I come to realize this suggestion makes sense.

I will try to compose a v2 patch following Michal's suggestion.

Signature · 1 line
Coly Li
#4 lore raw
Quoted 7 lines · Coly Li
Quoted 6 lines · Guoqing Jiang
Quoted 2 lines · Coly Li

-    scribble = kvmalloc_array(cnt, obj_size, flags); +    scribble = kvmalloc_array(cnt, obj_size, GFP_KERNEL);

Maybe it is simpler to call kvmalloc_array between memalloc_noio_save and memalloc_noio_restore.
And seems sched/mm.h need to be included per the report from LKP.

Quoted 7 lines

The falgs can be,
- GFP_KERNEL: when called from alloc_scratch_buffer()
- GFP_NOIO: when called from resize_chunks().

If the scope APIs are used inside scribble_alloc(), the first call path is restricted as noio, which is not expected. So I only use the scope APIs around the location where GFP_NOIO is used.

Thanks for the explanation. I assume it can be distinguished by check the flag, eg, FYI.

if (flag == GFP_NOIO)
    memalloc_noio_save();
kvmalloc_array();
if (flag == GFP_NOIO)
    memalloc_noio_restore();

Quoted 5 lines

Anyway, Michal Hocko suggests to add the scope APIs in mddev_suspend()/mddev_resume(). Then in the whole code path where md raid array is suspend, we don't need to worry the recursive memory reclaim I/Os onto the array. After checking the complicated raid5 code, I come to realize this suggestion makes sense.

Hmm, mddev_suspend/resume are called at lots of places, then it's better to check if all the places don't allocate memory with GFP_KERNEL.

And seems In level_store(), sysfs_create_group could be called between suspend and resume, then the two functions (kstrdup_const(name, GFP_KERNEL) and kmem_cache_zalloc(kernfs_node_cache, GFP_KERNEL)) could be triggered by the path:

sysfs_create_group ->internal_create_group -> kernfs_create_dir_ns -> kernfs_new_node -> __kernfs_new_node

Not know memalloc_noio_{save,restore} well, but I guess it is better to use them to mark a small scope, just my two cents.

Thanks,
Guoqing

author
#5 lore raw
Quoted 20 lines · Guoqing Jiang
Quoted 7 lines · Coly Li
Quoted 6 lines · Guoqing Jiang
Quoted 2 lines · Coly Li

-    scribble = kvmalloc_array(cnt, obj_size, flags); +    scribble = kvmalloc_array(cnt, obj_size, GFP_KERNEL);

Maybe it is simpler to call kvmalloc_array between memalloc_noio_save and memalloc_noio_restore.
And seems sched/mm.h need to be included per the report from LKP.

Quoted 7 lines

The falgs can be,
- GFP_KERNEL: when called from alloc_scratch_buffer()
- GFP_NOIO: when called from resize_chunks().

If the scope APIs are used inside scribble_alloc(), the first call path is restricted as noio, which is not expected. So I only use the scope APIs around the location where GFP_NOIO is used.

Thanks for the explanation. I assume it can be distinguished by check the flag,
eg, FYI.

Quoted 5 lines

if (flag == GFP_NOIO)
    memalloc_noio_save();
kvmalloc_array();
if (flag == GFP_NOIO)
    memalloc_noio_restore();

This was similar to my original idea, but finally I decide to accept Michal's suggestion to add them into mddev_suspend()/mddev_resume. Let me explain.

Quoted 9 lines
Quoted 5 lines

Anyway, Michal Hocko suggests to add the scope APIs in mddev_suspend()/mddev_resume(). Then in the whole code path where md raid array is suspend, we don't need to worry the recursive memory reclaim I/Os onto the array. After checking the complicated raid5 code, I come to realize this suggestion makes sense.

Hmm, mddev_suspend/resume are called at lots of places, then it's better to check if all the places don't allocate memory with GFP_KERNEL.

When mddev_suspend() is called, then all I/Os on the suspending raid array have to wait until mddev_resume() is called. Inside the suspending region, all memory allocation should be careful to avoid memory reclaim I/Os going back to the suspending raid array. If such recursive I/Os happen, we will experience deadlock.

Therefore we should be very careful to use GFP_KERNEL to allocate memory during the period when the raid array is suspending. No matter physically continuous or virtually continuous pages.

Quoted 9 lines

And seems In level_store(), sysfs_create_group could be called between suspend
and resume, then the two functions (kstrdup_const(name, GFP_KERNEL) and kmem_cache_zalloc(kernfs_node_cache, GFP_KERNEL)) could be triggered by the path:

sysfs_create_group ->internal_create_group -> kernfs_create_dir_ns -> kernfs_new_node -> __kernfs_new_node

From your information, the above code path is suspicious. Although the deadlock might not happen in practice, unless the raid array is used as the only fs volume, and all memory writeback or swap I/Os will only go into this array.

Checking all the location of memory allocation is trivial but possible, how to prevent new code with memory allocation to avoid the potential deadlock risk during the suspending period is still a problem.

Quoted 3 lines

Not know memalloc_noio_{save,restore} well, but I guess it is better to use them
to mark a small scope, just my two cents.

Since mddev_suspend() is the entry point to prevent I/Os on the array, it is the good location to restrict memory reclaim I/Os of memory allocation. Finally I realize this is a brilliant idea after Michal Hocko suggests me for a while.

Thanks.

Signature · 1 line
Coly Li
#6 lore raw
In reply to Guoqing Jiang

On Thu 09-04-20 23:38:13, Guoqing Jiang wrote:
[...]

Quoted 2 lines

Not know memalloc_noio_{save,restore} well, but I guess it is better to use them to mark a small scope, just my two cents.

This would go against the intentio of the api. It is really meant to define reclaim recursion problematic scope. If there is a clear entry point where any further allocation recursing to FS/IO could deadlock then it should be used at that level. This might be a lock which is taken from the reclaim or like this case a device is suspended and no IO is processed so anything that would wait for an IO or rely on IO making progress in the reclaim path would deadlock.

Please have a look at Documentation/core-api/gfp_mask-from-fs-io.rst and let me know is something could be made more clear or explicit. I am more than happy to improve the documentation.

Signature · 2 lines
Michal Hocko
SUSE Labs
#7 lore raw
Quoted 6 lines · Michal Hocko

On Thu 09-04-20 23:38:13, Guoqing Jiang wrote:
[...]

Quoted 2 lines

Not know memalloc_noio_{save,restore} well, but I guess it is better to use them to mark a small scope, just my two cents.

This would go against the intentio of the api. It is really meant to define reclaim recursion problematic scope.

Well, in current proposal, the scope is just when scribble_allo/kvmalloc_array is called.

memalloc_noio_save
scribble_allo/kvmalloc_array
memalloc_noio_restore

With the new proposal, the marked scope would be bigger than current one since there are lots of places call mddev_suspend/resume.

mddev_suspend
memalloc_noio_save
...
memalloc_noio_restore
mddev_resume

IMHO, if the current proposal works then what is the advantage to increase the scope. If all the callers of mddev_suspend/resume could suffer from the deadlock issue due to recursing fs io, then it is definitely need to use the new proposal.

Quoted 2 lines

If there is a clear entry point where any further allocation recursing to FS/IO could deadlock then it should be used at that level.

Agree.

At the end of mddev_suspend, I guess there is no FS/IO could happen to the array, because mddev->suspended++ and quiesce(mddev, 1) are called previously in mddev_suspend. And it makes me curious to go through to the call chain:

layout_store / chunk_size_store / update_raid_disks / update_array_info / md_check_recovery => pers->check_reshape => raid{5,6}_check_reshape => check_reshape => resize_chunks => scribble_alloc(..., GFP_NOIO)

Perhaps I missed something, but seems all the 5 original callers are not called between mddev_suspend and mddev_resume, if so, then with the new proposal, scribble_alloc() could not be protected by the memalloc_noio_{save,restore}.

Quoted 4 lines

This might be a lock which is
taken from the reclaim or like this case a device is suspended and no IO is processed so anything that would wait for an IO or rely on IO making progress in the reclaim path would deadlock.

Thanks for teaching.

Quoted 3 lines

Please have a look at Documentation/core-api/gfp_mask-from-fs-io.rst and let me know is something could be made more clear or explicit. I am more than happy to improve the documentation.

Thanks again for your lighting, will read it.

Regards,
Guoqing

#8 lore raw
Quoted 19 lines · Guoqing Jiang
Quoted 6 lines · Michal Hocko

On Thu 09-04-20 23:38:13, Guoqing Jiang wrote:
[...]

Quoted 2 lines

Not know memalloc_noio_{save,restore} well, but I guess it is better to use them to mark a small scope, just my two cents.

This would go against the intentio of the api. It is really meant to define reclaim recursion problematic scope.

Well, in current proposal, the scope is just when
scribble_allo/kvmalloc_array is called.

memalloc_noio_save
scribble_allo/kvmalloc_array
memalloc_noio_restore

With the new proposal, the marked scope would be bigger than current one since there
are lots of places call mddev_suspend/resume.

Quoted 8 lines

mddev_suspend
memalloc_noio_save
...
memalloc_noio_restore
mddev_resume

IMHO, if the current proposal works then what is the advantage to increase the scope.

The advantage is twofold. It serves the documentation purpose because it is clear _what_ and _why_ is the actual allocation restricted context. In this case mddev_{suspend,resume} because XYZ and you do not have to care about __GFP_IO for _any_ allocation inside the scope.

Quoted 3 lines

If all the callers of mddev_suspend/resume could suffer from the deadlock issue due to recursing fs io, then it is definitely need to use the new proposal.

Why some of them wouldn't? Isn't the mddev_suspend going to block any IO on the device? The thing is that you cannot tell which devices the allocator can issue IO for therefore GFP_NOIO is a global flag. Please also note that the scope API is bound to a process context so it only affects the current execution.

Signature · 2 lines
Michal Hocko
SUSE Labs
#9 lore raw
Quoted 25 lines · Michal Hocko
Quoted 24 lines · Guoqing Jiang
Quoted 6 lines · Michal Hocko

On Thu 09-04-20 23:38:13, Guoqing Jiang wrote:
[...]

Quoted 2 lines

Not know memalloc_noio_{save,restore} well, but I guess it is better to use them to mark a small scope, just my two cents.

This would go against the intentio of the api. It is really meant to define reclaim recursion problematic scope.

Well, in current proposal, the scope is just when
scribble_allo/kvmalloc_array is called.

memalloc_noio_save
scribble_allo/kvmalloc_array
memalloc_noio_restore

With the new proposal, the marked scope would be bigger than current one since there
are lots of places call mddev_suspend/resume.

mddev_suspend
memalloc_noio_save
...
memalloc_noio_restore
mddev_resume

Quoted 6 lines
Quoted 2 lines

IMHO, if the current proposal works then what is the advantage to increase the scope.

The advantage is twofold. It serves the documentation purpose because it is clear _what_ and _why_ is the actual allocation restricted context. In this case mddev_{suspend,resume} because XYZ and you do not have to care about __GFP_IO for _any_ allocation inside the scope.

Personally, I'd prefer fine grained protection scope, anyway just my own flavor. And I think Song has his opinion about the proposal, I will respect his decision.

Thanks,
Guoqing

#10 lore raw
Quoted 32 lines · Guoqing Jiang
Quoted 27 lines · Michal Hocko
Quoted 26 lines · Guoqing Jiang
Quoted 6 lines · Michal Hocko

On Thu 09-04-20 23:38:13, Guoqing Jiang wrote:
[...]

Quoted 2 lines

Not know memalloc_noio_{save,restore} well, but I guess it is better to use them to mark a small scope, just my two cents.

This would go against the intentio of the api. It is really meant to define reclaim recursion problematic scope.

Well, in current proposal, the scope is just when
scribble_allo/kvmalloc_array is called.

memalloc_noio_save
scribble_allo/kvmalloc_array
memalloc_noio_restore

With the new proposal, the marked scope would be bigger than current one since there
are lots of places call mddev_suspend/resume.

mddev_suspend
memalloc_noio_save
...
memalloc_noio_restore
mddev_resume

IMHO, if the current proposal works then what is the advantage to increase the scope.

Quoted 8 lines
Quoted 4 lines

The advantage is twofold. It serves the documentation purpose because it is clear _what_ and _why_ is the actual allocation restricted context. In this case mddev_{suspend,resume} because XYZ and you do not have to care about __GFP_IO for _any_ allocation inside the scope.

Personally, I'd prefer fine grained protection scope, anyway just my own flavor. And I think Song has his opinion about the proposal, I will respect his decision.

Thanks Coly, Guoqing, and Michal for these great inputs.

Coly's v3 set looks good to me. I will go ahead apply that version to md-next branch.

Thanks again,
Song

#11 lore raw
In reply to Coly Li

Hi Coly,

Thanks for the patch!

Quoted 45 lines · Coly Li

Commit b330e6a49dc3 ("md: convert to kvmalloc") uses kvmalloc_array() to allocate memory with GFP_NOIO flag in resize_chunks() via function scribble_alloc(),
2269 err = scribble_alloc(percpu, new_disks,
2270 new_sectors / STRIPE_SECTORS,
2271 GFP_NOIO);

The purpose of GFP_NOIO flag to kvmalloc_array() is to allocate non-physically continuous pages and avoid extra I/Os of page reclaim which triggered by memory allocation. When system memory is under heavy pressure, non-physically continuous pages allocation is more probably to success than allocating physically continuous pages.

But as a non GFP_KERNEL compatible flag, GFP_NOIO is not acceptible by kvmalloc_node() and the memory allocation indeed is handled with kmalloc_node() to allocate physically continuous pages. This is not the expected behavior of the original purpose when mistakenly using GFP_NOIO flag.

In this patch, the memalloc scope APIs memalloc_noio_save() and memalloc_noio_restore() are used when calling scribble_alloc(). Then when calling kvmalloc_array() with GFP_KERNEL mask, the scope APIs may indicatet the allocating context to avoid memory reclaim related I/Os, to avoid recursive I/O deadlock on the md raid array itself which is calling scribble_alloc() to allocate non-physically continuous pages.

This patch also removes gfp_t flags from scribble_alloc() parameters list, because the invalid GFP_NOIO is replaced by memalloc scope APIs.

Fixes b330e6a49dc3 ("md: convert to kvmalloc")
Cc Kent Overstreet <[email protected]>
Cc Michal Hocko <[email protected]>

---
drivers/md/raid5.c | 22 ++++++++++++++++------

1 file changed, 16 insertions(+), 6 deletions(-)
drivers/md/raid5.c +0 -0
index ba00e9877f02..6b23f8aba169 100644

drivers/md/raid5.c static int grow_stripes(struct r5conf *conf, int num) · line 2229

Quoted 2 lines
drivers/md/raid5.c +0 -0
@@ -2228,14 +2228,15 @@ static int grow_stripes(struct r5conf *conf, int num)
 * of the P and Q blocks.
 */

I noticed the comment before scribble_alloc() is outdated. Maybe fix also that as we are on it?

drivers/md/raid5.c static int grow_stripes(struct r5conf *conf, int num) · line 2237

Quoted 9 lines
drivers/md/raid5.c +2 -1
 static int scribble_alloc(struct raid5_percpu *percpu,
-			  int num, int cnt, gfp_t flags)
+			  int num, int cnt)
 {
 	size_t obj_size =
 		sizeof(struct page *) * (num+2) +
 		sizeof(addr_conv_t) * (num+2);
 	void *scribble;
+	unsigned int noio_flag;

I think we don't use noio_flag in scribble_alloc()?

Thanks,
Song

author
#12 lore raw
Quoted 44 lines · Song Liu

Hi Coly,

Thanks for the patch!

Quoted 40 lines · Coly Li

Commit b330e6a49dc3 ("md: convert to kvmalloc") uses kvmalloc_array() to allocate memory with GFP_NOIO flag in resize_chunks() via function scribble_alloc(),
2269 err = scribble_alloc(percpu, new_disks,
2270 new_sectors / STRIPE_SECTORS,
2271 GFP_NOIO);

The purpose of GFP_NOIO flag to kvmalloc_array() is to allocate non-physically continuous pages and avoid extra I/Os of page reclaim which triggered by memory allocation. When system memory is under heavy pressure, non-physically continuous pages allocation is more probably to success than allocating physically continuous pages.

But as a non GFP_KERNEL compatible flag, GFP_NOIO is not acceptible by kvmalloc_node() and the memory allocation indeed is handled with kmalloc_node() to allocate physically continuous pages. This is not the expected behavior of the original purpose when mistakenly using GFP_NOIO flag.

In this patch, the memalloc scope APIs memalloc_noio_save() and memalloc_noio_restore() are used when calling scribble_alloc(). Then when calling kvmalloc_array() with GFP_KERNEL mask, the scope APIs may indicatet the allocating context to avoid memory reclaim related I/Os, to avoid recursive I/O deadlock on the md raid array itself which is calling scribble_alloc() to allocate non-physically continuous pages.

This patch also removes gfp_t flags from scribble_alloc() parameters list, because the invalid GFP_NOIO is replaced by memalloc scope APIs.

Fixes b330e6a49dc3 ("md: convert to kvmalloc")
Cc Kent Overstreet <[email protected]>
Cc Michal Hocko <[email protected]>

---
drivers/md/raid5.c | 22 ++++++++++++++++------

1 file changed, 16 insertions(+), 6 deletions(-)
Quoted 11 lines

drivers/md/raid5.c static int grow_stripes(struct r5conf *conf, int num) · line 2229

Quoted 7 lines
drivers/md/raid5.c +0 -0
index ba00e9877f02..6b23f8aba169 100644
@@ -2228,14 +2228,15 @@ static int grow_stripes(struct r5conf *conf, int num)
 * of the P and Q blocks.
 */

I noticed the comment before scribble_alloc() is outdated. Maybe fix also that as we are on it?

OK, I will do that.

Quoted 10 lines

drivers/md/raid5.c static int grow_stripes(struct r5conf *conf, int num) · line 2237

Quoted 9 lines
drivers/md/raid5.c +2 -1
 static int scribble_alloc(struct raid5_percpu *percpu,
-			  int num, int cnt, gfp_t flags)
+			  int num, int cnt)
 {
 	size_t obj_size =
 		sizeof(struct page *) * (num+2) +
 		sizeof(addr_conv_t) * (num+2);
 	void *scribble;
+	unsigned int noio_flag;

I think we don't use noio_flag in scribble_alloc()?

You are right of cause. It should not be here :-)

Thanks.

Signature · 1 line
Coly Li