md/raid1,raid10: fix deadlock and bio accounting in read error path
v2 cover md/raid1,raid10: fix deadlock and bio accounting in read error path
Hi,
This series revisits an issue in the read error recovery path for raid1 and raid10 when bios are split and resubmitted.
In v1, the approach was to avoid splitting bios in the raid1 thread to prevent recursion and potential deadlocks. However, this was not ideal and too restrictive.
As suggested by Yu Kuai, this series instead handles the problem in md_handle_request() and allows bio splitting.
Link to Yu Kuai' email: https://lore.kernel.org/linux-raid/[email protected]/T/#m714020a38b60fc5f84b9a24f0c46acbe5d7342d6
This series fixes the above by:
- handling md cloned bios explicitly in md_handle_request()
- using md_cloned_bio() to reliably detect the error path
- fixing bio accounting to avoid double/missing accounting
Changes in v2:
- Split fixes into separate patches for clarity.
- Use md_cloned_bio() consistently to detect cloned bios.
- Recognize that raid10 has the same issue and fix it in this series
- Allow splitting bios.
- Handle md cloned bios explicitly in md_handle_request()
- Link v1: https://lore.kernel.org/linux-raid/[email protected]/
Abd-Alrhman Masalkhi (3):
md/raid1,raid10: fix deadlock in read error recovery path md/raid1,raid10: fix error-path detection with md_cloned_bio() md/raid1,raid10: fix bio accounting for split md cloned bios
drivers/md/md.c | 25 ++++++++++++++++--------- drivers/md/md.h | 5 +++++ drivers/md/raid1.c | 15 +++++++++++---- drivers/md/raid10.c | 28 ++++++++++++++++++---------- 4 files changed, 50 insertions(+), 23 deletions(-)
Signature · 1 line
2.43.0
PATCH v2 1/3 md/raid1,raid10: fix deadlock in read error recovery path
raid1d and raid10d may resubmit a split md cloned bio while handling a read error. In this case, resubmitting the bio can lead to a deadlock if the array is suspended before md_handle_request() acquires an active_io reference via percpu_ref_tryget_live().
Since the cloned bio already holds an active_io reference, trying to acquire another reference via percpu_ref_tryget_live() can lead to a deadlock while the array is suspended.
Fix this by using percpu_ref_get() for md cloned bios.
---
Changes in v2:
- Use md_cloned_bio() consistently to detect cloned bios.
- Recognize that raid10 has the same issue and fix it in this series
- Allow splitting bios.
- Handle md cloned bios explicitly in md_handle_request()
- Link v1: https://lore.kernel.org/linux-raid/[email protected]/
Please let me know if I should add a Suggested-by tag for Yu Kuai, as the solution approach was suggested during review.
Link to Yu Kuai' email: https://lore.kernel.org/linux-raid/[email protected]/T/#m714020a38b60fc5f84b9a24f0c46acbe5d7342d6
Thanks
Abd-alrhman
--- drivers/md/md.c | 25 ++++++++++++++++--------- drivers/md/md.h | 5 +++++ 2 files changed, 21 insertions(+), 9 deletions(-)
drivers/md/md.c +16 -9
@@ -396,17 +396,24 @@ static bool is_suspended(struct mddev *mddev, struct bio *bio)bool md_handle_request (struct mddev *mddev,struct bio *bio) { check_suspended: -if (is_suspended (mddev, bio)) { -/* Bail out if REQ_NOWAIT is set for the bio */ -if (bio->bi_opf &REQ_NOWAIT ) { -bio_wouldblock_error (bio); -return true; +if (unlikely (md_cloned_bio (mddev, bio))) { +/* +* This bio is an MD cloned bio and already holds an +* active_io reference, so percpu_ref_get() is safe here. +*/ +percpu_ref_get (&mddev->active_io); + }else { +if (is_suspended (mddev, bio)) { +/* Bail out if REQ_NOWAIT is set for the bio */ +if (bio->bi_opf &REQ_NOWAIT ) { +bio_wouldblock_error (bio); +return true; + } +wait_event (mddev->sb_wait, !is_suspended (mddev, bio)); } -wait_event (mddev->sb_wait, !is_suspended (mddev, bio)); +if (!percpu_ref_tryget_live (&mddev->active_io)) + goto check_suspended; } -if (!percpu_ref_tryget_live (&mddev->active_io)) - goto check_suspended; -if (!mddev->pers->make_request (mddev, bio)) {percpu_ref_put (&mddev->active_io);if (mddev_is_dm (mddev) && mddev->pers->prepare_suspend)
drivers/md/md.h +5 -0
@@ -1038,6 +1038,11 @@ void mddev_update_io_opt(struct mddev *mddev, unsigned int nr_stripes);extern const struct block_device_operations md_fops; +static inline bool md_cloned_bio (struct mddev *mddev,struct bio *bio) +{ +return bio->bi_pool == &mddev->io_clone_set; +} +/* * MD devices can be used undeneath by DM, in which case ->gendisk is NULL. */
Signature · 1 line
2.43.0
Re: [PATCH v2 1/3] md/raid1,raid10: fix deadlock in read error recovery path
Quoted 90 lines · Abd-Alrhman Masalkhi
raid1d and raid10d may resubmit a split md cloned bio while handling a read error. In this case, resubmitting the bio can lead to a deadlock if the array is suspended before
md_handle_request()acquires anactive_ioreference viapercpu_ref_tryget_live().Since the cloned bio already holds an
active_ioreference, trying to acquire another reference viapercpu_ref_tryget_live()can lead to a deadlock while the array is suspended.Fix this by using
percpu_ref_get()for md cloned bios.Fixes bb2a9acefaf9 ("md/raid1: switch to use md_account_bio() for io accounting")Fixes 820455238366 ("md/raid10: switch to use md_account_bio() for io accounting")Signed-off-by Abd-Alrhman Masalkhi <[email protected]>---
Changes in v2:
- Usemd_cloned_bio()consistently to detect cloned bios.
- Recognize that raid10 has the same issue and fix it in this series
- Allow splitting bios.
- Handle md cloned bios explicitly inmd_handle_request()
- Link v1: https://lore.kernel.org/linux-raid/[email protected]/Please let me know if I should add a Suggested-by tag for Yu Kuai, as the solution approach was suggested during review.
Link to Yu Kuai' email: https://lore.kernel.org/linux-raid/[email protected]/T/#m714020a38b60fc5f84b9a24f0c46acbe5d7342d6
Thanks
Abd-alrhman--- drivers/md/md.c | 25 ++++++++++++++++--------- drivers/md/md.h | 5 +++++ 2 files changed, 21 insertions(+), 9 deletions(-)drivers/md/md.c +16
-9@@ -396,17 +396,24 @@ static bool is_suspended(struct mddev *mddev, struct bio *bio)bool md_handle_request (struct mddev *mddev,struct bio *bio) { check_suspended: -if (is_suspended (mddev, bio)) { -/* Bail out if REQ_NOWAIT is set for the bio */ -if (bio->bi_opf &REQ_NOWAIT ) { -bio_wouldblock_error (bio); -return true; +if (unlikely (md_cloned_bio (mddev, bio))) { +/* +* This bio is an MD cloned bio and already holds an +* active_io reference, so percpu_ref_get() is safe here. +*/ +percpu_ref_get (&mddev->active_io); + }else { +if (is_suspended (mddev, bio)) { +/* Bail out if REQ_NOWAIT is set for the bio */ +if (bio->bi_opf &REQ_NOWAIT ) { +bio_wouldblock_error (bio); +return true; + } +wait_event (mddev->sb_wait, !is_suspended (mddev, bio)); } -wait_event (mddev->sb_wait, !is_suspended (mddev, bio)); +if (!percpu_ref_tryget_live (&mddev->active_io)) + goto check_suspended; } -if (!percpu_ref_tryget_live (&mddev->active_io)) - goto check_suspended; -if (!mddev->pers->make_request (mddev, bio)) {percpu_ref_put (&mddev->active_io);if (mddev_is_dm (mddev) && mddev->pers->prepare_suspend)drivers/md/md.h +5
-1@@ -1038,6 +1038,11 @@ void mddev_update_io_opt(struct mddev *mddev, unsigned int nr_stripes);extern const struct block_device_operations md_fops; +static inline bool md_cloned_bio (struct mddev *mddev,struct bio *bio) +{ +return bio->bi_pool == &mddev->io_clone_set; +} +/* * MD devices can be used undeneath by DM, in which case ->gendisk is NULL. */ --
Quoted 3 lines
2.43.0
This patch looks good to me.
Re: [PATCH v2 1/3] md/raid1,raid10: fix deadlock in read error recovery path
In reply to Abd-Alrhman MasalkhiQuoted 25 lines · Abd-Alrhman Masalkhi
raid1d and raid10d may resubmit a split md cloned bio while handling a read error. In this case, resubmitting the bio can lead to a deadlock if the array is suspended before
md_handle_request()acquires anactive_ioreference viapercpu_ref_tryget_live().Since the cloned bio already holds an
active_ioreference, trying to acquire another reference viapercpu_ref_tryget_live()can lead to a deadlock while the array is suspended.Fix this by using
percpu_ref_get()for md cloned bios.Fixes bb2a9acefaf9 ("md/raid1: switch to use md_account_bio() for io accounting")Fixes 820455238366 ("md/raid10: switch to use md_account_bio() for io accounting")Signed-off-by Abd-Alrhman Masalkhi<[email protected]>---
Changes in v2:
- Usemd_cloned_bio()consistently to detect cloned bios.
- Recognize that raid10 has the same issue and fix it in this series
- Allow splitting bios.
- Handle md cloned bios explicitly inmd_handle_request()
- Link v1:https://lore.kernel.org/linux-raid/[email protected]/Please let me know if I should add a Suggested-by tag for Yu Kuai, as the solution approach was suggested during review.
Quoted 8 lines
Link to Yu Kuai' email:https://lore.kernel.org/linux-raid/[email protected]/T/#m714020a38b60fc5f84b9a24f0c46acbe5d7342d6
Thanks
Abd-alrhman--- drivers/md/md.c | 25 ++++++++++++++++--------- drivers/md/md.h | 5 +++++ 2 files changed, 21 insertions(+), 9 deletions(-)
Signature · 2 lines
Thansk, Kuai
PATCH v2 2/3 md/raid1,raid10: fix error-path detection with md_cloned_bio()
In reply to Abd-Alrhman MasalkhiDetect the error path using md_cloned_bio() instead of relying on r1_bio in raid1 or r10_bio->read_slot in raid10, which may be NULL or -1 after splitting and resubmitting a failed bio.
As a result, the error path may not be recognized and memory allocations can incorrectly use GFP_NOIO instead of (GFP_NOIO | __GFP_HIGH), which can lead to a deadlock under memory pressure.
---
This patch depends on patch 1.
Changes in v2:
- New patch.
--- drivers/md/raid1.c | 13 ++++++++++--- drivers/md/raid10.c | 20 ++++++++++++++------ 2 files changed, 24 insertions(+), 9 deletions(-)
drivers/md/raid1.c +10 -3
@@ -1321,11 +1321,18 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,bool r1bio_existed = !!r1_bio;/* -* If r1_bio is set, we are blocking the raid1d thread -* so there is a tiny risk of deadlock. So ask for +* An md cloned bio indicates we are in the error path. +* This is more reliable than checking r1_bio, which might +* be NULL even in the error path if a failed bio was split. +*/ +bool err_path =md_cloned_bio (mddev, bio); + +/* +* If we are in the error path, we are blocking the raid1d +* thread so there is a tiny risk of deadlock. So ask for * emergency memory if needed. */ -gfp_t gfp = r1_bio ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ; +gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ;/* * Still need barrier for READ in case that whole
drivers/md/raid10.c +14 -6
@@ -1155,7 +1155,20 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,char b[BDEVNAME_SIZE ];int slot = r10_bio->read_slot;struct md_rdev *err_rdev =NULL ; -gfp_t gfp =GFP_NOIO ; + +/* +* An md cloned bio indicates we are in the error path. +* This is more reliable than checking slot, which might +* be -1 even in the error path if a failed bio was split. +*/ +bool err_path =md_cloned_bio (mddev, bio); + +/* +* If we are in the error path, we are blocking the raid10d +* thread so there is a tiny risk of deadlock. So ask for +* emergency memory if needed. +*/ +gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ;if (slot >=0 && r10_bio->devs[slot].rdev) {/*
@@ -1166,11 +1179,6 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,* we lose the device name in error messages. */ int disk; -/* -* As we are blocking raid10, it is a little safer to -* use __GFP_HIGH. -*/ - gfp =GFP_NOIO | __GFP_HIGH; disk = r10_bio->devs[slot].devnum; err_rdev = conf->mirrors[disk].rdev;
Signature · 1 line
2.43.0
Re: [PATCH v2 2/3] md/raid1,raid10: fix error-path detection with md_cloned_bio()
Quoted 36 lines · Abd-Alrhman Masalkhi
Detect the error path using
md_cloned_bio()instead of relying onr1_bioin raid1 orr10_bio->read_slotin raid10, which may be NULL or -1 after splitting and resubmitting a failed bio.As a result, the error path may not be recognized and memory allocations can incorrectly use
GFP_NOIOinstead of (GFP_NOIO|__GFP_HIGH), which can lead to a deadlock under memory pressure.Fixes 689389a06ce7 ("md/raid1: simplify handle_read_error().")Fixes 545250f24809 ("md/raid10: simplify handle_read_error()")Signed-off-by Abd-Alrhman Masalkhi <[email protected]>---
This patch depends on patch 1.Changes in v2:
- New patch.--- drivers/md/raid1.c | 13 ++++++++++--- drivers/md/raid10.c | 20 ++++++++++++++------ 2 files changed, 24 insertions(+), 9 deletions(-)drivers/md/raid1.c +3
-2@@ -1321,11 +1321,18 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,bool r1bio_existed = !!r1_bio;/* -* If r1_bio is set, we are blocking the raid1d thread -* so there is a tiny risk of deadlock. So ask for +* An md cloned bio indicates we are in the error path. +* This is more reliable than checking r1_bio, which might +* be NULL even in the error path if a failed bio was split.
drivers/md/raid1.c static void raid1_read_request(struct mddev *mddev, struct bio *bio, · line 1335
Quoted 10 lines
drivers/md/raid1.c +7
-1+*/ +bool err_path =md_cloned_bio (mddev, bio); + +/* +* If we are in the error path, we are blocking the raid1d +* thread so there is a tiny risk of deadlock. So ask for * emergency memory if needed. */ -gfp_t gfp = r1_bio ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ; +gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ;
Hi
This patch looks good to me.
Quoted 45 lines
drivers/md/raid1.c +0
-0/* * Still need barrier for READ in case that whole drivers/md/raid10.c +14
-7@@ -1155,7 +1155,20 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,char b[BDEVNAME_SIZE ];int slot = r10_bio->read_slot;struct md_rdev *err_rdev =NULL ; -gfp_t gfp =GFP_NOIO ; + +/* +* An md cloned bio indicates we are in the error path. +* This is more reliable than checking slot, which might +* be -1 even in the error path if a failed bio was split. +*/ +bool err_path =md_cloned_bio (mddev, bio); + +/* +* If we are in the error path, we are blocking the raid10d +* thread so there is a tiny risk of deadlock. So ask for +* emergency memory if needed. +*/ +gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ;if (slot >=0 && r10_bio->devs[slot].rdev) {/* @@ -1166,11 +1179,6 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,* we lose the device name in error messages. */ int disk; -/* -* As we are blocking raid10, it is a little safer to -* use __GFP_HIGH. -*/ - gfp =GFP_NOIO | __GFP_HIGH; disk = r10_bio->devs[slot].devnum; err_rdev = conf->mirrors[disk].rdev; --2.43.0
Re: [PATCH v2 2/3] md/raid1,raid10: fix error-path detection with md_cloned_bio()
In reply to Abd-Alrhman MasalkhiHi,
Quoted 8 lines · Abd-Alrhman Masalkhi
Detect the error path using
md_cloned_bio()instead of relying onr1_bioin raid1 orr10_bio->read_slotin raid10, which may be NULL or -1 after splitting and resubmitting a failed bio.As a result, the error path may not be recognized and memory allocations can incorrectly use
GFP_NOIOinstead of (GFP_NOIO|__GFP_HIGH), which can lead to a deadlock under memory pressure.
I don't think this patch will fix this problem. Because split bio is a new bio that will be resubmit by md_submit_bio(), such bio is not a md_cloned_bio yet until md_account_bio().
Quoted 78 lines
Fixes 689389a06ce7 ("md/raid1: simplify handle_read_error().")Fixes 545250f24809 ("md/raid10: simplify handle_read_error()")Signed-off-by Abd-Alrhman Masalkhi <[email protected]>---
This patch depends on patch 1.Changes in v2:
- New patch.--- drivers/md/raid1.c | 13 ++++++++++--- drivers/md/raid10.c | 20 ++++++++++++++------ 2 files changed, 24 insertions(+), 9 deletions(-)drivers/md/raid1.c +10
-3@@ -1321,11 +1321,18 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,bool r1bio_existed = !!r1_bio;/* -* If r1_bio is set, we are blocking the raid1d thread -* so there is a tiny risk of deadlock. So ask for +* An md cloned bio indicates we are in the error path. +* This is more reliable than checking r1_bio, which might +* be NULL even in the error path if a failed bio was split. +*/ +bool err_path =md_cloned_bio (mddev, bio); + +/* +* If we are in the error path, we are blocking the raid1d +* thread so there is a tiny risk of deadlock. So ask for * emergency memory if needed. */ -gfp_t gfp = r1_bio ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ; +gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ;/* * Still need barrier for READ in case that whole drivers/md/raid10.c +14
-6@@ -1155,7 +1155,20 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,char b[BDEVNAME_SIZE ];int slot = r10_bio->read_slot;struct md_rdev *err_rdev =NULL ; -gfp_t gfp =GFP_NOIO ; + +/* +* An md cloned bio indicates we are in the error path. +* This is more reliable than checking slot, which might +* be -1 even in the error path if a failed bio was split. +*/ +bool err_path =md_cloned_bio (mddev, bio); + +/* +* If we are in the error path, we are blocking the raid10d +* thread so there is a tiny risk of deadlock. So ask for +* emergency memory if needed. +*/ +gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ;if (slot >=0 && r10_bio->devs[slot].rdev) {/* @@ -1166,11 +1179,6 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,* we lose the device name in error messages. */ int disk; -/* -* As we are blocking raid10, it is a little safer to -* use __GFP_HIGH. -*/ - gfp =GFP_NOIO | __GFP_HIGH; disk = r10_bio->devs[slot].devnum; err_rdev = conf->mirrors[disk].rdev;
Signature · 2 lines
Thansk, Kuai
Xiao Ni
reviewerRe: [PATCH v2 2/3] md/raid1,raid10: fix error-path detection with md_cloned_bio()
Quoted 8 lines · Yu Kuai
Hi,
Quoted 4 lines · Abd-Alrhman Masalkhi
Detect the error path using
md_cloned_bio()instead of relying onr1_bioin raid1 orr10_bio->read_slotin raid10, which may be NULL or -1 after splitting and resubmitting a failed bio.
Quoted 8 lines
Quoted 4 lines
As a result, the error path may not be recognized and memory allocations can incorrectly use
GFP_NOIOinstead of (GFP_NOIO|__GFP_HIGH), which can lead to a deadlock under memory pressure.I don't think this patch will fix this problem. Because split bio is a new bio that will be resubmit by
md_submit_bio(), such bio is not amd_cloned_bioyet untilmd_account_bio().
Hi Kuai
bio = bio_submit_split_bioset(bio, max_sectors,
&conf->bio_split);bio_submit_split_bioset returns the split bio and the remainder bio will be submitted again which is already a md_cloned_bio.
Regards
Xiao
Quoted 84 lines
Quoted 69 lines
Fixes 689389a06ce7 ("md/raid1: simplify handle_read_error().")Fixes 545250f24809 ("md/raid10: simplify handle_read_error()")Signed-off-by Abd-Alrhman Masalkhi <[email protected]>---
This patch depends on patch 1.Changes in v2:
- New patch.--- drivers/md/raid1.c | 13 ++++++++++--- drivers/md/raid10.c | 20 ++++++++++++++------ 2 files changed, 24 insertions(+), 9 deletions(-)drivers/md/raid1.c +10
-3@@ -1321,11 +1321,18 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,bool r1bio_existed = !!r1_bio;/* -* If r1_bio is set, we are blocking the raid1d thread -* so there is a tiny risk of deadlock. So ask for +* An md cloned bio indicates we are in the error path. +* This is more reliable than checking r1_bio, which might +* be NULL even in the error path if a failed bio was split. +*/ +bool err_path =md_cloned_bio (mddev, bio); + +/* +* If we are in the error path, we are blocking the raid1d +* thread so there is a tiny risk of deadlock. So ask for * emergency memory if needed. */ -gfp_t gfp = r1_bio ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ; +gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ;/* * Still need barrier for READ in case that whole drivers/md/raid10.c +14
-1@@ -1155,7 +1155,20 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,char b[BDEVNAME_SIZE ];int slot = r10_bio->read_slot;struct md_rdev *err_rdev =NULL ; -gfp_t gfp =GFP_NOIO ; + +/* +* An md cloned bio indicates we are in the error path. +* This is more reliable than checking slot, which might +* be -1 even in the error path if a failed bio was split. +*/ +bool err_path =md_cloned_bio (mddev, bio); + +/* +* If we are in the error path, we are blocking the raid10d +* thread so there is a tiny risk of deadlock. So ask for +* emergency memory if needed. +*/ +gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ;if (slot >=0 && r10_bio->devs[slot].rdev) {/* @@ -1166,11 +1179,6 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,* we lose the device name in error messages. drivers/md/raid10.c static void raid10_read_request(struct mddev *mddev, struct bio *bio, · line 1184
Quoted 10 lines
drivers/md/raid10.c +0
-5*/ int disk; -/* -* As we are blocking raid10, it is a little safer to -* use __GFP_HIGH. -*/ - gfp =GFP_NOIO | __GFP_HIGH; disk = r10_bio->devs[slot].devnum; err_rdev = conf->mirrors[disk].rdev;--
Thansk,
Kuai
Yu Kuai
maintainerRe: [PATCH v2 2/3] md/raid1,raid10: fix error-path detection with md_cloned_bio()
Hi,
Quoted 15 lines · Xiao Ni
Quoted 7 lines · Yu Kuai
Hi,
Quoted 4 lines · Abd-Alrhman Masalkhi
Detect the error path using
md_cloned_bio()instead of relying onr1_bioin raid1 orr10_bio->read_slotin raid10, which may be NULL or -1 after splitting and resubmitting a failed bio.Quoted 7 lines
Quoted 4 lines
As a result, the error path may not be recognized and memory allocations can incorrectly use
GFP_NOIOinstead of (GFP_NOIO|__GFP_HIGH), which can lead to a deadlock under memory pressure.I don't think this patch will fix this problem. Because split bio is a new bio that will be resubmit by
md_submit_bio(), such bio is not amd_cloned_bioyet untilmd_account_bio().
Quoted 6 lines
Hi Kuai
bio = bio_submit_split_bioset(bio,
max_sectors,
&conf->bio_split);bio_submit_split_biosetreturns the split bio and the remainder bio will be submitted again which is already amd_cloned_bio.
Okay, I understand now this is a special case that current->bio_list is NULL. However, the problem still exist in the case original IO split again while resubmitting, and in this case current->bio_list will be set so new split io will be resubmit by md_submit_bio(). Which means patch 1 can't fix this case as well.
Perhaps it's better to check task_struct instead of bio_set.
Quoted 84 lines
Regards
XiaoQuoted 81 lines
Quoted 67 lines
Fixes 689389a06ce7 ("md/raid1: simplify handle_read_error().")Fixes 545250f24809 ("md/raid10: simplify handle_read_error()")Signed-off-by Abd-Alrhman Masalkhi <[email protected]>---
This patch depends on patch 1.Changes in v2:
- New patch.--- drivers/md/raid1.c | 13 ++++++++++--- drivers/md/raid10.c | 20 ++++++++++++++------ 2 files changed, 24 insertions(+), 9 deletions(-)drivers/md/raid1.c +10
-3@@ -1321,11 +1321,18 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,bool r1bio_existed = !!r1_bio;/* -* If r1_bio is set, we are blocking the raid1d thread -* so there is a tiny risk of deadlock. So ask for +* An md cloned bio indicates we are in the error path. +* This is more reliable than checking r1_bio, which might +* be NULL even in the error path if a failed bio was split. +*/ +bool err_path =md_cloned_bio (mddev, bio); + +/* +* If we are in the error path, we are blocking the raid1d +* thread so there is a tiny risk of deadlock. So ask for * emergency memory if needed. */ -gfp_t gfp = r1_bio ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ; +gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ;/* * Still need barrier for READ in case that whole drivers/md/raid10.c +14
-1@@ -1155,7 +1155,20 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,char b[BDEVNAME_SIZE ];int slot = r10_bio->read_slot;struct md_rdev *err_rdev =NULL ; -gfp_t gfp =GFP_NOIO ; + +/* +* An md cloned bio indicates we are in the error path. +* This is more reliable than checking slot, which might +* be -1 even in the error path if a failed bio was split. +*/ +bool err_path =md_cloned_bio (mddev, bio); + +/* +* If we are in the error path, we are blocking the raid10d +* thread so there is a tiny risk of deadlock. So ask for +* emergency memory if needed. +*/ +gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ;if (slot >=0 && r10_bio->devs[slot].rdev) {/* @@ -1166,11 +1179,6 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,* we lose the device name in error messages. drivers/md/raid10.c static void raid10_read_request(struct mddev *mddev, struct bio *bio, · line 1184
Quoted 10 lines
drivers/md/raid10.c +0
-5*/ int disk; -/* -* As we are blocking raid10, it is a little safer to -* use __GFP_HIGH. -*/ - gfp =GFP_NOIO | __GFP_HIGH; disk = r10_bio->devs[slot].devnum; err_rdev = conf->mirrors[disk].rdev;--
Thansk,
Kuai
Signature · 2 lines
Thansk, Kuai
Re: [PATCH v2 2/3] md/raid1,raid10: fix error-path detection with md_cloned_bio()
Hi Kuai
Quoted 25 lines · Yu Kuai
Hi,
Quoted 15 lines · Xiao Ni
Quoted 7 lines · Yu Kuai
Hi,
Quoted 4 lines · Abd-Alrhman Masalkhi
Detect the error path using
md_cloned_bio()instead of relying onr1_bioin raid1 orr10_bio->read_slotin raid10, which may be NULL or -1 after splitting and resubmitting a failed bio.Quoted 7 lines
Quoted 4 lines
As a result, the error path may not be recognized and memory allocations can incorrectly use
GFP_NOIOinstead of (GFP_NOIO|__GFP_HIGH), which can lead to a deadlock under memory pressure.I don't think this patch will fix this problem. Because split bio is a new bio that will be resubmit by
md_submit_bio(), such bio is not amd_cloned_bioyet untilmd_account_bio().Quoted 6 lines
Hi Kuai
bio = bio_submit_split_bioset(bio,
max_sectors,
&conf->bio_split);bio_submit_split_biosetreturns the split bio and the remainder bio will be submitted again which is already amd_cloned_bio.
Quoted 5 lines
Okay, I understand now this is a special case that
current->bio_listis NULL. However, the problem still exist in the case original IO split again while resubmitting, and in this casecurrent->bio_listwill be set so new split io will be resubmit bymd_submit_bio(). Which means patch 1 can't fix this case as well.
The original bio resubmitted via bio_list is not executing in the raid1d thread context, so blocking on is_suspended() is correct and will not cause a deadlock. The suspension deadlock risk only exists for md_cloned_bio executing in the raid1d thread context in the error path. And in this case it is always true that if we are executing in the raid1d thread context the bio will already be a md_cloned_bio on entry to md_handle_request(), even if current->bio_list was set, because (and as @Xiao has mentioned) every time we split a bio in the error path we are resubmitting the md_cloned_bio remainder, never the original bio.
Quoted 90 lines
Perhaps it's better to check
task_structinstead ofbio_set.Quoted 84 lines
Regards
XiaoQuoted 81 lines
Quoted 67 lines
Fixes 689389a06ce7 ("md/raid1: simplify handle_read_error().")Fixes 545250f24809 ("md/raid10: simplify handle_read_error()")Signed-off-by Abd-Alrhman Masalkhi <[email protected]>---
This patch depends on patch 1.Changes in v2:
- New patch.--- drivers/md/raid1.c | 13 ++++++++++--- drivers/md/raid10.c | 20 ++++++++++++++------ 2 files changed, 24 insertions(+), 9 deletions(-)drivers/md/raid1.c +10
-3@@ -1321,11 +1321,18 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,bool r1bio_existed = !!r1_bio;/* -* If r1_bio is set, we are blocking the raid1d thread -* so there is a tiny risk of deadlock. So ask for +* An md cloned bio indicates we are in the error path. +* This is more reliable than checking r1_bio, which might +* be NULL even in the error path if a failed bio was split. +*/ +bool err_path =md_cloned_bio (mddev, bio); + +/* +* If we are in the error path, we are blocking the raid1d +* thread so there is a tiny risk of deadlock. So ask for * emergency memory if needed. */ -gfp_t gfp = r1_bio ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ; +gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ;/* * Still need barrier for READ in case that whole drivers/md/raid10.c +14
-1@@ -1155,7 +1155,20 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,char b[BDEVNAME_SIZE ];int slot = r10_bio->read_slot;struct md_rdev *err_rdev =NULL ; -gfp_t gfp =GFP_NOIO ; + +/* +* An md cloned bio indicates we are in the error path. +* This is more reliable than checking slot, which might +* be -1 even in the error path if a failed bio was split. +*/ +bool err_path =md_cloned_bio (mddev, bio); + +/* +* If we are in the error path, we are blocking the raid10d +* thread so there is a tiny risk of deadlock. So ask for +* emergency memory if needed. +*/ +gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) :GFP_NOIO ;if (slot >=0 && r10_bio->devs[slot].rdev) {/* @@ -1166,11 +1179,6 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,* we lose the device name in error messages. drivers/md/raid10.c static void raid10_read_request(struct mddev *mddev, struct bio *bio, · line 1184
Quoted 10 lines
drivers/md/raid10.c +0
-5*/ int disk; -/* -* As we are blocking raid10, it is a little safer to -* use __GFP_HIGH. -*/ - gfp =GFP_NOIO | __GFP_HIGH; disk = r10_bio->devs[slot].devnum; err_rdev = conf->mirrors[disk].rdev;--
Thansk,
KuaiSignature · 2 lines
Thansk, Kuai
Signature · 2 lines
Best Regards, Abd-Alrhman
Yu Kuai
maintainerRe: [PATCH v2 2/3] md/raid1,raid10: fix error-path detection with md_cloned_bio()
Hi,
Quoted 9 lines · Abd-Alrhman Masalkhi
The original bio resubmitted via
bio_listis not executing in the raid1d thread context, so blocking onis_suspended()is correct and will not cause a deadlock. The suspension deadlock risk only exists formd_cloned_bioexecuting in the raid1d thread context in the error path. And in this case it is always true that if we are executing in the raid1d thread context the bio will already be amd_cloned_bioon entry tomd_handle_request(), even ifcurrent->bio_listwas set, because (and as @Xiao has mentioned) every time we split a bio in the error path we are resubmitting themd_cloned_bioremainder, never the original bio.
I do understand this, if original bio or resubmitted bio only split at raid1_read_request, there is no problem. What I feel is not safe is that bio can split somewhere else.
This is possible for raid5, However, take a closer look at raid1/10, I feel it's not possible for resubmitted io to split in bio_split_to_limits. Even if it happens, it's probably someone else to blame.
Signature · 2 lines
Thansk, Kuai
Re: [PATCH v2 2/3] md/raid1,raid10: fix error-path detection with md_cloned_bio()
Quoted 13 lines · Yu Kuai
Hi,
Quoted 9 lines · Abd-Alrhman Masalkhi
The original bio resubmitted via
bio_listis not executing in the raid1d thread context, so blocking onis_suspended()is correct and will not cause a deadlock. The suspension deadlock risk only exists formd_cloned_bioexecuting in the raid1d thread context in the error path. And in this case it is always true that if we are executing in the raid1d thread context the bio will already be amd_cloned_bioon entry tomd_handle_request(), even ifcurrent->bio_listwas set, because (and as @Xiao has mentioned) every time we split a bio in the error path we are resubmitting themd_cloned_bioremainder, never the original bio.
Quoted 8 lines
I do understand this, if original bio or resubmitted bio only split at
raid1_read_request, there is no problem. What I feel is not safe is that bio can split somewhere else.This is possible for raid5, However, take a closer look at raid1/10, I feel it's not possible for resubmitted io to split in
bio_split_to_limits. Even if it happens, it's probably someone else to blame.
Agreed. Since bio_split_to_limits() has been mentioned, I am wondering whether a resubmitted md_cloned_bio could end up being split there for raid456. I have not looked into the raid456 code yet; after finishing raid1 and raid10, I will move on to the others.
I am considering skipping the redundant call to bio_split_to_limits() when md_cloned_bio() is true. However, I am not sure whether this would be safe for raid456.
Quoted 3 lines
Signature · 2 lines
Thansk, Kuai
Signature · 2 lines
Best Regards, Abd-Alrhman
PATCH v2 3/3 md/raid1,raid10: fix bio accounting for split md cloned bios
In reply to Abd-Alrhman MasalkhiUse md_cloned_bio() to control bio accounting instead of relying on r1bio_existed in raid1 or the io_accounting flag in raid10.
The previous logic does not reliably reflect whether a bio is an md cloned bio. When a failed bio is split and resubmitted via bio_submit_split_bioset() on the error path, this can lead to either double accounting for md cloned bios, or missing accounting for bios returned from bio_submit_split_bioset()
Fix this by using md_cloned_bio() to detect md cloned bios and skip accounting accordingly.
---
This patch depends on patch 1.
Changes in v2:
- New patch.
--- drivers/md/raid1.c | 2 +- drivers/md/raid10.c | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-)
drivers/md/raid1.c +1 -1
@@ -1396,7 +1396,7 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio, } r1_bio->read_disk = rdisk; -if (!r1bio_existed) { +if (likely (!md_cloned_bio (mddev, bio))) {md_account_bio (mddev, &bio); r1_bio->master_bio = bio; }
drivers/md/raid10.c +4 -4
@@ -1146,7 +1146,7 @@ static bool regular_request_wait(struct mddev *mddev, struct r10conf *conf, }static void raid10_read_request (struct mddev *mddev,struct bio *bio, -struct r10bio *r10_bio,bool io_accounting) +struct r10bio *r10_bio) {struct r10conf *conf = mddev->private;struct bio *read_bio;
@@ -1226,7 +1226,7 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, } slot = r10_bio->read_slot; -if (io_accounting) { +if (likely (!md_cloned_bio (mddev, bio))) {md_account_bio (mddev, &bio); r10_bio->master_bio = bio; }
@@ -1552,7 +1552,7 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors) conf->geo.raid_disks);if (bio_data_dir (bio) ==READ ) -raid10_read_request (mddev, bio, r10_bio, true); +raid10_read_request (mddev, bio, r10_bio);else raid10_write_request (mddev, bio, r10_bio); }
@@ -2872,7 +2872,7 @@ static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio)rdev_dec_pending (rdev, mddev); r10_bio->state =0 ; -raid10_read_request (mddev, r10_bio->master_bio, r10_bio, false); +raid10_read_request (mddev, r10_bio->master_bio, r10_bio);/* * allow_barrier after re-submit to ensure no sync io * can be issued while regular io pending.
Signature · 1 line
2.43.0
Re: [PATCH v2 3/3] md/raid1,raid10: fix bio accounting for split md cloned bios
Quoted 80 lines · Abd-Alrhman Masalkhi
Use
md_cloned_bio()to control bio accounting instead of relying onr1bio_existedin raid1 or theio_accountingflag in raid10.The previous logic does not reliably reflect whether a bio is an md cloned bio. When a failed bio is split and resubmitted via
bio_submit_split_bioset()on the error path, this can lead to either double accounting for md cloned bios, or missing accounting for bios returned frombio_submit_split_bioset()Fix this by using
md_cloned_bio()to detect md cloned bios and skip accounting accordingly.Fixes bb2a9acefaf9 ("md/raid1: switch to use md_account_bio() for io accounting")Fixes 820455238366 ("md/raid10: switch to use md_account_bio() for io accounting")Signed-off-by Abd-Alrhman Masalkhi <[email protected]>---
This patch depends on patch 1.Changes in v2:
- New patch.--- drivers/md/raid1.c | 2 +- drivers/md/raid10.c | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-)drivers/md/raid1.c +1
-1@@ -1396,7 +1396,7 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio, } r1_bio->read_disk = rdisk; -if (!r1bio_existed) { +if (likely (!md_cloned_bio (mddev, bio))) {md_account_bio (mddev, &bio); r1_bio->master_bio = bio; }drivers/md/raid10.c +4
-5@@ -1146,7 +1146,7 @@ static bool regular_request_wait(struct mddev *mddev, struct r10conf *conf, }static void raid10_read_request (struct mddev *mddev,struct bio *bio, -struct r10bio *r10_bio,bool io_accounting) +struct r10bio *r10_bio) {struct r10conf *conf = mddev->private;struct bio *read_bio;@@ -1226,7 +1226,7 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, } slot = r10_bio->read_slot; -if (io_accounting) { +if (likely (!md_cloned_bio (mddev, bio))) {md_account_bio (mddev, &bio); r10_bio->master_bio = bio; }@@ -1552,7 +1552,7 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors) conf->geo.raid_disks);if (bio_data_dir (bio) ==READ ) -raid10_read_request (mddev, bio, r10_bio, true); +raid10_read_request (mddev, bio, r10_bio);else raid10_write_request (mddev, bio, r10_bio); }@@ -2872,7 +2872,7 @@ static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio)rdev_dec_pending (rdev, mddev); r10_bio->state =0 ; -raid10_read_request (mddev, r10_bio->master_bio, r10_bio, false); +raid10_read_request (mddev, r10_bio->master_bio, r10_bio);/* * allow_barrier after re-submit to ensure no sync io * can be issued while regular io pending. --
Quoted 3 lines
2.43.0
This patch looks good to me.
Re: [PATCH v2 3/3] md/raid1,raid10: fix bio accounting for split md cloned bios
In reply to Abd-Alrhman MasalkhiHi,
Quoted 9 lines · Abd-Alrhman Masalkhi
Use
md_cloned_bio()to control bio accounting instead of relying onr1bio_existedin raid1 or theio_accountingflag in raid10.The previous logic does not reliably reflect whether a bio is an md cloned bio. When a failed bio is split and resubmitted via
bio_submit_split_bioset()on the error path, this can lead to either double accounting for md cloned bios, or missing accounting for bios returned frombio_submit_split_bioset()
Quoted 5 lines
Fix this by using
md_cloned_bio()to detect md cloned bios and skip accounting accordingly.Fixes bb2a9acefaf9 ("md/raid1: switch to use md_account_bio() for io accounting")Fixes 820455238366 ("md/raid10: switch to use md_account_bio() for io accounting")
The same as patch 2, I don't think this patch fix any problem. And I think current IO accounting is correct.
Quoted 64 lines
Signed-off-by Abd-Alrhman Masalkhi <[email protected]>---
This patch depends on patch 1.Changes in v2:
- New patch.--- drivers/md/raid1.c | 2 +- drivers/md/raid10.c | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-)drivers/md/raid1.c +1
-1@@ -1396,7 +1396,7 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio, } r1_bio->read_disk = rdisk; -if (!r1bio_existed) { +if (likely (!md_cloned_bio (mddev, bio))) {md_account_bio (mddev, &bio); r1_bio->master_bio = bio; }drivers/md/raid10.c +4
-4@@ -1146,7 +1146,7 @@ static bool regular_request_wait(struct mddev *mddev, struct r10conf *conf, }static void raid10_read_request (struct mddev *mddev,struct bio *bio, -struct r10bio *r10_bio,bool io_accounting) +struct r10bio *r10_bio) {struct r10conf *conf = mddev->private;struct bio *read_bio;@@ -1226,7 +1226,7 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, } slot = r10_bio->read_slot; -if (io_accounting) { +if (likely (!md_cloned_bio (mddev, bio))) {md_account_bio (mddev, &bio); r10_bio->master_bio = bio; }@@ -1552,7 +1552,7 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors) conf->geo.raid_disks);if (bio_data_dir (bio) ==READ ) -raid10_read_request (mddev, bio, r10_bio, true); +raid10_read_request (mddev, bio, r10_bio);else raid10_write_request (mddev, bio, r10_bio); }@@ -2872,7 +2872,7 @@ static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio)rdev_dec_pending (rdev, mddev); r10_bio->state =0 ; -raid10_read_request (mddev, r10_bio->master_bio, r10_bio, false); +raid10_read_request (mddev, r10_bio->master_bio, r10_bio);/* * allow_barrier after re-submit to ensure no sync io * can be issued while regular io pending.
Signature · 2 lines
Thansk, Kuai
Yu Kuai
AppliedQuoted 32 lines · Abd-Alrhman Masalkhi
Hi,
This series revisits an issue in the read error recovery path for raid1 and raid10 when bios are split and resubmitted.
In v1, the approach was to avoid splitting bios in the raid1 thread to prevent recursion and potential deadlocks. However, this was not ideal and too restrictive.
As suggested by Yu Kuai, this series instead handles the problem in
md_handle_request()and allows bio splitting.Link to Yu Kuai' email: https://lore.kernel.org/linux-raid/[email protected]/T/#m714020a38b60fc5f84b9a24f0c46acbe5d7342d6
This series fixes the above by:
- handling md cloned bios explicitly inmd_handle_request()
- usingmd_cloned_bio()to reliably detect the error path
- fixing bio accounting to avoid double/missing accountingChanges in v2:
- Split fixes into separate patches for clarity.
- Usemd_cloned_bio()consistently to detect cloned bios.
- Recognize that raid10 has the same issue and fix it in this series
- Allow splitting bios.
- Handle md cloned bios explicitly inmd_handle_request()
- Link v1: https://lore.kernel.org/linux-raid/[email protected]/Abd-Alrhman Masalkhi (3):
md/raid1,raid10: fix deadlock in read error recovery path md/raid1,raid10: fix error-path detection withmd_cloned_bio()md/raid1,raid10: fix bio accounting for split md cloned bios
Quoted 5 lines
drivers/md/md.c | 25 ++++++++++++++++--------- drivers/md/md.h | 5 +++++ drivers/md/raid1.c | 15 +++++++++++---- drivers/md/raid10.c | 28 ++++++++++++++++++---------- 4 files changed, 50 insertions(+), 23 deletions(-)
Applied to md-7.2
Quoted 1 line
Signature · 2 lines
Thansk, Kuai