md: avoid modifying spares while the array is not suspended
PATCH 1/1 md: avoid modifying spares while the array is not suspended
remove_spares() and remove_and_add_spares() modify the array's rdev configuration. These operations are only safe after the array has been suspended.
Today, md_start_sync() can call md_choose_sync_action() even when the array has not been suspended. As a result, md_choose_sync_action() may remove or replace rdevs while normal I/O is still accessing them.
The race can occur as follows:
raid10d Worker Normal IO ____________ _______________________ ______________________
raid10_write_request()
wait_blocked_dev()set Blocked
set Faulty
Skip Faulty rdev rrdev->nr_pending++ .repl_bio = bio
removeable_rdev = false .
array not suspended .
lock mddev goto err_handle
lock mddev (wait)
.
update sb .
clear Blocked .
.
unlock mddev .
lock mddev (acquires)
remove_spares()
removeable_rdev = true
raid10_remove_disk()
rdev = replacement
replacement = NULL
rdev_dec_pending(NULL)
unlock mddev (NULL)->nr_pending--
In this case, rdev_dec_pending() is called with a NULL pointer, resulting in a NULL pointer dereference when attempting to decrement nr_pending.
Fix this by ensuring that remove_spares() and remove_and_add_spares() are only called after the array has been suspended, preventing concurrent rdev configuration changes while normal I/O is in progress.
--- drivers/md/md.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)
drivers/md/md.c +13 -5
@@ -10116,7 +10116,8 @@ static int remove_and_add_spares(struct mddev *mddev,return spares; } -static bool md_choose_sync_action (struct mddev *mddev,int *spares) +static bool md_choose_sync_action (struct mddev *mddev,int *spares, +bool array_suspended) {/* Check if reshape is in progress first. */ if (mddev->reshape_position != MaxSector) {
@@ -10132,7 +10133,9 @@ static bool md_choose_sync_action(struct mddev *mddev, int *spares)/* Check if resync is in progress. */ if (mddev->resync_offset < MaxSector) { -remove_spares (mddev,NULL ); +if (array_suspended) +remove_spares (mddev,NULL ); +set_bit (MD_RECOVERY_SYNC , &mddev->recovery);clear_bit (MD_RECOVERY_RECOVER , &mddev->recovery);clear_bit (MD_RECOVERY_LAZY_RECOVER , &mddev->recovery);
@@ -10144,7 +10147,11 @@ static bool md_choose_sync_action(struct mddev *mddev, int *spares)* also removed and re-added, to allow the personality to fail the * re-add. */ - *spares =remove_and_add_spares (mddev,NULL ); +if (array_suspended) + *spares =remove_and_add_spares (mddev,NULL ); +else + *spares =0 ; +if (*spares ||test_bit (MD_RECOVERY_LAZY_RECOVER , &mddev->recovery)) {clear_bit (MD_RECOVERY_SYNC , &mddev->recovery);clear_bit (MD_RECOVERY_CHECK , &mddev->recovery);
@@ -10189,11 +10196,12 @@ static void md_start_sync(struct work_struct *ws)* As we only add devices that are already in-sync, we can * activate the spares immediately. */ -remove_and_add_spares (mddev,NULL ); +if (suspend) +remove_and_add_spares (mddev,NULL ); goto not_running; } -if (!md_choose_sync_action (mddev, &spares)) +if (!md_choose_sync_action (mddev, &spares, suspend)) goto not_running;if (!mddev->pers->sync_request)
Signature · 1 line
2.43.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] The patch prevents the recovery thread from starting or resuming for existing rebuilding spares, silently abandoning recovery and leaving the array permanently degraded.
Pre-existing issues:
- [High] The patch addresses a TOCTOU race for state_store but misses other vulnerable paths like slot_store and cluster updates, which still call remove_and_add_spares() without array suspension. --
commit 16cf64f20d25bc671dc52422ec0a44672674e524
Author: Abd-Alrhman Masalkhi <[email protected]>
md: avoid modifying spares while the array is not suspended
This commit ensures that remove_spares() and remove_and_add_spares() are only called after the array has been suspended. This prevents a race condition where normal I/O could trigger a NULL pointer dereference when rdevs are removed or replaced concurrently.
Quoted 5 lines
drivers/md/md.c +0
-0
drivers/md/md.c static bool md_choose_sync_action(struct mddev *mddev, int *spares) · line 10161
Quoted 9 lines
drivers/md/md.c +5
-1@@ -10151,7 +10154,11 @@ static bool md_choose_sync_action(struct mddev *mddev, int *spares)* also removed and re-added, to allow the personality to fail the * re-add. */ - *spares =remove_and_add_spares (mddev,NULL ); +if (array_suspended) + *spares =remove_and_add_spares (mddev,NULL ); +else + *spares =0 ; +
[Severity: High]
Does this else branch prevent the recovery thread from starting or resuming for existing rebuilding spares?
In md_start_sync(), suspend is determined by checking md_spares_need_change(mddev). For an existing recovering spare receiving normal I/O (where nr_pending > 0), both rdev_removeable() and rdev_addable() can evaluate to false. This causes md_spares_need_change() to return false, leaving suspend as false.
When md_choose_sync_action() is called with array_suspended == false, this new else branch forces *spares = 0. By completely bypassing the counting of existing recovering spares, md_choose_sync_action() will return false and skip starting the sync thread.
Since nr_pending dropping to zero on a non-faulty spare doesn't trigger a new MD_RECOVERY_NEEDED event, could this silently abandon recovery and leave the array permanently degraded?
[Severity: High]
This is a pre-existing issue, but does restricting remove_and_add_spares() calls to suspended arrays in md_choose_sync_action() leave other vulnerable paths exposed?
The commit addresses a TOCTOU race by ensuring remove_and_add_spares() is only called when the array is suspended, preventing concurrent I/O from encountering a cleared replacement pointer.
However, slot_store() can still call remove_and_add_spares() without an array suspension when a user writes "none" (slot == -1):
drivers/md/md.c:slot_store() {
...
clear_bit(Blocked, &rdev->flags);
remove_and_add_spares(rdev->mddev, rdev);
if (rdev->raid_disk >= 0)
return -EBUSY;
...
}Similarly, the MD cluster component updates the array configuration in check_sb_changes() by calling remove_and_add_spares() while the array is not suspended.
Could normal I/O still increment nr_pending on a replacement drive just before remove_spares() removes the faulty drive and clears the replacement pointer in these paths, reproducing the exact TOCTOU crash this patch aims to fix?
Quoted 13 lines
drivers/md/md.c +3
-2@@ -10196,11 +10203,12 @@ static void md_start_sync(struct work_struct *ws)* As we only add devices that are already in-sync, we can * activate the spares immediately. */ -remove_and_add_spares (mddev,NULL ); +if (suspend) +remove_and_add_spares (mddev,NULL ); goto not_running; } -if (!md_choose_sync_action (mddev, &spares)) +if (!md_choose_sync_action (mddev, &spares, suspend)) goto not_running;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
yu kuai
maintainerHi,
Quoted 70 lines · Abd-Alrhman Masalkhi
remove_spares()andremove_and_add_spares()modify the array's rdev configuration. These operations are only safe after the array has been suspended.Today,
md_start_sync()can callmd_choose_sync_action()even when the array has not been suspended. As a result,md_choose_sync_action()may remove or replace rdevs while normal I/O is still accessing them.The race can occur as follows:
raid10d Worker Normal IO ____________ _______________________ ______________________raid10_write_request() wait_blocked_dev()set Blocked
set Faulty
Skip Faulty rdev rrdev->nr_pending++ .repl_bio = bio
removeable_rdev= false .
array not suspended .
lock mddev gotoerr_handle
lock mddev (wait)
.
update sb .
clear Blocked ..unlock mddev .
lock mddev (acquires)
remove_spares()
removeable_rdev= true
raid10_remove_disk()
rdev = replacement
replacement = NULL
rdev_dec_pending(NULL)
unlock mddev (NULL)->nr_pending--In this case,
rdev_dec_pending()is called with a NULL pointer, resulting in a NULL pointer dereference when attempting to decrementnr_pending.Fix this by ensuring that
remove_spares()andremove_and_add_spares()are only called after the array has been suspended, preventing concurrent rdev configuration changes while normal I/O is in progress.Fixes bc08041b32ab ("md: suspend array in md_start_sync() if array need reconfiguration")Reported-by sashiko-bot <[email protected]>Signed-off-by Abd-Alrhman Masalkhi <[email protected]>--- drivers/md/md.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)drivers/md/md.c +2
-1@@ -10116,7 +10116,8 @@ static int remove_and_add_spares(struct mddev *mddev,return spares; } -static bool md_choose_sync_action (struct mddev *mddev,int *spares) +static bool md_choose_sync_action (struct mddev *mddev,int *spares, +bool array_suspended) {/* Check if reshape is in progress first. */ if (mddev->reshape_position != MaxSector) {
drivers/md/md.c static bool md_choose_sync_action(struct mddev *mddev, int *spares) · line 10137
Quoted 6 lines
drivers/md/md.c +2
-1@@ -10132,7 +10133,9 @@ static bool md_choose_sync_action(struct mddev *mddev, int *spares)/* Check if resync is in progress. */ if (mddev->resync_offset < MaxSector) { -remove_spares (mddev,NULL ); +if (array_suspended) +remove_spares (mddev,NULL );
The problem looks real, however, I think this will cause a change that user will be awared, if there are really spares that can be removed from conf, but array is not suspended here, user will still expect rdev will be removed from conf automatically.
In md_start_sync, if suspend is false, can we check again after mddev_lock? If suspend is supposed to be true, we can release the lock and retry with suspend = true.
Quoted 32 lines
drivers/md/md.c +9
-3+set_bit (MD_RECOVERY_SYNC , &mddev->recovery);clear_bit (MD_RECOVERY_RECOVER , &mddev->recovery);clear_bit (MD_RECOVERY_LAZY_RECOVER , &mddev->recovery);@@ -10144,7 +10147,11 @@ static bool md_choose_sync_action(struct mddev *mddev, int *spares)* also removed and re-added, to allow the personality to fail the * re-add. */ - *spares =remove_and_add_spares (mddev,NULL ); +if (array_suspended) + *spares =remove_and_add_spares (mddev,NULL ); +else + *spares =0 ; +if (*spares ||test_bit (MD_RECOVERY_LAZY_RECOVER , &mddev->recovery)) {clear_bit (MD_RECOVERY_SYNC , &mddev->recovery);clear_bit (MD_RECOVERY_CHECK , &mddev->recovery);@@ -10189,11 +10196,12 @@ static void md_start_sync(struct work_struct *ws)* As we only add devices that are already in-sync, we can * activate the spares immediately. */ -remove_and_add_spares (mddev,NULL ); +if (suspend) +remove_and_add_spares (mddev,NULL ); goto not_running; } -if (!md_choose_sync_action (mddev, &spares)) +if (!md_choose_sync_action (mddev, &spares, suspend)) goto not_running;if (!mddev->pers->sync_request)
Signature · 2 lines
Thanks, Kuai
Hi Kuai,
Quoted 84 lines · yu kuai
Hi,
Quoted 70 lines · Abd-Alrhman Masalkhi
remove_spares()andremove_and_add_spares()modify the array's rdev configuration. These operations are only safe after the array has been suspended.Today,
md_start_sync()can callmd_choose_sync_action()even when the array has not been suspended. As a result,md_choose_sync_action()may remove or replace rdevs while normal I/O is still accessing them.The race can occur as follows:
raid10d Worker Normal IO ____________ _______________________ ______________________raid10_write_request() wait_blocked_dev()set Blocked
set Faulty
Skip Faulty rdev rrdev->nr_pending++ .repl_bio = bio
removeable_rdev= false .
array not suspended .
lock mddev gotoerr_handle
lock mddev (wait)
.
update sb .
clear Blocked ..unlock mddev .
lock mddev (acquires)
remove_spares()
removeable_rdev= true
raid10_remove_disk()
rdev = replacement
replacement = NULL
rdev_dec_pending(NULL)
unlock mddev (NULL)->nr_pending--In this case,
rdev_dec_pending()is called with a NULL pointer, resulting in a NULL pointer dereference when attempting to decrementnr_pending.Fix this by ensuring that
remove_spares()andremove_and_add_spares()are only called after the array has been suspended, preventing concurrent rdev configuration changes while normal I/O is in progress.Fixes bc08041b32ab ("md: suspend array in md_start_sync() if array need reconfiguration")Reported-by sashiko-bot <[email protected]>Signed-off-by Abd-Alrhman Masalkhi <[email protected]>--- drivers/md/md.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)drivers/md/md.c +2
-1@@ -10116,7 +10116,8 @@ static int remove_and_add_spares(struct mddev *mddev,return spares; } -static bool md_choose_sync_action (struct mddev *mddev,int *spares) +static bool md_choose_sync_action (struct mddev *mddev,int *spares, +bool array_suspended) {/* Check if reshape is in progress first. */ if (mddev->reshape_position != MaxSector) {drivers/md/md.c static bool md_choose_sync_action(struct mddev *mddev, int *spares) · line 10137
Quoted 6 lines
drivers/md/md.c +2
-1@@ -10132,7 +10133,9 @@ static bool md_choose_sync_action(struct mddev *mddev, int *spares)/* Check if resync is in progress. */ if (mddev->resync_offset < MaxSector) { -remove_spares (mddev,NULL ); +if (array_suspended) +remove_spares (mddev,NULL );The problem looks real, however, I think this will cause a change that user will be awared, if there are really spares that can be removed from conf, but array is not suspended here, user will still expect rdev will be removed from conf automatically.
Quoted 3 lines
In
md_start_sync, if suspend is false, can we check again aftermddev_lock? If suspend is supposed to be true, we can release the lock and retry with suspend = true.
Yes, I see, and your approach is much better. But what do you think about taking the lock first and then checking only once?
Quoted 36 lines
Quoted 32 lines
drivers/md/md.c +9
-3+set_bit (MD_RECOVERY_SYNC , &mddev->recovery);clear_bit (MD_RECOVERY_RECOVER , &mddev->recovery);clear_bit (MD_RECOVERY_LAZY_RECOVER , &mddev->recovery);@@ -10144,7 +10147,11 @@ static bool md_choose_sync_action(struct mddev *mddev, int *spares)* also removed and re-added, to allow the personality to fail the * re-add. */ - *spares =remove_and_add_spares (mddev,NULL ); +if (array_suspended) + *spares =remove_and_add_spares (mddev,NULL ); +else + *spares =0 ; +if (*spares ||test_bit (MD_RECOVERY_LAZY_RECOVER , &mddev->recovery)) {clear_bit (MD_RECOVERY_SYNC , &mddev->recovery);clear_bit (MD_RECOVERY_CHECK , &mddev->recovery);@@ -10189,11 +10196,12 @@ static void md_start_sync(struct work_struct *ws)* As we only add devices that are already in-sync, we can * activate the spares immediately. */ -remove_and_add_spares (mddev,NULL ); +if (suspend) +remove_and_add_spares (mddev,NULL ); goto not_running; } -if (!md_choose_sync_action (mddev, &spares)) +if (!md_choose_sync_action (mddev, &spares, suspend)) goto not_running;if (!mddev->pers->sync_request)Signature · 2 lines
Thanks, Kuai
Signature · 2 lines
Best Regards, Abd-Alrhman
yu kuai
maintainerHi,
Quoted 9 lines · Abd-Alrhman Masalkhi
Quoted 7 lines
The problem looks real, however, I think this will cause a change that user will be awared, if there are really spares that can be removed from conf, but array is not suspended here, user will still expect rdev will be removed from conf automatically.
In
md_start_sync, if suspend is false, can we check again aftermddev_lock? If suspend is supposed to be true, we can release the lock and retry with suspend = true.Yes, I see, and your approach is much better. But what do you think about taking the lock first and then checking only once?
I don't get what you mean. If we take the lock and then check that array should suspend, we still have to release the lock before we suspend the array.
Signature · 2 lines
Thanks, Kuai
Hi Kuai,
Quoted 7 lines · yu kuai
Hi,
Quoted 4 lines · Abd-Alrhman Masalkhi
Quoted 4 lines
The problem looks real, however, I think this will cause a change that user will be awared, if there are really spares that can be removed from conf, but array is not suspended here, user will still expect rdev will be removed from conf automatically.
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
In
md_start_sync, if suspend is false, can we check again aftermddev_lock? If suspend is supposed to be true, we can release the lock and retry with suspend = true.Yes, I see, and your approach is much better. But what do you think about taking the lock first and then checking only once?
I don't get what you mean. If we take the lock and then check that array should suspend, we still have to release the lock before we suspend the array.
Sorry, I was not clear. I meant, do we need to check twice, once before taking the lock and once after? It seems that the check before taking the lock is redundant. since the result would need to be checked again after taking the lock anyway.
Could we drop the check before taking the lock and only check whether suspension is needed once while holding it? If suspension is needed, we would release the lock, suspend the array, and then reacquire the lock.
Quoted 3 lines
Signature · 2 lines
Thanks, Kuai
Signature · 2 lines
Best Regards, Abd-Alrhman
yu kuai
maintainerHi,
Quoted 18 lines · Abd-Alrhman Masalkhi
Hi Kuai,
Quoted 7 lines · yu kuai
Hi,
Quoted 4 lines · Abd-Alrhman Masalkhi
Quoted 4 lines
The problem looks real, however, I think this will cause a change that user will be awared, if there are really spares that can be removed from conf, but array is not suspended here, user will still expect rdev will be removed from conf automatically.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
In
md_start_sync, if suspend is false, can we check again aftermddev_lock? If suspend is supposed to be true, we can release the lock and retry with suspend = true.Yes, I see, and your approach is much better. But what do you think about taking the lock first and then checking only once?
I don't get what you mean. If we take the lock and then check that array should suspend, we still have to release the lock before we suspend the array.
Quoted 8 lines
Sorry, I was not clear. I meant, do we need to check twice, once before taking the lock and once after? It seems that the check before taking the lock is redundant. since the result would need to be checked again after taking the lock anyway.
Could we drop the check before taking the lock and only check whether suspension is needed once while holding it? If suspension is needed, we would release the lock, suspend the array, and then reacquire the lock.
Thanks for the explanation, I understand now. Howerver, I still prefer to check first before holding the lock. Because the checking is much lower overhead than acquire reconfig_mutex, and the race window that rdev become spare is small, so it's unlikely we'll acquire reconfig_mtuex twice.
Quoted 4 lines
Quoted 4 lines
Signature · 3 lines
-- Thanks, Kuai
Signature · 2 lines
Thanks, Kuai
Hi Kuai,
Quoted 26 lines · yu kuai
Hi,
Quoted 23 lines · Abd-Alrhman Masalkhi
Hi Kuai,
Quoted 7 lines · yu kuai
Hi,
Quoted 4 lines · Abd-Alrhman Masalkhi
Quoted 4 lines
The problem looks real, however, I think this will cause a change that user will be awared, if there are really spares that can be removed from conf, but array is not suspended here, user will still expect rdev will be removed from conf automatically.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
In
md_start_sync, if suspend is false, can we check again aftermddev_lock? If suspend is supposed to be true, we can release the lock and retry with suspend = true.Yes, I see, and your approach is much better. But what do you think about taking the lock first and then checking only once?
I don't get what you mean. If we take the lock and then check that array should suspend, we still have to release the lock before we suspend the array.
Sorry, I was not clear. I meant, do we need to check twice, once before taking the lock and once after? It seems that the check before taking the lock is redundant. since the result would need to be checked again after taking the lock anyway.
Quoted 9 lines
Quoted 3 lines
Could we drop the check before taking the lock and only check whether suspension is needed once while holding it? If suspension is needed, we would release the lock, suspend the array, and then reacquire the lock.
Thanks for the explanation, I understand now. Howerver, I still prefer to check first before holding the lock. Because the checking is much lower overhead than acquire
reconfig_mutex, and the race window that rdev become spare is small, so it's unlikely we'll acquirereconfig_mtuextwice.
I see, thanks for the explanation. I'll send a v2 shortly.
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Signature · 3 lines
-- Thanks, KuaiSignature · 2 lines
Thanks, Kuai
Signature · 2 lines
Best Regards, Abd-Alrhman