md/cluster: block reshape requests with resync job initiated from remote node
PATCH v2 1/1 md/cluster: block reshape requests with resync job initiated from remote node
In cluster env, a node can start resync job when the resync cmd was executed on a different node. Reshape requests should be blocked for resync job initiated by any node. Current code only checks local resync condition to block reshape requests. This results in a chaos result when resync was doing on different node.
Fix this by adding the extra check MD_RESYNCING_REMOTE.
*** Repro steps ***
Test env
node A & B share 3 iSCSI luns: sdg/sdh/sdi. Each lun size is 1GB. The disk size is more large the issue is more likely to trigger. (more resync time, more easily trigger issue)
Test script
```
ssh root@node2 "mdadm -S --scan"
mdadm -S --scan
mdadm --zero-superblock /dev/sd{g,h,i}
for i in {g,h,i};do dd if=/dev/zero of=/dev/sd$i oflag=direct bs=1M \
count=20; done
echo "mdadm create array"
mdadm -C /dev/md0 -b clustered -e 1.2 -n 2 -l mirror /dev/sdg /dev/sdh
echo "set up array on node2"
ssh root@node2 "mdadm -A /dev/md0 /dev/sdg /dev/sdh"
sleep 5
mdadm --manage --add /dev/md0 /dev/sdi
mdadm --wait /dev/md0
mdadm --grow --raid-devices=3 /dev/md0
mdadm /dev/md0 --fail /dev/sdg
mdadm /dev/md0 --remove /dev/sdg
#mdadm --wait /dev/md0
mdadm --grow --raid-devices=2 /dev/md0
```There is a workaround:
when adding the --wait before second --grow, the issue will disappear.
Test results
Every cmd line in test script can be finish, but array status is wrong. There are 3 results. (part of output by: mdadm -D /dev/md0)
<case 1> : normal test result.
```
Number Major Minor RaidDevice State
1 8 112 0 active sync /dev/sdh
2 8 128 1 active sync /dev/sdi
```<case 2> : "--faulty" data still exist on disk metadata area.
```
Number Major Minor RaidDevice State
- 0 0 0 removed
1 8 112 1 active sync /dev/sdh
2 8 128 2 active sync /dev/sdi
0 8 96 - faulty /dev/sdg
```<case 3> : "--remove" data still exist on disk metadata area.
```
Number Major Minor RaidDevice State
- 0 0 0 removed
1 8 112 1 active sync /dev/sdh
2 8 128 2 active sync /dev/sdi
```*** Analysis ***
code flow of the problem scenario
```
node A node B
------------------------------ ---------------
mdadm --grow -n 3 md0
+ raid1_reshape
mddev->raid_disks: 2=>3
start resync job, it will block A resync job
mddev->raid_disks: 2=>3
mdadm md0 --fail sdg
+ update disk: array sb & bitmap sb
+ send METADATA_UPDATE
(resync job blocking)
(B continue doing "--grow -n 3" resync job)
recv METADATA_UPDATE
+ read disk metadata
+ raid1_error
+ set MD_RECOVERY_INTR to break resync
... ...
md_check_recovery
+ remove_and_add_spares return 1
+ set MD_RECOVERY_RECOVER, later restart resync
mdadm md0 --remove sdg
+ md_cluster_ops->remove_disk
| + send REMOVE
+ md_kick_rdev_from_array
+ update disk: array sb & bitmap sb
(resync job blocking)
(B continue doing "--grow -n 3" resync job)
recv REMOVE
+ process_remove_disk doesn't set mddev->sb_flags,
so it doesn't update disk sb & bitmap sb.
......
md_check_recovery
+ md_kick_rdev_from_array
mdadm --grow -n 2 md0
+ raid1_reshape
| mddev->raid_disks: 3=>2
+ send METADATA_UPDATED
(B continue doing "--grow -n 3" resync job)
recv METADATA_UPDATE
+ check_sb_changes
update_raid_disks return -EBUSY
update failed for mddev->raid_disks: 3=>2
(B never successfully update mddev->raid_disks: 3=>2)
when B finish "--grow -n 3" resync job
+ use mddev->raid_disks:3 to update array sb & bitmap sb
+ send METADATA_UPDATED
recv METADATA_UPDATED
+ read wrong raid_disks to update
kernel data.
```---
v2:
- for clearly, split patch-v1 into two single patch to review.
- revise patch subject & comments.
- add test result in comments.
v1:
- add cover-letter
- add more descriptions in commit log
v0:
- create 2 patches, patch 1/2 is this patch.
--- drivers/md/md.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/md/md.c +6 -2
@@ -7278,6 +7278,7 @@ static int update_raid_disks(struct mddev *mddev, int raid_disks)return -EINVAL ;if (mddev->sync_thread ||test_bit (MD_RECOVERY_RUNNING , &mddev->recovery) || +test_bit (MD_RESYNCING_REMOTE , &mddev->recovery) || mddev->reshape_position != MaxSector)return -EBUSY ;
@@ -9662,8 +9663,11 @@ static void check_sb_changes(struct mddev *mddev, struct md_rdev *rdev) } } -if (mddev->raid_disks !=le32_to_cpu (sb->raid_disks)) -update_raid_disks (mddev,le32_to_cpu (sb->raid_disks)); +if (mddev->raid_disks !=le32_to_cpu (sb->raid_disks)) { + ret =update_raid_disks (mddev,le32_to_cpu (sb->raid_disks)); +if (ret) +pr_warn ("md: updating array disks failed. %d\n" , ret); + }/* * Since mddev->delta_disks has already updated in update_raid_disks,
Signature · 1 line
2.27.0
Hello Song & Guoqing,
Thank you for your kindly review.
I feel it's unclear and mess up to put patch 1 & 2 together. So I split them and first send patch 1 for review.
I am not sure whether I should add "reviewd-by Song & Guoqing" in comments. Because the code is correct, comments need to revise.
btw,
For patch 2, I need some time to answer Guoqing's question. please wait.
Thanks.
Quoted 28 lines · Zhao Heming
In cluster env, a node can start resync job when the resync cmd was executed on a different node. Reshape requests should be blocked for resync job initiated by any node. Current code only checks local resync condition to block reshape requests. This results in a chaos result when resync was doing on different node.
Fix this by adding the extra check
MD_RESYNCING_REMOTE.... ...
Signed-off-by Zhao Heming <[email protected]>---
v2:
- for clearly, split patch-v1 into two single patch to review.
- revise patch subject & comments.
- add test result in comments.v1:
- add cover-letter
- add more descriptions in commit logv0:
- create 2 patches, patch 1/2 is this patch.--- drivers/md/md.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)