Strange / inconsistent behavior with mdadm -I -R

From: Martin Wilck <[email protected]>

When an array is assembled incrementally with mdadm -I -R and the array switches to "active" mode, md starts a recovery.

If the array was clean, the "fullsync" flag will be 0. Skip the full recovery in this case, as RAID1 does (the code was actually copied from the sync_request() method of RAID1).

---
 drivers/md/raid10.c |   16 ++++++++++++++++
 1 files changed, 16 insertions(+), 0 deletions(-)

drivers/md/raid10.c +9 -0

index 64d4824..e373d88 100644
@@ -2863,6 +2863,22 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr,
 		if (init_resync(conf))
 			return 0;
 
+	/*
+	 * Allow skipping a full rebuild for incremental assembly
+	 * of a clean array, like RAID1 does.
+	 */
+	if (mddev->bitmap == NULL &&
+	    mddev->recovery_cp == MaxSector &&
+	    !test_bit(MD_RECOVERY_REQUESTED, &mddev->recovery) &&
+	    conf->fullsync == 0) {
+		*skipped = 1;

Hello Neil,

for my DDF/RAID10 work, I have been trying to figure out how mdadm -I -R is supposed to behave, and I have found strangeness I'd like to clarify, lest I make a mistake in my DDF/RAID10 code.

My test case is incremental assembly of a clean array running mdadm -I -R by hand for each array device in turn.

1) native md and containers behave differently for RAID 1

Both native and container RAID 1 are started in auto-read-only mode when the 1st disk is added. When the 2nd disk is added, the native md switches to "active" and starts a recovery which finishes immediately. Container arrays (tested: DDF), on the other hand, do not switch to "active" until a write attempt is made on the array. The problem is in the native case: after the switch to "active", no more disks can be added any more ("can only add $DISK as a spare").

IMO the container behavior makes more sense and matches the man page better than the native behavior. Do you agree? Would it be hard to fix that?

2) RAID1 skips recovery for clean arrays, RAID10 does not

Native RAID 10 behaves similar to RAID1 as described above. When the array can be started, it does so, in auto-read-mode. When the next disk is added after that, recovery starts, and the array switches to "active", and further disks can't be added the "simple way" any more. There's one important difference: in the RAID 10 case, the recovery doesn't finish immediately. Rather, md does a full recovery of the added disk although it was clean. This is wrong; I have come up with a patch for this which I will send in a follow-up email.

With -R, it only stays inactive until we have the minimum devices needed for a functioning array. Then it will switch to 'read-auto' or, if the array is known and all expected devices are present, it will switch to 'active'.

Maybe if you could provide a sequence of "mdadm" commands that produces an outcome different to what you would expect - that would reduce the chance that I get confused.

Quoted 3 lines · NeilBrown

With -R, it only stays inactive until we have the minimum devices needed for a functioning array. Then it will switch to 'read-auto' or, if the array is known and all expected devices are present, it will switch to 'active'.

That's the point - I see the array switch to active *before* all expected devices are present. That causes further additions to fail.

Quoted 3 lines

Maybe if you could provide a sequence of "mdadm" commands that produces an outcome different to what you would expect - that would reduce the chance that I get confused.

Here is the sequence of mdadm commands to illustrate what I mean. I have a RAID10 array and I add the devices using mdadm -I -R in the sequence 1,3,2,4. After adding device 3, the array will be started in auto-read-only mode, which is fine.

But then as soon as the next disk (/dev/tosh/rd2) is added, the array switches to "active" although it is neither written to, nor all disks have been added yet. Consequently, adding disk 4 fails.

I expected the array to remain "auto-read-only" until either all 4 devices are present, or it is opened for writing. This is how the container case is behaving (almost - it doesn't switch to active automatically until it's written to).

# ./mdadm -C /dev/md0 -l 1 -n 4 /dev/tosh/rd[1-4] -pn2
mdadm: array /dev/md0 started.
(wait for initial build to finish)
# mdadm -S /dev/md0
mdadm: stopped /dev/md0
# ./mdadm -v -I /dev/tosh/rd1 -R
mdadm: /dev/tosh/rd1 attached to /dev/md/0, not enough to start (1).
# ./mdadm -v -I /dev/tosh/rd3 -R
mdadm: /dev/tosh/rd3 attached to /dev/md/0, which has been started.
# cat /proc/mdstat
Personalities : [raid1] [raid10]
md0 : active (auto-read-only) raid10 dm-6[2] dm-4[0]
      2094080 blocks super 1.2 512K chunks 2 near-copies [4/2] [U_U_]
# ./mdadm -v -I /dev/tosh/rd2 -R; cat /proc/mdstat
mdadm: /dev/tosh/rd2 attached to /dev/md/0 which is already active.
Personalities : [raid1] [raid10]
md0 : active raid10 dm-5[1] dm-6[2] dm-4[0]
      2094080 blocks super 1.2 512K chunks 2 near-copies [4/2] [U_U_]
      [>....................]  recovery =  0.0% (0/1047040)
finish=1090.6min speed=0K/sec
(wait for recovery to finish)
# ./mdadm -v -I /dev/tosh/rd4 -R
mdadm: can only add /dev/tosh/rd4 to /dev/md/0 as a spare, and
force-spare is not set.
mdadm: failed to add /dev/tosh/rd4 to existing array /dev/md/0: Invalid
argument.

Thanks,
Martin

Quoted 54 lines · On Tue, 19 Mar 2013 19:29:44 +0100 Martin Wilck <[email protected]> wrote:

Quoted 3 lines · NeilBrown

With -R, it only stays inactive until we have the minimum devices needed for a functioning array. Then it will switch to 'read-auto' or, if the array is known and all expected devices are present, it will switch to 'active'.

That's the point - I see the array switch to active *before* all expected devices are present. That causes further additions to fail.

Quoted 3 lines

Maybe if you could provide a sequence of "mdadm" commands that produces an outcome different to what you would expect - that would reduce the chance that I get confused.

Here is the sequence of mdadm commands to illustrate what I mean. I have a RAID10 array and I add the devices using mdadm -I -R in the sequence 1,3,2,4. After adding device 3, the array will be started in auto-read-only mode, which is fine.

But then as soon as the next disk (/dev/tosh/rd2) is added, the array switches to "active" although it is neither written to, nor all disks have been added yet. Consequently, adding disk 4 fails.

I expected the array to remain "auto-read-only" until either all 4 devices are present, or it is opened for writing. This is how the container case is behaving (almost - it doesn't switch to active automatically until it's written to).

# ./mdadm -C /dev/md0 -l 1 -n 4 /dev/tosh/rd[1-4] -pn2
mdadm: array /dev/md0 started.
(wait for initial build to finish)
# mdadm -S /dev/md0
mdadm: stopped /dev/md0
# ./mdadm -v -I /dev/tosh/rd1 -R
mdadm: /dev/tosh/rd1 attached to /dev/md/0, not enough to start (1).
# ./mdadm -v -I /dev/tosh/rd3 -R
mdadm: /dev/tosh/rd3 attached to /dev/md/0, which has been started.
# cat /proc/mdstat
Personalities : [raid1] [raid10]
md0 : active (auto-read-only) raid10 dm-6[2] dm-4[0]
      2094080 blocks super 1.2 512K chunks 2 near-copies [4/2] [U_U_]
# ./mdadm -v -I /dev/tosh/rd2 -R; cat /proc/mdstat
mdadm: /dev/tosh/rd2 attached to /dev/md/0 which is already active.
Personalities : [raid1] [raid10]
md0 : active raid10 dm-5[1] dm-6[2] dm-4[0]
      2094080 blocks super 1.2 512K chunks 2 near-copies [4/2] [U_U_]
      [>....................]  recovery =  0.0% (0/1047040)
finish=1090.6min speed=0K/sec
(wait for recovery to finish)
# ./mdadm -v -I /dev/tosh/rd4 -R
mdadm: can only add /dev/tosh/rd4 to /dev/md/0 as a spare, and
force-spare is not set.
mdadm: failed to add /dev/tosh/rd4 to existing array /dev/md/0: Invalid
argument.

Quoted 2 lines

Thanks,
Martin

Thanks, that makes is all very clear.

The problem is that the ADD_NEW_DISK ioctl automatically converts from read-auto to active.
There are two approaches I could take to addressing this.
1/ change ADD_NEW_DISK to not cause that conversion. I think that would need
to be conditional as some times it really should be changed.
2/ change mdadm to not use ADD_NEW_DISK but instead add the disk my setting it
up via sysfs.

I'm not sure which is best and neither is completely straight forward. So for now: I'll get back to you.

I spend a while experimenting and such and I now have a patch which I am happy with and I think fixes your issue. It is included below.

If you could test and confirm, I'd appreciate it.

I'll try to get to your DDF patches soon, but it might not be until next week.

I'll try to get to your DDF patches soon, but it might not be until next week.

When an array is assembled incrementally with mdadm -I -R and the array switches to "active" mode, md starts a recovery.

If the array was clean, the "fullsync" flag will be 0. Skip the full recovery in this case, as RAID1 does (the code was actually copied from the sync_request() method of RAID1).

Signed-off-by Martin Wilck <[email protected]>

---
 drivers/md/raid10.c |   16 ++++++++++++++++
 1 files changed, 16 insertions(+), 0 deletions(-)

drivers/md/raid10.c +9 -0

index 64d4824..e373d88 100644
@@ -2863,6 +2863,22 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr,
 		if (init_resync(conf))
 			return 0;
 
+	/*
+	 * Allow skipping a full rebuild for incremental assembly
+	 * of a clean array, like RAID1 does.
+	 */
+	if (mddev->bitmap == NULL &&
+	    mddev->recovery_cp == MaxSector &&
+	    !test_bit(MD_RECOVERY_REQUESTED, &mddev->recovery) &&
+	    conf->fullsync == 0) {
+		*skipped = 1;