RAID5 losing initial synchronization on restart when one disk is spare

DISCUSSION Hubert Verstraete 4 participants 14 messages Jun 4, 2008 – Jun 12, 2008 MD RAID

Share & tools

Hubert Verstraete

author

2008-06-04 10:13 UTC #0 lore raw

Hello

According to mdadm's man page:
"When creating a RAID5 array, mdadm will automatically create a degraded array with an extra spare drive. This is because building the spare into a degraded array is in general faster than resyncing the parity on a non-degraded, but not clean, array. This feature can be over-ridden with the --force option."

Unfortunately, I'm seeing a kind of bug when I create a RAID5 array with an internal bitmap, then stop the array before the initial synchronization is done and restart the array.

1° When I create the array with an internal bitmap:
mdadm -C /dev/md_d1 -e 1.2 -l 5 -n 4 -b internal -R /dev/sd?
I see the last disk as a spare disk. After the restart of the array, all disks are seen active and the array is not continuing the aborted synchronization! Note that I did not use the --assume-clean option.

2° When I create the array without a bitmap:
mdadm -C /dev/md_d1 -e 1.2 -l 5 -n 4 -R /dev/sd?
I see the last disk as a spare disk. After the restart of the array, the spare disk is still a spare disk and the array continues the synchronization where it had stopped.

In the case 1°, is this a bug or did I miss something?
Secondly, what could be the consequences of this non-performed synchronization ?

Kernel version: 2.6.26-rc4
mdadm version: 2.6.2

Thanks,
Hubert
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hubert Verstraete

author

2008-06-10 11:57 UTC #1 lore raw

Quoted 20 lines · Hubert Verstraete

Hello
According to mdadm's man page:
"When creating a RAID5 array, mdadm will automatically create a degraded array with an extra spare drive. This is because building the spare into a degraded array is in general faster than resyncing the parity on a non-degraded, but not clean, array. This feature can be over-ridden with the --force option."
Unfortunately, I'm seeing a kind of bug when I create a RAID5 array with an internal bitmap, then stop the array before the initial synchronization is done and restart the array.
1° When I create the array with an internal bitmap:
mdadm -C /dev/md_d1 -e 1.2 -l 5 -n 4 -b internal -R /dev/sd?
I see the last disk as a spare disk. After the restart of the array, all disks are seen active and the array is not continuing the aborted synchronization! Note that I did not use the --assume-clean option.
2° When I create the array without a bitmap:
mdadm -C /dev/md_d1 -e 1.2 -l 5 -n 4 -R /dev/sd?
I see the last disk as a spare disk. After the restart of the array, the spare disk is still a spare disk and the array continues the synchronization where it had stopped.

Quoted 8 lines

In the case 1°, is this a bug or did I miss something?
Secondly, what could be the consequences of this non-performed synchronization ?
Kernel version: 2.6.26-rc4
mdadm version: 2.6.2
Thanks,
Hubert

For the record, the new stable kernel 2.6.25.6 has the same issue.
I thought maybe the patch "md: fix prexor vs sync_request race" could have fixed this, unfortunately not.

Regards,
Hubert
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Dan Williams

2008-06-10 22:56 UTC #2 lore raw

Quoted 35 lines · Hubert Verstraete

Quoted 34 lines · Hubert Verstraete
Hello
According to mdadm's man page:
"When creating a RAID5 array, mdadm will automatically create a degraded array with an extra spare drive. This is because building the spare into a degraded array is in general faster than resyncing the parity on a non-degraded, but not clean, array. This feature can be over-ridden with the --force option."
Unfortunately, I'm seeing a kind of bug when I create a RAID5 array with an internal bitmap, then stop the array before the initial synchronization is done and restart the array.
1° When I create the array with an internal bitmap:
mdadm -C /dev/md_d1 -e 1.2 -l 5 -n 4 -b internal -R /dev/sd? I see the last disk as a spare disk. After the restart of the array, all disks are seen active and the array is not continuing the aborted synchronization!
Note that I did not use the --assume-clean option.
2° When I create the array without a bitmap:
mdadm -C /dev/md_d1 -e 1.2 -l 5 -n 4 -R /dev/sd?
I see the last disk as a spare disk. After the restart of the array, the spare disk is still a spare disk and the array continues the synchronization where it had stopped.
In the case 1°, is this a bug or did I miss something? Secondly, what could be the consequences of this non-performed synchronization ?
Kernel version: 2.6.26-rc4
mdadm version: 2.6.2

Quoted 7 lines

Quoted 2 lines
Thanks,
Hubert
For the record, the new stable kernel 2.6.25.6 has the same issue. I thought maybe the patch "md: fix prexor vs sync_request race" could have fixed this, unfortunately not.

I am able to reproduce this here, and I notice that it does not happen with v0.90 superblocks. In the v0.90 case when the array is stopped the last disk remains marked as spare. The following hack seems to achieve the same effect for v1 arrays, but I wonder if it is correct... Neil?

drivers/md/md.c +0 -0

index e9380b5..c38425f 100644
@@ -1234,6 +1234,7 @@ static int super_1_validate(mddev_t *mddev,

mdk_rdev_t *rdev)
role = le16_to_cpu(sb->dev_roles[rdev->desc_nr]); switch(role) {
case 0xffff: /* spare */
+ set_bit(NeedRebuild, &rdev->flags);
break;
case 0xfffe: /* faulty */
set_bit(Faulty, &rdev->flags);
@@ -1321,7 +1322,8 @@ static void super_1_sync(mddev_t *mddev, mdk_rdev_t *rdev)
sb->dev_roles[i] = cpu_to_le16(0xfffe);
else if (test_bit(In_sync, &rdev2->flags))
sb->dev_roles[i] = cpu_to_le16(rdev2->raid_disk);
- else if (rdev2->raid_disk >= 0 && rdev2->recovery_offset > 0)
+ else if (rdev2->raid_disk >= 0 && rdev2->recovery_offset > 0 &&
+ !test_bit(NeedRebuild, &rdev2->flags))
sb->dev_roles[i] = cpu_to_le16(rdev2->raid_disk);
else
sb->dev_roles[i] = cpu_to_le16(0xffff);

include/linux/raid/md_k.h +4 -0

index 3dea9f5..79201d6 100644
@@ -87,6 +87,10 @@ struct mdk_rdev_s
 #define Blocked		8		/* An error occured on an externally
 					 * managed array, don't allow writes
 					 * until it is cleared */
+#define NeedRebuild	9		/* device needs to go through a rebuild
+					 * cycle before its 'role' can be saved
+					 * to disk
+					 */
 	wait_queue_head_t blocked_wait;
 
 	int desc_nr;			/* descriptor index in the superblock */

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hubert Verstraete

author

2008-06-11 09:27 UTC #3 lore raw

Quoted 42 lines · Dan Williams

Quoted 31 lines · Hubert Verstraete
Quoted 30 lines · Hubert Verstraete
Hello
According to mdadm's man page:
"When creating a RAID5 array, mdadm will automatically create a degraded array with an extra spare drive. This is because building the spare into a degraded array is in general faster than resyncing the parity on a non-degraded, but not clean, array. This feature can be over-ridden with the --force option."
Unfortunately, I'm seeing a kind of bug when I create a RAID5 array with an internal bitmap, then stop the array before the initial synchronization is done and restart the array.
1° When I create the array with an internal bitmap:
mdadm -C /dev/md_d1 -e 1.2 -l 5 -n 4 -b internal -R /dev/sd? I see the last disk as a spare disk. After the restart of the array, all disks are seen active and the array is not continuing the aborted synchronization!
Note that I did not use the --assume-clean option.
2° When I create the array without a bitmap:
mdadm -C /dev/md_d1 -e 1.2 -l 5 -n 4 -R /dev/sd?
I see the last disk as a spare disk. After the restart of the array, the spare disk is still a spare disk and the array continues the synchronization where it had stopped.
In the case 1°, is this a bug or did I miss something? Secondly, what could be the consequences of this non-performed synchronization ?
Quoted 9 lines
Quoted 5 lines
Kernel version: 2.6.26-rc4
mdadm version: 2.6.2
Thanks,
Hubert
For the record, the new stable kernel 2.6.25.6 has the same issue. I thought maybe the patch "md: fix prexor vs sync_request race" could have fixed this, unfortunately not.

Quoted 5 lines

I am able to reproduce this here, and I notice that it does not happen with v0.90 superblocks. In the v0.90 case when the array is stopped the last disk remains marked as spare. The following hack seems to achieve the same effect for v1 arrays, but I wonder if it is correct... Neil?

Thanks Dan.
I quickly tried your patch on 2.6.25.6, unfortunately it did not fix the issue.

Neil Brown

2008-06-11 23:40 UTC #4 lore raw

In reply to Dan Williams

Quoted 6 lines · On Tuesday June 10, [email protected] wrote:

I am able to reproduce this here, and I notice that it does not happen with v0.90 superblocks. In the v0.90 case when the array is stopped the last disk remains marked as spare. The following hack seems to achieve the same effect for v1 arrays, but I wonder if it is correct... Neil?

No, not correct.

The fact that a v1 array included spares in the array before recovery completes is deliberate. It allows recovery to be restarted from where it got up to if the array is shut down while recovery is happening.
If you don't mark the drive as a part of the array (though not in_sync), then there is no opportunity for this optimisation.

Thanks,
NeilBrown

Hubert Verstraete

author

2008-06-11 14:44 UTC #5 lore raw

In reply to Hubert Verstraete

Quoted 27 lines · Hubert Verstraete

Quoted 26 lines · Hubert Verstraete
Hello
According to mdadm's man page:
"When creating a RAID5 array, mdadm will automatically create a degraded array with an extra spare drive. This is because building the spare into a degraded array is in general faster than resyncing the parity on a non-degraded, but not clean, array. This feature can be over-ridden with the --force option."
Unfortunately, I'm seeing a kind of bug when I create a RAID5 array with an internal bitmap, then stop the array before the initial synchronization is done and restart the array.
1° When I create the array with an internal bitmap:
mdadm -C /dev/md_d1 -e 1.2 -l 5 -n 4 -b internal -R /dev/sd?
I see the last disk as a spare disk. After the restart of the array, all disks are seen active and the array is not continuing the aborted synchronization! Note that I did not use the --assume-clean option.
2° When I create the array without a bitmap:
mdadm -C /dev/md_d1 -e 1.2 -l 5 -n 4 -R /dev/sd?
I see the last disk as a spare disk. After the restart of the array, the spare disk is still a spare disk and the array continues the synchronization where it had stopped.
In the case 1°, is this a bug or did I miss something?
Secondly, what could be the consequences of this non-performed synchronization ?
Kernel version: 2.6.26-rc4
mdadm version: 2.6.2

Quoted 8 lines

Quoted 2 lines
Thanks,
Hubert
For the record, the new stable kernel 2.6.25.6 has the same issue.
I thought maybe the patch "md: fix prexor vs sync_request race" could have fixed this, unfortunately not.
Regards,
Hubert

By the way and FYI, with my configuration, all disks on the same controller, internal bitmap, v1 superblock, ... the initial RAID-5 synchronization duration is the same whether I'm using the option --force or not.

Hubert
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Neil Brown

2008-06-11 23:38 UTC #6 lore raw

Quoted 5 lines · On Wednesday June 11, [email protected] wrote:

By the way and FYI, with my configuration, all disks on the same controller, internal bitmap, v1 superblock, ... the initial RAID-5 synchronization duration is the same whether I'm using the option --force or not.

For this to be a valid test, you need to fill one drive up with garbage to ensure that a resync is no a no-op.

If you don't use the "--force" option, then the recovery process will read from N-1 drives and write to 1 drive, all completely sequentially so it will go at a predictable speed.

When you use "--force" it will read from N drive and check parity. When it finds an error it will re-write that parity block. So if the parity blocks happen to be all correct (as probably was the case in your experiment), it will run nice and fast. If the parity blocks happen to all be wrong (as is likely when first creating an array on drives that weren't an array before) it will be much slower.

NeilBrown

Hubert Verstraete

author

2008-06-12 13:05 UTC #7 lore raw

Quoted 11 lines · Neil Brown

Quoted 2 lines · On Wednesday June 11, [email protected] wrote:
By the way and FYI, with my configuration, all disks on the same controller, internal bitmap, v1 superblock, ... the initial RAID-5 synchronization duration is the same whether I'm using the option --force or not.
For this to be a valid test, you need to fill one drive up with garbage to ensure that a resync is no a no-op.
If you don't use the "--force" option, then the recovery process will read from N-1 drives and write to 1 drive, all completely sequentially so it will go at a predictable speed.

Quoted 6 lines

When you use "--force" it will read from N drive and check parity. When it finds an error it will re-write that parity block. So if the parity blocks happen to be all correct (as probably was the case in your experiment), it will run nice and fast. If the parity blocks happen to all be wrong (as is likely when first creating an array on drives that weren't an array before) it will be much slower.

I've just filled all the drives with /dev/zero and am currently building a new array. Is this a valid test or should I fill the drives with /dev/random ?

Hubert

Dan Williams

2008-06-12 16:59 UTC #8 lore raw

Quoted 23 lines · Hubert Verstraete

Quoted 15 lines · Neil Brown
Quoted 5 lines · On Wednesday June 11, [email protected] wrote:
By the way and FYI, with my configuration, all disks on the same controller, internal bitmap, v1 superblock, ... the initial RAID-5 synchronization duration is the same whether I'm using the option --force or not.
For this to be a valid test, you need to fill one drive up with garbage to ensure that a resync is no a no-op.
If you don't use the "--force" option, then the recovery process will read from N-1 drives and write to 1 drive, all completely sequentially so it will go at a predictable speed.
Quoted 6 lines
When you use "--force" it will read from N drive and check parity. When it finds an error it will re-write that parity block. So if the parity blocks happen to be all correct (as probably was the case in your experiment), it will run nice and fast. If the parity blocks happen to all be wrong (as is likely when first creating an array on drives that weren't an array before) it will be much slower.

Quoted 4 lines

I've just filled all the drives with /dev/zero and am currently building a new array. Is this a valid test or should I fill the drives with /dev/random ?

No, /dev/zero will not work for this test since the xor sum of zeroed blocks is zero. The check operation has the following stages: read the parity from disk, verify it is correct, and if it is not correct calculate / writeback correct parity. So, you need to ensure that the parity check fails. Using /dev/urandom should be sufficient. Do not use /dev/random as it will exhaust the entropy pool and then block.

--
Dan

Hubert Verstraete

author

2008-06-12 18:11 UTC #9 lore raw

Quoted 24 lines · Dan Williams

Quoted 19 lines · Hubert Verstraete
Quoted 12 lines · Neil Brown
Quoted 4 lines · On Wednesday June 11, [email protected] wrote:
By the way and FYI, with my configuration, all disks on the same controller, internal bitmap, v1 superblock, ... the initial RAID-5 synchronization duration is the same whether I'm using the option --force or not.
For this to be a valid test, you need to fill one drive up with garbage to ensure that a resync is no a no-op.
If you don't use the "--force" option, then the recovery process will read from N-1 drives and write to 1 drive, all completely sequentially so it will go at a predictable speed.
Quoted 6 lines
When you use "--force" it will read from N drive and check parity. When it finds an error it will re-write that parity block. So if the parity blocks happen to be all correct (as probably was the case in your experiment), it will run nice and fast. If the parity blocks happen to all be wrong (as is likely when first creating an array on drives that weren't an array before) it will be much slower.
Quoted 3 lines
I've just filled all the drives with /dev/zero and am currently building a new array. Is this a valid test or should I fill the drives with /dev/random ?

Quoted 6 lines

No, /dev/zero will not work for this test since the xor sum of zeroed blocks is zero. The check operation has the following stages: read the parity from disk, verify it is correct, and if it is not correct calculate / writeback correct parity. So, you need to ensure that the parity check fails. Using /dev/urandom should be sufficient. Do not use /dev/random as it will exhaust the entropy pool and then block.

Confirmed, I got the same resync time after zeroing the hard drives.

Nevertheless, let's say I build the arrays on brand new disks.
If I can assume these disks are full of zero, I can freely choose my initial resync method :) And it seems my new Seagate disk is made of zero (currently running a diff with /dev/zero).

Thanks for explanations.
Hubert

Neil Brown

2008-06-11 23:45 UTC #10 lore raw

In reply to Hubert Verstraete

Quoted 20 lines · On Wednesday June 4, [email protected] wrote:

Hello
According to mdadm's man page:
"When creating a RAID5 array, mdadm will automatically create a degraded array with an extra spare drive. This is because building the spare into a degraded array is in general faster than resyncing the parity on a non-degraded, but not clean, array. This feature can be over-ridden with the --force option."
Unfortunately, I'm seeing a kind of bug when I create a RAID5 array with an internal bitmap, then stop the array before the initial synchronization is done and restart the array.
1° When I create the array with an internal bitmap:
mdadm -C /dev/md_d1 -e 1.2 -l 5 -n 4 -b internal -R /dev/sd? I see the last disk as a spare disk. After the restart of the array, all disks are seen active and the array is not continuing the aborted synchronization!
Note that I did not use the --assume-clean option.

Quoted 7 lines

2° When I create the array without a bitmap:
mdadm -C /dev/md_d1 -e 1.2 -l 5 -n 4 -R /dev/sd?
I see the last disk as a spare disk. After the restart of the array, the spare disk is still a spare disk and the array continues the synchronization where it had stopped.
In the case 1°, is this a bug or did I miss something?

Thanks for the detailed report. Yes, this is a bug.

The following patch fixes it, though I'm not 100% sure this is the right fix (it may cause too much resync in some cases, which is better than not enough, but not ideal).

Quoted 2 lines

Secondly, what could be the consequences of this non-performed synchronization ?

If you lose a drive, the data might get corrupted.

When writing to the array, the new parity block will sometimes be calculated assuming that it was previously correct. If all updates to a particular parity block are of this sort, then it will still be incorrect when you lose a drive, and data recovered based on that parity block will be incorrect.

Until you lose a drive, it will have no visible effect.

NeilBrown

Signed-off-by Neil Brown <[email protected]>

drivers/md/raid5.c +3 -1

diff .prev/drivers/md/raid5.c ./drivers/md/raid5.c
@@ -4094,7 +4094,9 @@ static int run(mddev_t *mddev)
 				" disk %d\n", bdevname(rdev->bdev,b),
 				raid_disk);
 			working_disks++;
-		}
+		} else
+			/* Cannot rely on bitmap to complete recovery */
+			conf->fullsync = 1;
 	}
 
 	/*

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

David Greaves

2008-06-12 08:03 UTC #11 lore raw

Quoted 7 lines · Neil Brown

When writing to the array, the new parity block will sometimes be calculated assuming that it was previously correct. If all updates to a particular parity block are of this sort, then it will still be incorrect when you lose a drive, and data recovered based on that parity block will be incorrect.
Until you lose a drive, it will have no visible effect.

There is a slight chance that this happened to me recently - would a
echo check > /sys/block/mdX/md/sync_action
detect this?
and would
echo repair > /sys/block/mdX/md/sync_action
fix it?

David

Dan Williams

2008-06-12 17:01 UTC #12 lore raw

Quoted 7 lines · David Greaves

Quoted 6 lines · Neil Brown
When writing to the array, the new parity block will sometimes be calculated assuming that it was previously correct. If all updates to a particular parity block are of this sort, then it will still be incorrect when you lose a drive, and data recovered based on that parity block will be incorrect.

Quoted 9 lines

Quoted 1 line
Until you lose a drive, it will have no visible effect.
There is a slight chance that this happened to me recently - would a
echo check > /sys/block/mdX/md/sync_action
detect this?
and would
echo repair > /sys/block/mdX/md/sync_action
fix it?

'Yes' on both questions.

--
Dan

Hubert Verstraete

author

2008-06-12 09:12 UTC #13 lore raw

In reply to Neil Brown

Quoted 37 lines · Neil Brown

Quoted 16 lines · On Wednesday June 4, [email protected] wrote:
Hello
According to mdadm's man page:
"When creating a RAID5 array, mdadm will automatically create a degraded array with an extra spare drive. This is because building the spare into a degraded array is in general faster than resyncing the parity on a non-degraded, but not clean, array. This feature can be over-ridden with the --force option."
Unfortunately, I'm seeing a kind of bug when I create a RAID5 array with an internal bitmap, then stop the array before the initial synchronization is done and restart the array.
1° When I create the array with an internal bitmap:
mdadm -C /dev/md_d1 -e 1.2 -l 5 -n 4 -b internal -R /dev/sd?
I see the last disk as a spare disk. After the restart of the array, all disks are seen active and the array is not continuing the aborted synchronization! Note that I did not use the --assume-clean option.
Quoted 5 lines
2° When I create the array without a bitmap:
mdadm -C /dev/md_d1 -e 1.2 -l 5 -n 4 -R /dev/sd?
I see the last disk as a spare disk. After the restart of the array, the spare disk is still a spare disk and the array continues the synchronization where it had stopped.
In the case 1°, is this a bug or did I miss something?
Thanks for the detailed report. Yes, this is a bug.
The following patch fixes it, though I'm not 100% sure this is the right fix (it may cause too much resync in some cases, which is better than not enough, but not ideal).
NeilBrown
Signed-off-by Neil Brown <[email protected]>
diff .prev/drivers/md/raid5.c ./drivers/md/raid5.c

drivers/md/raid5.c static int run(mddev_t *mddev) · line 4102

Quoted 10 lines

drivers/md/raid5.c +3 -1

@@ -4094,7 +4094,9 @@ static int run(mddev_t *mddev)
 				" disk %d\n", bdevname(rdev->bdev,b),
 				raid_disk);
 			working_disks++;
-		}
+		} else
+			/* Cannot rely on bitmap to complete recovery */
+			conf->fullsync = 1;
 	}
 
 	/*

Thanks Neil, I can confirm this solves this issue. Regarding the eventual unwanted resync, I can't say.

Regards,
Hubert Verstraete
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html