reshape fixlets for 2.6.37
cover reshape fixlets for 2.6.37
Hi Neil,
Here are two fixes that we discovered while implementing the external reshape code. The second one is tagged for -stable as current versions of mdadm may not assemble a native array that is being expanded by more than 1 disk. Incremental assembly, however, does work in this case.
---
Dan Williams (2):
md/raid5: skip wait for MD_CHANGE_DEVS acknowledgement in the external case md/raid5: initialize ->recovery_offset when growing raid_disks
drivers/md/raid5.c | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-)
PATCH 1/2 md/raid5: skip wait for MD_CHANGE_DEVS acknowledgement in the external case
mdmon is orchestrating the reshape progress and we rely on it to manage the reshape position and metadata updates.
--- drivers/md/raid5.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
drivers/md/raid5.c +3 -3
@@ -4243,7 +4243,7 @@ static sector_t reshape_request(mddev_t *mddev, sector_t sector_nr, int *skippedset_bit (MD_CHANGE_DEVS , &mddev->flags);md_wakeup_thread (mddev->thread);wait_event (mddev->sb_wait, mddev->flags ==0 || -kthread_should_stop ()); + !mddev->persistent ||kthread_should_stop ());spin_lock_irq (&conf->device_lock); conf->reshape_safe = mddev->reshape_position;spin_unlock_irq (&conf->device_lock);
@@ -4344,8 +4344,8 @@ static sector_t reshape_request(mddev_t *mddev, sector_t sector_nr, int *skippedset_bit (MD_CHANGE_DEVS , &mddev->flags);md_wakeup_thread (mddev->thread);wait_event (mddev->sb_wait, - !test_bit (MD_CHANGE_DEVS , &mddev->flags) - ||kthread_should_stop ()); + !test_bit (MD_CHANGE_DEVS , &mddev->flags) || + !mddev->persistent ||kthread_should_stop ());spin_lock_irq (&conf->device_lock); conf->reshape_safe = mddev->reshape_position;spin_unlock_irq (&conf->device_lock);
Re: [PATCH 1/2] md/raid5: skip wait for MD_CHANGE_DEVS acknowledgement in the external case
Quoted 23 lines · On Fri, 22 Oct 2010 11:03:54 -0700 Dan Williams <[email protected]>…
mdmon is orchestrating the reshape progress and we rely on it to manage the reshape position and metadata updates.
Reported-by Adam Kwolek <[email protected]>Signed-off-by Dan Williams <[email protected]>--- drivers/md/raid5.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)drivers/md/raid5.c +1
-1@@ -4243,7 +4243,7 @@ static sector_t reshape_request(mddev_t *mddev, sector_t sector_nr, int *skippedset_bit (MD_CHANGE_DEVS , &mddev->flags);md_wakeup_thread (mddev->thread);wait_event (mddev->sb_wait, mddev->flags ==0 || -kthread_should_stop ()); + !mddev->persistent ||kthread_should_stop ());spin_lock_irq (&conf->device_lock); conf->reshape_safe = mddev->reshape_position;spin_unlock_irq (&conf->device_lock);
drivers/md/raid5.c static sector_t reshape_request(mddev_t *mddev, sector_t sector_nr, int *skipped · line 4351
Quoted 10 lines
drivers/md/raid5.c +2
-2@@ -4344,8 +4344,8 @@ static sector_t reshape_request(mddev_t *mddev, sector_t sector_nr, int *skippedset_bit (MD_CHANGE_DEVS , &mddev->flags);md_wakeup_thread (mddev->thread);wait_event (mddev->sb_wait, - !test_bit (MD_CHANGE_DEVS , &mddev->flags) - ||kthread_should_stop ()); + !test_bit (MD_CHANGE_DEVS , &mddev->flags) || + !mddev->persistent ||kthread_should_stop ());spin_lock_irq (&conf->device_lock); conf->reshape_safe = mddev->reshape_position;spin_unlock_irq (&conf->device_lock);
Do we really need this?
If we set MD_CHANGE_DEVS as wake up the thread, the thread will call md_update_sb which, in the !persistent case, will clear the bit and wakeup sb_wait. So we should block on those wait_events anyway...
Am I missing something?
And as ->persistent doesn't change, I would rather have that test outside the wait_event() to make the meaning more explicit. So if I have missed something, I'll make that change.
Thanks,
NeilBrown
Re: [PATCH 1/2] md/raid5: skip wait for MD_CHANGE_DEVS acknowledgement in the external case
Quoted 31 lines · Neil Brown
Quoted 28 lines · On Fri, 22 Oct 2010 11:03:54 -0700 Dan Williams<[email protected]>…
mdmon is orchestrating the reshape progress and we rely on it to manage the reshape position and metadata updates.
Reported-by Adam Kwolek<[email protected]>Signed-off-by Dan Williams<[email protected]>--- drivers/md/raid5.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)drivers/md/raid5.c +1
-3@@ -4243,7 +4243,7 @@ static sector_t reshape_request(mddev_t *mddev, sector_t sector_nr, int *skippedset_bit (MD_CHANGE_DEVS ,&mddev->flags);md_wakeup_thread (mddev->thread);wait_event (mddev->sb_wait, mddev->flags ==0 || -kthread_should_stop ()); + !mddev->persistent ||kthread_should_stop ());spin_lock_irq (&conf->device_lock); conf->reshape_safe = mddev->reshape_position;spin_unlock_irq (&conf->device_lock);@@ -4344,8 +4344,8 @@ static sector_t reshape_request(mddev_t *mddev, sector_t sector_nr, int *skippedset_bit (MD_CHANGE_DEVS ,&mddev->flags);md_wakeup_thread (mddev->thread); wait_event(mddev->sb_wait, - !test_bit (MD_CHANGE_DEVS ,&mddev->flags) - ||kthread_should_stop ());
Quoted 8 lines
drivers/md/raid5.c static sector_t reshape_request(mddev_t *mddev, sector_t sector_nr, int *skipped · line 4249
Quoted 5 lines
drivers/md/raid5.c +2
-0+ !test_bit (MD_CHANGE_DEVS ,&mddev->flags) || + !mddev->persistent || kthread_should_stop());spin_lock_irq (&conf->device_lock); conf->reshape_safe = mddev->reshape_position;spin_unlock_irq (&conf->device_lock);Do we really need this?
Sorry, not anymore, this patch predated the md_update_sb() rework and is no longer needed as far as I can see.
--
Dan
PATCH 2/2 md/raid5: initialize ->recovery_offset when growing raid_disks
In reply to Dan WilliamsWe mark the disk in-sync, and also need to init ->recovery_offset lest we confuse older versions of mdadm that don't consider this case at assembly (i.e. that when growing we assume the disk is insync).
mdadm: /dev/md0 assembled from 3 drives and 1 rebuilding - not enough to start the array.
--- drivers/md/raid5.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
drivers/md/raid5.c +1 -0
@@ -5580,6 +5580,7 @@ static int raid5_start_reshape(mddev_t *mddev)if (raid5_add_disk (mddev, rdev) ==0 ) {char nm[20 ];if (rdev->raid_disk >= conf->previous_raid_disks) { + rdev->recovery_offset = MaxSector;set_bit (In_sync, &rdev->flags); added_devices++; }else
Re: [PATCH 2/2] md/raid5: initialize ->recovery_offset when growing raid_disks
Quoted 17 lines · On Fri, 22 Oct 2010 11:03:59 -0700 Dan Williams <[email protected]>…
We mark the disk in-sync, and also need to init ->recovery_offset lest we confuse older versions of mdadm that don't consider this case at assembly (i.e. that when growing we assume the disk is insync).
mdadm:
/dev/md0assembled from 3 drives and 1 rebuilding - not enough to start the array.Cc <[email protected]>Signed-off-by Dan Williams <[email protected]>--- drivers/md/raid5.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)drivers/md/raid5.c +0
-0
drivers/md/raid5.c static int raid5_start_reshape(mddev_t *mddev) · line 5586
Quoted 7 lines
drivers/md/raid5.c +1
-0@@ -5580,6 +5580,7 @@ static int raid5_start_reshape(mddev_t *mddev)if (raid5_add_disk (mddev, rdev) ==0 ) {char nm[20 ];if (rdev->raid_disk >= conf->previous_raid_disks) { + rdev->recovery_offset = MaxSector;set_bit (In_sync, &rdev->flags); added_devices++; }else
Sorry, but I'm not getting this one....
rdev->recovery_offset is only ever used when In_sync is clear. So it makes no sense to give it a value when In_sync is set.
Maybe there are some places where we clear In_sync that need to have recovery_offset set to zero, but it isn't obvious to me that that would explain your symptom.
Can you give a bit more detail of the problem you are seeing please?
Thanks,
NeilBrown
Re: [PATCH 2/2] md/raid5: initialize ->recovery_offset when growing raid_disks
Quoted 31 lines · Neil Brown
Quoted 17 lines · On Fri, 22 Oct 2010 11:03:59 -0700 Dan Williams<[email protected]>…
We mark the disk in-sync, and also need to init ->recovery_offset lest we confuse older versions of mdadm that don't consider this case at assembly (i.e. that when growing we assume the disk is insync).
mdadm:
/dev/md0assembled from 3 drives and 1 rebuilding - not enough to start the array.Cc <[email protected]>Signed-off-by Dan Williams<[email protected]>--- drivers/md/raid5.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)drivers/md/raid5.c +0
-0drivers/md/raid5.c static int raid5_start_reshape(mddev_t *mddev) · line 5586
Quoted 7 lines
drivers/md/raid5.c +1
-0@@ -5580,6 +5580,7 @@ static int raid5_start_reshape(mddev_t *mddev)if (raid5_add_disk (mddev, rdev) ==0 ) {char nm[20 ];if (rdev->raid_disk>= conf->previous_raid_disks) { + rdev->recovery_offset = MaxSector;set_bit (In_sync,&rdev->flags); added_devices++; }else Sorry, but I'm not getting this one....
Quoted 8 lines
rdev->recovery_offsetis only ever used whenIn_syncis clear. So it makes no sense to give it a value whenIn_syncis set.Maybe there are some places where we clear
In_syncthat need to haverecovery_offsetset to zero, but it isn't obvious to me that that would explain your symptom.Can you give a bit more detail of the problem you are seeing please?
mdadm -A an array growing raid disks by more than max_degraded.
Here is the related mdadm fix, but I figured it was worhtwhile to also address this in the kernel for the corner case of new kernel + old mdadm.
Quoted 20 lines
commit
156a33719d956fe90cbb1625b13beb52a0d6fd87Author: Dan Williams <[email protected]> Date: Mon Jul 12 12:03:13 2010 -0700Assemble: fix assembly in the
delta_disks>max_degradedcaseIncremental assembly works on such an array because the kernel sees the disk as in-sync and that the array is reshaping. Teach
Assemble()the same assumptions.This is only needed on kernels that do not initialize ->recovery_offset when activating spares for reshape.
Signed-off-by Dan Williams <[email protected]>Assemble.c +0
-0
Assemble.c int Assemble(struct supertype *st, char *mddev, · line 812
Quoted 10 lines
Assemble.c +3
-1@@ -804,7 +804,9 @@ int Assemble(struct supertype *st, char *mddev, devices[most_recent].i.events) { devices[j].uptodate =1 ;if (i < content->array.raid_disks) { -if (devices[j].i.recovery_start == MaxSector) { +if (devices[j].i.recovery_start == MaxSector || + (content->reshape_active && + j >= content->array.raid_disks - content->delta_disks)) { okcnt++; avail[i]=1 ; }else
--
Dan
Re: [PATCH 2/2] md/raid5: initialize ->recovery_offset when growing raid_disks
Quoted 71 lines · On Tue, 26 Oct 2010 00:26:33 -0700 Dan Williams <[email protected]>…
Quoted 31 lines · Neil Brown
Quoted 17 lines · On Fri, 22 Oct 2010 11:03:59 -0700 Dan Williams<[email protected]>…
We mark the disk in-sync, and also need to init ->recovery_offset lest we confuse older versions of mdadm that don't consider this case at assembly (i.e. that when growing we assume the disk is insync).
mdadm:
/dev/md0assembled from 3 drives and 1 rebuilding - not enough to start the array.Cc <[email protected]>Signed-off-by Dan Williams<[email protected]>--- drivers/md/raid5.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)drivers/md/raid5.c +0
-0drivers/md/raid5.c static int raid5_start_reshape(mddev_t *mddev) · line 5586
Quoted 7 lines
drivers/md/raid5.c +1
-0@@ -5580,6 +5580,7 @@ static int raid5_start_reshape(mddev_t *mddev)if (raid5_add_disk (mddev, rdev) ==0 ) {char nm[20 ];if (rdev->raid_disk>= conf->previous_raid_disks) { + rdev->recovery_offset = MaxSector;set_bit (In_sync,&rdev->flags); added_devices++; }else Sorry, but I'm not getting this one....
Quoted 8 lines
rdev->recovery_offsetis only ever used whenIn_syncis clear. So it makes no sense to give it a value whenIn_syncis set.Maybe there are some places where we clear
In_syncthat need to haverecovery_offsetset to zero, but it isn't obvious to me that that would explain your symptom.Can you give a bit more detail of the problem you are seeing please?
mdadm -A an array growing raid disks by more than
max_degraded.Here is the related mdadm fix, but I figured it was worhtwhile to also address this in the kernel for the corner case of new kernel + old mdadm.
Quoted 25 lines
commit
156a33719d956fe90cbb1625b13beb52a0d6fd87Author: Dan Williams <[email protected]> Date: Mon Jul 12 12:03:13 2010 -0700Assemble: fix assembly in the
delta_disks>max_degradedcaseIncremental assembly works on such an array because the kernel sees the disk as in-sync and that the array is reshaping. Teach
Assemble()the same assumptions.This is only needed on kernels that do not initialize ->recovery_offset when activating spares for reshape.
Signed-off-by Dan Williams <[email protected]>Assemble.c +1
-1@@ -804,7 +804,9 @@ int Assemble(struct supertype *st, char *mddev, devices[most_recent].i.events) { devices[j].uptodate =1 ;if (i < content->array.raid_disks) { -if (devices[j].i.recovery_start == MaxSector) { + if (devices[j].i.recovery_start == MaxSector ||
Quoted 8 lines
Assemble.c int Assemble(struct supertype *st, char *mddev, · line 812
Quoted 5 lines
Assemble.c +2
-0+ (content->reshape_active && + j >= content->array.raid_disks - content->delta_disks)) { okcnt++; avail[i]=1 ; } else--
Dan
I think the issue here is probably that the metadata handler isn't setting recovery_start "properly" for devices that are in-sync just as far as the the current reshape has progressed.
I really don't think the kernel has anything to do with it as it doesn't report any recovery_start value at all when the device is 'in_sync'.
So I think this patch is wrong and you need to fix the getinfo_super method to set recovery_start to MaxSector when the device is a new device in an array being reshaped.
Could that make sense?
NeilBrown
Re: [PATCH 2/2] md/raid5: initialize ->recovery_offset when growing raid_disks
Quoted 84 lines · Neil Brown
Quoted 71 lines · On Tue, 26 Oct 2010 00:26:33 -0700 Dan Williams<[email protected]>…
Quoted 31 lines · Neil Brown
Quoted 17 lines · On Fri, 22 Oct 2010 11:03:59 -0700 Dan Williams<[email protected]>…
We mark the disk in-sync, and also need to init ->recovery_offset lest we confuse older versions of mdadm that don't consider this case at assembly (i.e. that when growing we assume the disk is insync).
mdadm:
/dev/md0assembled from 3 drives and 1 rebuilding - not enough to start the array.Cc <[email protected]>Signed-off-by Dan Williams<[email protected]>--- drivers/md/raid5.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)drivers/md/raid5.c +0
-0drivers/md/raid5.c static int raid5_start_reshape(mddev_t *mddev) · line 5586
Quoted 7 lines
drivers/md/raid5.c +1
-0@@ -5580,6 +5580,7 @@ static int raid5_start_reshape(mddev_t *mddev)if (raid5_add_disk (mddev, rdev) ==0 ) {char nm[20 ];if (rdev->raid_disk>= conf->previous_raid_disks) { + rdev->recovery_offset = MaxSector;set_bit (In_sync,&rdev->flags); added_devices++; }else Sorry, but I'm not getting this one....
Quoted 8 lines
rdev->recovery_offsetis only ever used whenIn_syncis clear. So it makes no sense to give it a value whenIn_syncis set.Maybe there are some places where we clear
In_syncthat need to haverecovery_offsetset to zero, but it isn't obvious to me that that would explain your symptom.Can you give a bit more detail of the problem you are seeing please?
mdadm -A an array growing raid disks by more than
max_degraded.Here is the related mdadm fix, but I figured it was worhtwhile to also address this in the kernel for the corner case of new kernel + old mdadm.
Quoted 25 lines
commit
156a33719d956fe90cbb1625b13beb52a0d6fd87Author: Dan Williams<[email protected]> Date: Mon Jul 12 12:03:13 2010 -0700Assemble: fix assembly in the
delta_disks>max_degradedcaseIncremental assembly works on such an array because the kernel sees the disk as in-sync and that the array is reshaping. Teach
Assemble()the same assumptions.This is only needed on kernels that do not initialize ->recovery_offset when activating spares for reshape.
Signed-off-by Dan Williams<[email protected]>Assemble.c +1
-1@@ -804,7 +804,9 @@ int Assemble(struct supertype *st, char *mddev, devices[most_recent].i.events) { devices[j].uptodate =1 ;if (i< content->array.raid_disks) { -if (devices[j].i.recovery_start == MaxSector) { + if (devices[j].i.recovery_start == MaxSector ||Quoted 8 lines
Assemble.c int Assemble(struct supertype *st, char *mddev, · line 812
Quoted 5 lines
Assemble.c +2
-0+ (content->reshape_active&& + j>= content->array.raid_disks - content->delta_disks)) { okcnt++; avail[i]=1 ; } else--
Dan
Quoted 6 lines
I think the issue here is probably that the metadata handler isn't setting
recovery_start"properly" for devices that are in-sync just as far as the the current reshape has progressed.I really don't think the kernel has anything to do with it as it doesn't report any
recovery_startvalue at all when the device is 'in_sync'.
This was a few months back, but it looks like I looked at the mdadm fix and thought I could derive a kernel fix, but yes, the real problem is in the mdadm metadata handler.
Quoted 1 line
So I think this patch is wrong
The kernel patch, not the mdadm patch.
Quoted 5 lines
and you need to fix the
getinfo_supermethod
to setrecovery_startto MaxSector when the device is a new device in an array being reshaped.Could that make sense?
Yes, I can push this fix down to super1.c to avoid:
# mdadm -A /dev/md1 /dev/loop[0-3] mdadm: /dev/md1 assembled from 3 drives and 1 rebuilding - not enough to start the array.
--
Dan