dm raid: deadlock/corruptor fixes
Subsystems (2)
cover dm raid: deadlock/corruptor fixes
This series of raid patches critically fixes:
- a race causing hangs occuring on tiny devices
- a reshape deadlock / potential data corruption
- a superblock update problem rebuilding individual component devices
In addition, it contains (patches 4+5, less critical) removal of code duplication deciphering the synchonization action and avoids to create a bitmap when using a raid4/5/6 journal device.
Fixes pass all lvm2 raid tests.
Heinz Mauelshagen (5):
dm-raid: fix reshape race on small devices
dm-raid: fix stripe adding reshape deadlock
dm-raid: correct explicit superblock update requests
dm-raid: share decipher_sync_action
dm-raid: disable bitmap when journaled
Documentation/device-mapper/dm-raid.txt | 2 + drivers/md/dm-raid.c | 177 +++++++++--------------- 2 files changed, 71 insertions(+), 108 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/5 dm-raid: fix reshape race on small devices
This race does not occur with usual raid device sizes but with small ones (e.g. those created by the lvm2 test suite).
Race scenario:
Loading a new mapping table, the dm-raid target's constructor retrieves the volatile reshaping state from the raid superblocks.
When the new table is activated in a following resume, the actual reshape position is retrieved. The reshape driven by the previous mapping can already have finished on small and/or fast devices thus updating raid superblocks about the new raid layout.
This causes the actual array state (e.g. stripe size reshape finished) to be inconsistent with the one in the new mapping causing hangs with left behind devices.
Fix by keeping the array frozen until the reloaded table is resumed.
Whilst on this, add/fix comments.
--- Documentation/device-mapper/dm-raid.txt | 1 + drivers/md/dm-raid.c | 58 +++---------------------- 2 files changed, 8 insertions(+), 51 deletions(-)
Documentation/device-mapper/dm-raid.txt +1 -0
@@ -348,3 +348,4 @@ Version History 1.13.1 Fix deadlock caused by early md_stop_writes(). Also fix size an state races. 1.13.2 Fix raid redundancy validation and avoid keeping raid set frozen +1.13.3 Fix reshape race on small devices
drivers/md/dm-raid.c +7 -51
135 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH 2/5 dm-raid: fix stripe adding reshape deadlock
In reply to Heinz MauelshagenWhen initiating a stripe adding reshape, a deadlock between md_stop_writes() waiting for the sync thread to stop and the running sync thread waiting for inactive stripes occurs (this frequently happens on single-core but rarely
on multi-core systems).
Resolve by setting MD_RECOVERY_WAIT to request the main MD resynchronization thread worker function md_do_sync() to bail out when initiating the reshape via constructor arguments. Don't set the flag when reloading without those arguments and avoid superfluous mddev_{suspend,resume} setting up reshape.
Passes all lvm2 raid tests.
--- Documentation/device-mapper/dm-raid.txt | 1 + drivers/md/dm-raid.c | 13 ++++--------- 2 files changed, 5 insertions(+), 9 deletions(-)
Documentation/device-mapper/dm-raid.txt +1 -0
@@ -349,3 +349,4 @@ Version History state races. 1.13.2 Fix raid redundancy validation and avoid keeping raid set frozen 1.13.3 Fix reshape race on small devices +1.14.0 Fix stripe adding reshape deadlock/potential data corruption
drivers/md/dm-raid.c +4 -9
@@ -3871,14 +3871,13 @@ static int rs_start_reshape(struct raid_set *rs)struct mddev *mddev = &rs->md;struct md_personality *pers = mddev->pers; +/* Don't allow the sync thread to work until the table gets reloaded. */ +set_bit (MD_RECOVERY_WAIT , &mddev->recovery); + r =rs_setup_reshape (rs);if (r)return r; -/* Need to be resumed to be able to start reshape, recovery is frozen until raid_resume() though */ -if (test_and_clear_bit (RT_FLAG_RS_SUSPENDED , &rs->runtime_flags)) -mddev_resume (mddev); -/* * Check any reshape constraints enforced by the personalility *
@@ -3902,10 +3901,6 @@ static int rs_start_reshape(struct raid_set *rs) } } -/* Suspend because a resume will happen in raid_resume() */ -set_bit (RT_FLAG_RS_SUSPENDED , &rs->runtime_flags); -mddev_suspend (mddev); -/* * Now reshape got set up, update superblocks to * reflect the fact so that a table reload will
@@ -4002,7 +3997,7 @@ static void raid_resume(struct dm_target *ti)static struct target_type raid_target = { .name ="raid" , - .version = {1 ,13 ,3 }, + .version = {1 ,14 ,0 }, .module =THIS_MODULE , .ctr = raid_ctr, .dtr = raid_dtr,
Signature · 1 line
2.17.1
PATCH 3/5 dm-raid: correct explicit superblock update requests
In reply to Heinz MauelshagenCritical:
Request superblock updates when particular devices are requested to be rebuild (e.g. via lvconvert --replace ...) to avoid racy "New device injected into existing raid set without 'delta_disks'
or 'rebuild' parameter specified" kernel error messages.
Cleanup:
Remove request to write superblocks in super_load() when new, because raid_ctr() requests it already via flag RT_FLAG_UPDATE_SBS for new arrays (it's performed in preresume).
Avoid explicit superblocks update in rs_start_reshape() because it's already requested before for new reshapes in rs_prepare_reshape().
--- drivers/md/dm-raid.c | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-)
drivers/md/dm-raid.c +6 -12
@@ -2202,9 +2202,6 @@ static int super_load(struct md_rdev *rdev, struct md_rdev *refdev)set_bit (FirstUse, &rdev->flags); sb->compat_features =cpu_to_le32 (FEATURE_FLAG_SUPPORTS_V190 ); -/* Force writing of superblocks to disk */ -set_bit (MD_SB_CHANGE_DEVS , &rdev->mddev->sb_flags); -/* Any superblock is better than none, choose that if given */ return refdev ?0 :1 ; }
@@ -3126,6 +3123,11 @@ static int raid_ctr(struct dm_target *ti, unsigned int argc, char **argv)set_bit (RT_FLAG_UPDATE_SBS , &rs->runtime_flags);rs_set_new (rs); }else if (rs_is_recovering (rs)) { +/* Rebuild particular devices */ +if (test_bit (__CTR_FLAG_REBUILD, &rs->ctr_flags)) { +set_bit (RT_FLAG_UPDATE_SBS , &rs->runtime_flags); +rs_setup_recovery (rs, MaxSector); + }/* A recovering raid set may be resized */ ;/* skip setup rs */ }else if (rs_is_reshaping (rs)) {
@@ -3234,7 +3236,6 @@ static int raid_ctr(struct dm_target *ti, unsigned int argc, char **argv) } r =md_start (&rs->md); -if (r) { ti->error ="Failed to start raid array" ;mddev_unlock (&rs->md);
@@ -3846,7 +3847,7 @@ static int __load_dirty_region_bitmap(struct raid_set *rs)return r; } -/* Enforce updating all superblocks */ +/* Enforce updating new superblocks */ static void rs_update_sbs (struct raid_set *rs) {struct mddev *mddev = &rs->md;
@@ -3901,13 +3902,6 @@ static int rs_start_reshape(struct raid_set *rs) } } -/* -* Now reshape got set up, update superblocks to -* reflect the fact so that a table reload will -* access proper superblock content in the ctr. -*/ -rs_update_sbs (rs); -return 0 ; }
Signature · 1 line
2.17.1
PATCH 4/5 dm-raid: share decipher_sync_action
In reply to Heinz Mauelshagendecipher_sync_action() is only called from raid_status() thus duplicating tests to define the current synchronization action (frozen, check, resync, ...) in rs_get_progress().
Change it to return an enum defining the sync states and share it to avoid code duplication. Introduce sync_str() to translate from enum to string in raid_status().
--- drivers/md/dm-raid.c | 88 ++++++++++++++++++++++++++------------------ 1 file changed, 52 insertions(+), 36 deletions(-)
drivers/md/dm-raid.c +52 -36
@@ -3330,32 +3330,55 @@ static int raid_map(struct dm_target *ti, struct bio *bio)return DM_MAPIO_SUBMITTED ; } -/* Return string describing the current sync action of @mddev */ -static const char *decipher_sync_action (struct mddev *mddev,unsigned long recovery) +/* Return sync state string for @state */ +enum sync_state { st_frozen, st_reshape, st_resync, st_check, st_repair, st_recover, st_idle }; +static const char *sync_str (enum sync_state state) +{ +/* Has to be in above sync_state order! */ +static const char *sync_strs[] = { +"frozen" , +"reshape" , +"resync" , +"check" , +"repair" , +"recover" , +"idle" + }; + +BUG_ON (!__within_range (state,0 ,ARRAY_SIZE (sync_strs) -1 )); + +return sync_strs[state]; +}; + +/* Return enum sync_state for @mddev derived from @recovery flags */ +static const enum sync_state decipher_sync_action (struct mddev *mddev,unsigned long recovery) {if (test_bit (MD_RECOVERY_FROZEN , &recovery)) -return "frozen" ; +return st_frozen; -/* The MD sync thread can be done with io but still be running */ +/* The MD sync thread can be done with io or be interrupted but still be running */ if (!test_bit (MD_RECOVERY_DONE , &recovery) && (test_bit (MD_RECOVERY_RUNNING , &recovery) || (!mddev->ro &&test_bit (MD_RECOVERY_NEEDED , &recovery)))) {if (test_bit (MD_RECOVERY_RESHAPE , &recovery)) -return "reshape" ; +return st_reshape;if (test_bit (MD_RECOVERY_SYNC , &recovery)) {if (!test_bit (MD_RECOVERY_REQUESTED , &recovery)) -return "resync" ; -else if (test_bit (MD_RECOVERY_CHECK , &recovery)) -return "check" ; -return "repair" ; +return st_resync; +if (test_bit (MD_RECOVERY_CHECK , &recovery)) +return st_check; +return st_repair; }if (test_bit (MD_RECOVERY_RECOVER , &recovery)) -return "recover" ; +return st_recover; + +if (mddev->reshape_position != MaxSector) +return st_reshape; } -return "idle" ; +return st_idle; }/*
@@ -3389,6 +3412,7 @@ static sector_t rs_get_progress(struct raid_set *rs, unsigned long recovery, sector_t resync_max_sectors) {sector_t r; +enum sync_state state;struct mddev *mddev = &rs->md;clear_bit (RT_FLAG_RS_IN_SYNC , &rs->runtime_flags);
@@ -3399,20 +3423,14 @@ static sector_t rs_get_progress(struct raid_set *rs, unsigned long recovery,set_bit (RT_FLAG_RS_IN_SYNC , &rs->runtime_flags); }else { -if (!test_bit (__CTR_FLAG_NOSYNC, &rs->ctr_flags) && - !test_bit (MD_RECOVERY_INTR , &recovery) && - (test_bit (MD_RECOVERY_NEEDED , &recovery) || -test_bit (MD_RECOVERY_RESHAPE , &recovery) || -test_bit (MD_RECOVERY_RUNNING , &recovery))) - r = mddev->curr_resync_completed; -else + state =decipher_sync_action (mddev, recovery); + +if (state == st_idle && !test_bit (MD_RECOVERY_INTR , &recovery)) r = mddev->recovery_cp; +else + r = mddev->curr_resync_completed; -if (r >= resync_max_sectors && - (!test_bit (MD_RECOVERY_REQUESTED , &recovery) || - (!test_bit (MD_RECOVERY_FROZEN , &recovery) && - !test_bit (MD_RECOVERY_NEEDED , &recovery) && - !test_bit (MD_RECOVERY_RUNNING , &recovery)))) { +if (state == st_idle && r >= resync_max_sectors) {/* * Sync complete. */
@@ -3420,24 +3438,20 @@ static sector_t rs_get_progress(struct raid_set *rs, unsigned long recovery,if (test_bit (MD_RECOVERY_RECOVER , &recovery))set_bit (RT_FLAG_RS_IN_SYNC , &rs->runtime_flags); - }else if (test_bit (MD_RECOVERY_RECOVER , &recovery)) { + }else if (state == st_recover)/* * In case we are recovering, the array is not in sync * and health chars should show the recovering legs. */ ; - - }else if (test_bit (MD_RECOVERY_SYNC , &recovery) && - !test_bit (MD_RECOVERY_REQUESTED , &recovery)) { +else if (state == st_resync)/* * If "resync" is occurring, the raid set * is or may be out of sync hence the health * characters shall be 'a'. */ set_bit (RT_FLAG_RS_RESYNCING , &rs->runtime_flags); - - }else if (test_bit (MD_RECOVERY_RESHAPE , &recovery) && - !test_bit (MD_RECOVERY_REQUESTED , &recovery)) { +else if (state == st_reshape)/* * If "reshape" is occurring, the raid set * is or may be out of sync hence the health
@@ -3445,7 +3459,7 @@ static sector_t rs_get_progress(struct raid_set *rs, unsigned long recovery,*/ set_bit (RT_FLAG_RS_RESYNCING , &rs->runtime_flags); - }else if (test_bit (MD_RECOVERY_REQUESTED , &recovery)) { + else if (state == st_check || state == st_repair)/* * If "check" or "repair" is occurring, the raid set has * undergone an initial sync and the health characters
@@ -3453,12 +3467,12 @@ static sector_t rs_get_progress(struct raid_set *rs, unsigned long recovery,*/ set_bit (RT_FLAG_RS_IN_SYNC , &rs->runtime_flags); - }else { + else {struct md_rdev *rdev;/* * We are idle and recovery is needed, prevent 'A' chars race -* caused by components still set to in-sync by constrcuctor. +* caused by components still set to in-sync by constructor. */ if (test_bit (MD_RECOVERY_NEEDED , &recovery))set_bit (RT_FLAG_RS_RESYNCING , &rs->runtime_flags);
@@ -3522,7 +3536,7 @@ static void raid_status(struct dm_target *ti, status_type_t type, progress =rs_get_progress (rs, recovery, resync_max_sectors); resync_mismatches = (mddev->last_sync_action && !strcasecmp (mddev->last_sync_action,"check" )) ?atomic64_read (&mddev->resync_mismatches) :0 ; - sync_action =decipher_sync_action (&rs->md, recovery); + sync_action =sync_str (decipher_sync_action (&rs->md, recovery));/* HM FIXME: do we want another state char for raid0? It shows 'D'/'A'/'-' now */ for (i =0 ; i < rs->raid_disks; i++)
@@ -3739,12 +3753,14 @@ static void raid_postsuspend(struct dm_target *ti)struct raid_set *rs = ti->private;if (!test_and_set_bit (RT_FLAG_RS_SUSPENDED , &rs->runtime_flags)) { +BUG_ON (rs->md.suspended); +/* Writes have to be stopped before suspending to avoid deadlocks. */ -if (!test_bit (MD_RECOVERY_FROZEN , &rs->md.recovery)) -md_stop_writes (&rs->md); +md_stop_writes (&rs->md);mddev_lock_nointr (&rs->md);mddev_suspend (&rs->md); + rs->md.ro =1 ;mddev_unlock (&rs->md); } }
Signature · 1 line
2.17.1
PATCH 5/5 dm-raid: disable bitmap when journaled
In reply to Heinz MauelshagenWhen a journal device for raid4/5/6 is used, don't create a mutually exclusive write intent bitmap.
--- drivers/md/dm-raid.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/md/dm-raid.c +1 -1
@@ -2472,7 +2472,7 @@ static int super_validate(struct raid_set *rs, struct md_rdev *rdev) }/* Enable bitmap creation for RAID levels != 0 */ - mddev->bitmap_info.offset =rt_is_raid0 (rs->raid_type) ?0 :to_sector (4096 ); + mddev->bitmap_info.offset = (rt_is_raid0 (rs->raid_type) || rs->journal_dev.dev) ?0 :to_sector (4096 ); mddev->bitmap_info.default_offset = mddev->bitmap_info.offset;if (!test_and_clear_bit (FirstUse, &rdev->flags)) {
Signature · 1 line
2.17.1