mirror force resync option
PATCH 1/1 mirror force resync option
Patch originally developed by Jun'ichi Nomura and altered to be cluster capable.
brassow
This patch adds the --forcesync option to lvchange, allowing users to force mirrors to resynchronize.
This operation is performed by clearing out the log device. It requires that the mirror be deactivated to do this. If the mirror is not currently active, the log device is activated, cleared, and deactivated. If the mirror is currently active, the mirror is deactivated, the log device is activated and cleared, and the mirror is reactivated.
If the mirror is in use, it can not be deactivated; and therefore, can not be forcesynced. This creates a particular anomaly in the cluster context. If the mirror is active on some nodes in the cluster, but not the node where the command is issued, the command "succeeds" - even though the remote nodes maintain the same sync status. I view this as a bug in the 'deactivate_lv' code, since it should fail if it is unable to deactivate the lv on all nodes. Comments on this are welcome.
man/lvchange.8 +4 -0
@@ -7,6 +7,7 @@ lvchange \- change attributes of a logic [\-A/\-\-autobackup y/n] [\-a/\-\-available y/n/ey/en/ly/ln] [\-\-alloc AllocationPolicy] [\-C/\-\-contiguous y/n] [\-d/\-\-debug] [\-\-deltag Tag] +[\-\-forcesync] [\-h/\-?/\-\-help] [\-\-ignorelockingfailure] [\-\-monitor {y|n}]
@@ -40,6 +41,9 @@ logical volumes. It's only possible to c logical volume's allocation policy to contiguous, if all of the allocated physical extents are already contiguous. .TP +.I \-\-forcesync +Enforce resynching the mirrored logical volumes. +.TP .I \-\-minor minor Set the minor number. .TP
tools/args.h +1 -0
@@ -46,6 +46,7 @@ arg(alloc_ARG, '\0', "alloc", alloc_arg) arg(separator_ARG, '\0', "separator", string_arg) arg(mirrorsonly_ARG, '\0', "mirrorsonly", NULL) arg(nosync_ARG, '\0', "nosync", NULL) +arg(forcesync_ARG, '\0', "forcesync", NULL) arg(corelog_ARG, '\0', "corelog", NULL) arg(monitor_ARG, '\0', "monitor", yes_no_arg)arg (config_ARG,'\0' ,"config" , string_arg)
tools/commands.h +2 -1
@@ -61,6 +61,7 @@ xx(lvchange,"\t[-d|--debug]\n" "\t[--deltag Tag]\n" "\t[-f|--force]\n" +"\t[--forcesync]\n" "\t[-h|--help]\n" "\t[--ignorelockingfailure]\n" "\t[--monitor {y|n}]\n"
@@ -75,7 +76,7 @@ xx(lvchange,"\tLogicalVolume[Path] [LogicalVolume[Path]...]\n" , alloc_ARG, autobackup_ARG, available_ARG, contiguous_ARG, force_ARG, - ignorelockingfailure_ARG, major_ARG, minor_ARG, monitor_ARG, + forcesync_ARG, ignorelockingfailure_ARG, major_ARG, minor_ARG, monitor_ARG, partial_ARG, permission_ARG, persistent_ARG, readahead_ARG, refresh_ARG, addtag_ARG, deltag_ARG, test_ARG)
tools/lvchange.c +103 -2
124 lines not on this page · raw message
Quoted 3 lines · Jon Brassow
I view
this as a bug in the 'deactivate_lv' code, since it should fail if it is unable to deactivate the lv on all nodes.
Correct, if that's true. But the code doesn't show any deactivation attempt! (see below)
Quoted 1 line
+Enforce resynching the mirrored logical volumes.
Needs explaining better here for someone who doesn't know what that means/involves:-)
Review
Quoted 2 lines
(quoted diff) +2
-0+ if (!(lv->status & MIRRORED)) + return 1;
Are you sure this shouldn't be an error?
tools/lvchange.c static int lvchange_refresh(struct cmd_c · line 196
Quoted 2 lines
tools/lvchange.c +2
-0+if (!lv_info (cmd, lv, &info,0 ) || !info.exists) + active =0 ;
This is currently defined as telling you if the device is active on the local node. (Patrick's looking at extending it to the cluster.)
tools/lvchange.c static int lvchange_refresh(struct cmd_c · line 203
Quoted 6 lines
tools/lvchange.c +6
-0+if (active) { +if (!deactivate_lv (cmd, lv)) { +log_error ("Unable to deactivate, %s, for sync status change" , lv->name); +log_error ("Hint: %s may be in-use" , lv->name); +return 0 ; + }
On a cluster, you need to call deactivate_lv *even if* it's not active on the local node. So test for the VG clustered flag and use that to decide whether to do the lv_info/active test or not. (That's only a local optimisation normally to assist with the error messages.)
tools/lvchange.c static int lvchange_refresh(struct cmd_c · line 211
Quoted 7 lines
tools/lvchange.c +7
-0+/* Must activate log, or we can't zero it */ +if (!activate_lv (cmd,first_seg (lv)->log_lv)) { +log_error ("Unable to activate, %s, for sync status change" , +first_seg (lv)->log_lv->name); +activate_lv (cmd, lv); +return 0 ; + }
Interesting. That extension to the interface needs more comment: you need to
demonstrate that the operation is either atomic (which it doesn't appear to be at first sight), or else if it's not atomic, there must be no failure modes that lead to corruption, or people thinking a complete resync got forced when it actually didn't.
Normally you're not allowed to activate only part of a device tree - you can't
activate a mirror log except as part of the mirror. Notice how lvcreate
and lvconvert manipulate the log independently before attaching it to the mirror. However the recent changes to prepare for stacked devices should be sufficient for that particular activation to work, so that's probably acceptable now.
A breakdown into atomic operations might be:
Detach the log (leaving the [inactive] mirror with a core log), wipe the log, then reattach it.
Or you could introduce an additional metadata state to indicate the log needs to be wiped, wipe it, then revert to the previous metadata state. Or have a combination of both (so that a future lvchange -ay can complete the process if interrupted).
tools/lvchange.c static int lvchange_refresh(struct cmd_c · line 216
Quoted 4 lines
tools/lvchange.c +4
-0+/* +* Note: there is no need to deactivate the log before +* we activate the mirror +*/
Because of the stacking preparation now, yes, so long as set_lv() syncs the block device correctly.
Alasdair
Signature · 1 line
[email protected]
Quoted 7 lines · Alasdair G Kergon
Quoted 3 lines · Jon Brassow
I view
this as a bug in the 'deactivate_lv' code, since it should fail if it is unable to deactivate the lv on all nodes.Correct, if that's true. But the code doesn't show any deactivation attempt! (see below)
Good spot. I was incorrectly understanding the lv_info... line to identify active lv's on other machines.
Quoted 4 lines
Quoted 1 line
+Enforce resynching the mirrored logical volumes.
Needs explaining better here for someone who doesn't know what that means/involves:-)
Explanation expanded.
Quoted 5 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ if (!(lv->status & MIRRORED)) + return 1;Are you sure this shouldn't be an error?
I haven't changed this. So, if tried on a linear volume, there would be no error (but nothing would be done). This is easily changed if you like.
Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ if (!lv_info(cmd, lv, &info, 0) || !info.exists) + active = 0;This is currently defined as telling you if the device is active on the local node. (Patrick's looking at extending it to the cluster.)
right.
Quoted 13 lines
Review
Quoted 8 lines
(quoted diff) +3
-0+ if (active) { + if (!deactivate_lv(cmd, lv)) { + log_error("Unable to deactivate, %s, for sync status change",
lv->name);(quoted diff) +3
-0+ log_error("Hint: %s may be in-use", lv->name); + return 0; + }On a cluster, you need to call
deactivate_lv*even if* it's not active on the local node. So test for the VG clustered flag and use that to decide whether to do the lv_info/active test or not. (That's only a local optimisation normally to assist with the error messages.)
right
Quoted 30 lines
Review
Quoted 8 lines
(quoted diff) +7
-0+ /* Must activate log, or we can't zero it */ + if (!activate_lv(cmd, first_seg(lv)->log_lv)) { + log_error("Unable to activate, %s, for sync status change", + first_seg(lv)->log_lv->name); + activate_lv(cmd, lv); + return 0; + }Interesting. That extension to the interface needs more comment: you need to
demonstrate that the operation is either atomic (which it doesn't appear to be at first sight), or else if it's not atomic, there must be no failure modes that lead to corruption, or people thinking a complete resync got forced when it actually didn't.Normally you're not allowed to activate only part of a device tree - you can't
activate a mirror log except as part of the mirror. Notice how lvcreate
and lvconvert manipulate the log independently before attaching it to the mirror. However the recent changes to prepare for stacked devices should be sufficient for that particular activation to work, so that's probably acceptable now.A breakdown into atomic operations might be:
Detach the log (leaving the [inactive] mirror with a core log), wipe the log, then reattach it.
Or you could introduce an additional metadata state to indicate the log needs to be wiped, wipe it, then revert to the previous metadata state. Or have a combination of both (so that a future lvchange -ay can complete the process if interrupted).
Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ /* + * Note: there is no need to deactivate the log before + * we activate the mirror + */Because of the stacking preparation now, yes, so long as
set_lv()syncs the block device correctly.
Basically, I only added more to the man page and a check for CLUSTERED in tools/lvchange.c. I've also added some comments in the patch as to why I believe the operation is atomic - posing no inconsistencies should the operation/machine fail while part way through.
brassow
This patch adds the --forcesync option to lvchange, allowing users to force mirrors to resynchronize.
This operation is performed by clearing out the log device. It requires that the mirror be deactivated to do this. If the mirror is not currently active, the log device is activated, cleared, and deactivated. If the mirror is currently active, the mirror is deactivated, the log device is activated and cleared, and the mirror is reactivated.
If the mirror is in use, it can not be deactivated; and therefore, can not be forcesynced.
A note about atomicity... When clearing the log device, the first sector that gets overwritten is the one containing the log magic number. Once overwritten, the mirror will resynchronize by default. So, the machine can die before or after the first sector is written. This results in the desired action being taken or not taken, but not partially taken. As far as the activation/deactivation process that is happening, every case is accounted for by an error message and backout efforts.
man/lvchange.8 +9 -0
@@ -7,6 +7,7 @@ lvchange \- change attributes of a logic [\-A/\-\-autobackup y/n] [\-a/\-\-available y/n/ey/en/ly/ln] [\-\-alloc AllocationPolicy] [\-C/\-\-contiguous y/n] [\-d/\-\-debug] [\-\-deltag Tag] +[\-\-forcesync] [\-h/\-?/\-\-help] [\-\-ignorelockingfailure] [\-\-monitor {y|n}]
@@ -40,6 +41,14 @@ logical volumes. It's only possible to c logical volume's allocation policy to contiguous, if all of the allocated physical extents are already contiguous. .TP +.I \-\-forcesync +This option only make sense if the logical volume being operated +on is a mirrored logical volume. This option is used to +resynchronize the devices in a mirrored logical volume. Since +resynchronization can take considerable time and resources, you +should consider using this option only if you have reason to +believe that the devices in your mirror are no longer the same. +.TP .I \-\-minor minor Set the minor number. .TP
tools/args.h +1 -0
@@ -46,6 +46,7 @@ arg(alloc_ARG, '\0', "alloc", alloc_arg) arg(separator_ARG, '\0', "separator", string_arg) arg(mirrorsonly_ARG, '\0', "mirrorsonly", NULL) arg(nosync_ARG, '\0', "nosync", NULL) +arg(forcesync_ARG, '\0', "forcesync", NULL) arg(corelog_ARG, '\0', "corelog", NULL) arg(monitor_ARG, '\0', "monitor", yes_no_arg)arg (config_ARG,'\0' ,"config" , string_arg)
tools/commands.h +2 -1
@@ -61,6 +61,7 @@ xx(lvchange,"\t[-d|--debug]\n" "\t[--deltag Tag]\n" "\t[-f|--force]\n" +"\t[--forcesync]\n" "\t[-h|--help]\n" "\t[--ignorelockingfailure]\n" "\t[--monitor {y|n}]\n"
@@ -75,7 +76,7 @@ xx(lvchange,"\tLogicalVolume[Path] [LogicalVolume[Path]...]\n" , alloc_ARG, autobackup_ARG, available_ARG, contiguous_ARG, force_ARG, - ignorelockingfailure_ARG, major_ARG, minor_ARG, monitor_ARG, + forcesync_ARG, ignorelockingfailure_ARG, major_ARG, minor_ARG, monitor_ARG, partial_ARG, permission_ARG, persistent_ARG, readahead_ARG, refresh_ARG, addtag_ARG, deltag_ARG, test_ARG)
tools/lvchange.c +105 -2
126 lines not on this page · raw message