Implement {begin|after}-resync-source handlers

These small patches duplicate the resync-target handlers for the other end of the connection.

Some notes:

- I'm not familiar with drbdadm itself, so I'm not sure if the changes to it are sane; I did check and test that the kernel changes are fine and fire the proper callbacks, but was not able to test the drbdadm configuration
- Aborting if the userspace helper exits with non-zero code is done to mirror the before-resync-target helper; but ignoring its result is another choice, if tearing down the connections is deemed to harsh

Iustin Pop (3):
Implement before|after-resync-source in drbdadm Make the drbd driver generate resync-source callbacks Update documentation for resync-source handlers

 documentation/drbd.conf.xml |   27 ++++++++++++++++++++++++++-
 drbd/drbd_worker.c          |   14 ++++++++++++++
 scripts/drbd.conf.example   |    4 ++++
 scripts/global_common.conf  |    2 ++
 user/drbdadm_main.c         |    2 ++
 user/drbdadm_scanner.fl     |    2 ++
 6 files changed, 50 insertions(+), 1 deletions(-)

---
 scripts/global_common.conf |    2 ++
 user/drbdadm_main.c        |    2 ++
 user/drbdadm_scanner.fl    |    2 ++
 3 files changed, 6 insertions(+), 0 deletions(-)

index 5269515..ad07ce2 100644
@@ -15,6 +15,8 @@ common {
 		# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
 		# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
 		# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
+		# before-resync-source /bin/true;
+		# after-resync-target /bin/true;
 	}
 
 	startup {

index b697fd6..38d427a 100644
@@ -327,6 +327,8 @@ struct adm_cmd cmds[] = {
 
 	{"before-resync-target", adm_khelper, DRBD_acf3_handler},
 	{"after-resync-target", adm_khelper, DRBD_acf3_handler},
+	{"before-resync-source", adm_khelper, DRBD_acf3_handler},
+	{"after-resync-source", adm_khelper, DRBD_acf3_handler},
 	{"pri-on-incon-degr", adm_khelper, DRBD_acf3_handler},
 	{"pri-lost-after-sb", adm_khelper, DRBD_acf3_handler},
 	{"fence-peer", adm_khelper, DRBD_acf3_handler},

index d4cad32..a4a9441 100644
@@ -141,6 +141,8 @@ fence-peer		{ DP; CP; return TK_HANDLER_OPTION;	}
 local-io-error		{ DP; CP; return TK_HANDLER_OPTION;     }
 before-resync-target	{ DP; CP; return TK_HANDLER_OPTION;	}
 after-resync-target	{ DP; CP; return TK_HANDLER_OPTION;	}
+before-resync-source	{ DP; CP; return TK_HANDLER_OPTION;	}
+after-resync-source	{ DP; CP; return TK_HANDLER_OPTION;	}
 memlimit		{ DP; CP; return TK_PROXY_OPTION;       }
 read-loops		{ DP; CP; return TK_PROXY_OPTION;       }
 compression		{ DP; CP; return TK_PROXY_OPTION;       }

1.7.0.1

---
 drbd/drbd_worker.c |   14 ++++++++++++++
 1 files changed, 14 insertions(+), 0 deletions(-)

index df50bef..bee4ce2 100644
@@ -761,6 +761,9 @@ int drbd_resync_finished(struct drbd_conf *mdev)
 		if (os.conn == C_SYNC_TARGET || os.conn == C_PAUSED_SYNC_T)
 			khelper_cmd = "after-resync-target";
 
+		if (os.conn == C_SYNC_SOURCE || os.conn == C_PAUSED_SYNC_S)
+			khelper_cmd = "after-resync-source";
+
 		if (mdev->csums_tfm && mdev->rs_total) {
 			const unsigned long s = mdev->rs_same_csum;
 			const unsigned long t = mdev->rs_total;

@@ -1365,6 +1368,17 @@ void drbd_start_resync(struct drbd_conf *mdev, enum drbd_conns side)
 			drbd_force_state(mdev, NS(conn, C_DISCONNECTING));
 			return;
 		}
+	} else {
+		/* Notify the user-space helper that we're becoming
+		   the source of a resync */
+		r = drbd_khelper(mdev, "before-resync-source");
+		r = (r >> 8) & 0xff;
+		if (r > 0) {
+			dev_info(DEV, "before-resync-source handler returned %d, "
+			     "dropping connection.\n", r);
+			drbd_force_state(mdev, NS(conn, C_DISCONNECTING));
+			return;
+		}
 	}
 
 	drbd_state_lock(mdev);

1.7.0.1

---
 documentation/drbd.conf.xml |   27 ++++++++++++++++++++++++++-
 scripts/drbd.conf.example   |    4 ++++
 2 files changed, 30 insertions(+), 1 deletions(-)

index 8e4fdfd..656d2ce 100644
@@ -330,7 +330,8 @@ resource r0 {
     <option>pri-on-incon-degr</option>, <option>pri-lost-after-sb</option>,
     <option>pri-lost</option>, <option>fence-peer</option> (formerly oudate-peer),
     <option>local-io-error</option>, <option>initial-split-brain</option>, <option>split-brain</option>,
-    <option>before-resync-target</option>, <option>after-resync-target</option>.
+    <option>before-resync-target</option>, <option>after-resync-target</option>,
+    <option>before-resync-source</option>, <option>after-resync-source</option>.
   </para>
           </listitem>
         </varlistentry>

@@ -1509,6 +1510,30 @@ the unit is 1 second.
   </para>
           </listitem>
         </varlistentry>
+        <varlistentry>
+          <term>
+            <option>before-resync-source <replaceable>cmd</replaceable></option>
+          </term>
+          <listitem>
+            <para><indexterm significance="normal"><primary>drbd.conf</primary><secondary>before-resync-source </secondary></indexterm>
+  DRBD calls this handler just before a resync beginns on the node
+  that becomes resync source. It might be used to tweak the resync parameters
+  or the backing block device.
+  </para>
+          </listitem>
+        </varlistentry>
+        <varlistentry>
+          <term>
+            <option>after-resync-source <replaceable>cmd</replaceable></option>
+          </term>
+          <listitem>
+            <para><indexterm significance="normal"><primary>drbd.conf</primary><secondary>after-resync-source </secondary></indexterm>
+  DRBD calls this handler just after a resync operation finished on
+  the node which served as the resync source. It might be used to
+  unpause any other minors that were suspended, log the resync end, etc.
+  </para>
+          </listitem>
+        </varlistentry>
       </variablelist>
     </refsect2>
     <refsect2>

index cc58814..6ae8816 100644
@@ -174,6 +174,10 @@ resource r0 {
     # -- <additional lvcreate options>
     #before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
     #after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
+
+    # Example setting for the resync-source-handlers.
+    #before-resync-source "/usr/bin/logger Begun resync";
+    #after-resync-source "/usr/bin/logger Finished resync";
   }
 
   startup {

1.7.0.1

Am Montag 17 Mai 2010 14:37:57 schrieb Iustin Pop:

Hi Iustin,

The patches look okay so far, I am considering to apply them to our tree.

* Have you ever considered to use the "drbdsetup events" interface instead?

* Are you going to use them in your organisation?

Best,
Phil

Yep, I thought about it, and might use it if the handler method is not good enough (for whatever reason). But I'd rather like to use the handlers, since they allow reaction before the actual sync starts, and thus they eliminate a window of "uncontrolled" syncs (see below why we want to control them).

That too, but better said we're going to use them in Ganeti (opensource, cluster-based virtualization manager). Background: Ganeti uses only the low-level drbdsetup interface, and not the drbdadm one, since we manage the minor numbers assignments dynamically. This dynamic part hits us, because we cannot make easy 'resync-after' rules (especially as on bigger Ganeti clusters, like 20-40 nodes, you could get easily into circular dependencies if you're not careful.

So the plan is to write a custom usermode_helper (right now Ganeti documentations says to use /bin/true :), so we're not losing any functionality by moving to a custom helper), that will simply restrict DRBD to a single sync (either incoming or outgoing) per machine, in order to not overload any machine's I/O resources. When a begin-sync request comes in, either source or target, the helper will look if any other minors are undergoing syncs, and if so will drbdsetup pause-sync the minor - I hope that this is safe to do from the callback, my few tests show it so. And when a after-resync-* requests comes, we look if there are any paused syncs (on our end), and if so then we'll unpause one.

Now, if drbd (the kernel driver) would have a setting like 'max_syncs', then we wouldn't have to implement anything in Ganeti, but that is probably more work in the kernel - which is harder, compared with a simple userspace shell script.

Hope my explanation makes sense…
iustin

[..]

Hi Iustin,

Yes, makes sense. It will go in, after the 8.3.8 release dust settled.

Best,
Phil

: Dipl-Ing Philipp Reisner
: LINBIT | Your Way to High Availability
: Tel: +43-1-8178292-50, Fax: +43-1-8178292-82
: http://www.linbit.com

DRBD(R) and LINBIT(R) are registered trademarks of LINBIT, Austria.