Skip to contentSkip to conversation

ocfs2/dlm: dlm_is_lockres_migrateable() returns boolean

PATCH New Sunil Mushran 4 participants 13 messages Apr 26, 2011 – May 24, 2011 OCFS2

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 ocfs2/dlm: dlm_is_lockres_migrateable() returns boolean

Patch cleans up the gunk added by commit 388c4bcb4e63e88fb1f312a2f5f9eb2623afcf5b. dlm_is_lockres_migrateable() now returns 1 if lockresource is deemed migrateable and 0 if not.

---
 fs/ocfs2/dlm/dlmcommon.h |   12 ++++
 fs/ocfs2/dlm/dlmmaster.c |  135 +++++++++++++++++----------------------------
 2 files changed, 63 insertions(+), 84 deletions(-)
fs/ocfs2/dlm/dlmcommon.h +12 -0
index 4bdf7ba..1aac42a 100644
@@ -401,6 +401,18 @@ static inline int dlm_lvb_is_empty(char *lvb)
 	return 1;
 }
 
+static inline char *dlm_list_in_text(enum dlm_lockres_list idx)
+{
+	if (idx == DLM_GRANTED_LIST)
+		return "granted";
+	else if (idx == DLM_CONVERTING_LIST)
+		return "converting";
+	else if (idx == DLM_BLOCKED_LIST)
+		return "blocked";
+	else
+		return "unknown";
+}
+
 static inline struct list_head *
 dlm_list_idx_to_ptr(struct dlm_lock_resource *res, enum dlm_lockres_list idx)
 {
fs/ocfs2/dlm/dlmmaster.c +51 -84

201 lines not on this page · raw message

Signature · 1 line
1.7.1
#1 lore raw

PATCH 2/3 ocfs2/dlm: Add new dlm message DLM_BEGIN_EXIT_DOMAIN_MSG

This patch adds a new dlm message DLM_BEGIN_EXIT_DOMAIN_MSG and ups the dlm protocol to 1.2.

o2dlm sends this new message in dlm_unregister_domain() to mark the beginning of the exit domain. This message is sent to all nodes in the domain.

Currently o2dlm has no way of informing other nodes of its impending exit. This information is useful as the other nodes could disregard the exiting node in certain operations. For example, in resource migration. If two or more nodes were umounting in parallel, it would be more efficient if o2dlm were to choose a non-exiting node to be the new master node rather than an exiting one.

---
 fs/ocfs2/dlm/dlmcommon.h   |    2 +
 fs/ocfs2/dlm/dlmdebug.c    |    6 +++
 fs/ocfs2/dlm/dlmdomain.c   |   84 ++++++++++++++++++++++++++++++++++++++------
 fs/ocfs2/dlm/dlmrecovery.c |    1 +
 4 files changed, 82 insertions(+), 11 deletions(-)
fs/ocfs2/dlm/dlmcommon.h +2 -0
index 1aac42a..d602abb 100644
@@ -144,6 +144,7 @@ struct dlm_ctxt
 	wait_queue_head_t dlm_join_events;
 	unsigned long live_nodes_map[BITS_TO_LONGS(O2NM_MAX_NODES)];
 	unsigned long domain_map[BITS_TO_LONGS(O2NM_MAX_NODES)];
+	unsigned long exit_domain_map[BITS_TO_LONGS(O2NM_MAX_NODES)];
 	unsigned long recovery_map[BITS_TO_LONGS(O2NM_MAX_NODES)];
 	struct dlm_recovery_ctxt reco;
 	spinlock_t master_lock;
@@ -460,6 +461,7 @@ enum {
 	DLM_FINALIZE_RECO_MSG		= 518,
 	DLM_QUERY_REGION		= 519,
 	DLM_QUERY_NODEINFO		= 520,
+	DLM_BEGIN_EXIT_DOMAIN_MSG	= 521,
 };
 
 struct dlm_reco_node_data
fs/ocfs2/dlm/dlmdebug.c +6 -0
index 04a32be..56f82cb 100644
@@ -756,6 +756,12 @@ static int debug_state_print(struct dlm_ctxt *dlm, char *buf, int len)
 				 buf + out, len - out);
 	out += snprintf(buf + out, len - out, "\n");
 
+	/* Exit Domain Map: xx xx xx */
+	out += snprintf(buf + out, len - out, "Exit Domain Map: ");
+	out += stringify_nodemap(dlm->exit_domain_map, O2NM_MAX_NODES,
+				 buf + out, len - out);
+	out += snprintf(buf + out, len - out, "\n");
+
 	/* Live Map: xx xx xx */
 	out += snprintf(buf + out, len - out, "Live Map: ");
 	out += stringify_nodemap(dlm->live_nodes_map, O2NM_MAX_NODES,
fs/ocfs2/dlm/dlmdomain.c +73 -11

149 lines not on this page · raw message

fs/ocfs2/dlm/dlmrecovery.c +1 -0
index f1beb6f..7efab6d 100644
@@ -2393,6 +2393,7 @@ static void __dlm_hb_node_down(struct dlm_ctxt *dlm, int idx)
 
 	mlog(0, "node %u being removed from domain map!\n", idx);
 	clear_bit(idx, dlm->domain_map);
+	clear_bit(idx, dlm->exit_domain_map);
 	/* wake up migration waiters if a node goes down.
 	 * perhaps later we can genericize this for other waiters. */
 	wake_up(&dlm->migration_wq);
Signature · 1 line
1.7.1
#2 lore raw

PATCH 2/3 ocfs2/dlm: Add new dlm message DLM_BEGIN_EXIT_DOMAIN_MSG

Quoted 6 lines · Sunil Mushran

This patch adds a new dlm message DLM_BEGIN_EXIT_DOMAIN_MSG and ups the dlm protocol to 1.2.

o2dlm sends this new message in dlm_unregister_domain() to mark the beginning of the exit domain. This message is sent to all nodes in the domain.

Quoted 8 lines

Currently o2dlm has no way of informing other nodes of its impending exit. This information is useful as the other nodes could disregard the exiting node in certain operations. For example, in resource migration. If two or more nodes were umounting in parallel, it would be more efficient if o2dlm were to choose a non-exiting node to be the new master node rather than an exiting one.

Looks good, glad we're checking proto versions before sending the message ;)

	--Mark

--
Mark Fasheh

#3 lore raw

PATCH 3/3 ocfs2/dlm: Do not migrate resource to a node that is leaving the domain

In reply to Sunil Mushran

During dlm domain shutdown, o2dlm has to free all the lock resources. Ones that have no locks and references are freed. Ones that have locks and/or references are migrated to another node.

The first task in migration is finding a target. Currently we scan the lock resource and find one node that either has a lock or a reference. This is not very efficient in a parallel umount case as we might end up migrating the lock resource to a node which itself may have to migrate it to a third node.

The patch scans the dlm->exit_domain_map to ensure the target node is not leaving the domain. If no valid target node is found, o2dlm does not migrate the resource but instead waits for the unlock and deref messages that will allow it to free the resource.

---
 fs/ocfs2/dlm/dlmdomain.c |   10 ++-
 fs/ocfs2/dlm/dlmmaster.c |  139 ++++++++++++++++-----------------------------
 2 files changed, 57 insertions(+), 92 deletions(-)
fs/ocfs2/dlm/dlmdomain.c +7 -3
index 3aff23f..6ed6b95 100644
@@ -451,14 +451,18 @@ redo_bucket:
 			dropped = dlm_empty_lockres(dlm, res);
 
 			spin_lock(&res->spinlock);
-			__dlm_lockres_calc_usage(dlm, res);
-			iter = res->hash_node.next;
+			if (dropped)
+				__dlm_lockres_calc_usage(dlm, res);
+			else
+				iter = res->hash_node.next;
 			spin_unlock(&res->spinlock);
 
 			dlm_lockres_put(res);
 
-			if (dropped)
+			if (dropped) {
+				cond_resched_lock(&dlm->spinlock);
 				goto redo_bucket;
+			}
 		}
 		cond_resched_lock(&dlm->spinlock);
 		num += n;
fs/ocfs2/dlm/dlmmaster.c +50 -89

220 lines not on this page · raw message

Signature · 1 line
1.7.1
#4 lore raw

PATCH 3/3 ocfs2/dlm: Do not migrate resource to a node that is leaving the domain

Quoted 252 lines · Sunil Mushran

During dlm domain shutdown, o2dlm has to free all the lock resources. Ones that have no locks and references are freed. Ones that have locks and/or references are migrated to another node.

The first task in migration is finding a target. Currently we scan the lock resource and find one node that either has a lock or a reference. This is not very efficient in a parallel umount case as we might end up migrating the lock resource to a node which itself may have to migrate it to a third node.

The patch scans the dlm->exit_domain_map to ensure the target node is not leaving the domain. If no valid target node is found, o2dlm does not migrate the resource but instead waits for the unlock and deref messages that will allow it to free the resource.

---
 fs/ocfs2/dlm/dlmdomain.c |   10 ++-
 fs/ocfs2/dlm/dlmmaster.c |  139 ++++++++++++++++-----------------------------
 2 files changed, 57 insertions(+), 92 deletions(-)
fs/ocfs2/dlm/dlmdomain.c +7 -3
index 3aff23f..6ed6b95 100644
@@ -451,14 +451,18 @@ redo_bucket:
 			dropped = dlm_empty_lockres(dlm, res);
 
 			spin_lock(&res->spinlock);
-			__dlm_lockres_calc_usage(dlm, res);
-			iter = res->hash_node.next;
+			if (dropped)
+				__dlm_lockres_calc_usage(dlm, res);
+			else
+				iter = res->hash_node.next;
 			spin_unlock(&res->spinlock);
 
 			dlm_lockres_put(res);
 
-			if (dropped)
+			if (dropped) {
+				cond_resched_lock(&dlm->spinlock);
 				goto redo_bucket;
+			}
 		}
 		cond_resched_lock(&dlm->spinlock);
 		num += n;
fs/ocfs2/dlm/dlmmaster.c +40 -81
index 3e59ff9..4499d86 100644
@@ -2396,8 +2396,7 @@ static int dlm_is_lockres_migrateable(struct dlm_ctxt *dlm,
 
 
 static int dlm_migrate_lockres(struct dlm_ctxt *dlm,
-			       struct dlm_lock_resource *res,
-			       u8 target)
+			       struct dlm_lock_resource *res, u8 target)
 {
 	struct dlm_master_list_entry *mle = NULL;
 	struct dlm_master_list_entry *oldmle = NULL;
@@ -2411,25 +2410,15 @@ static int dlm_migrate_lockres(struct dlm_ctxt *dlm,
 	if (!dlm_grab(dlm))
 		return -EINVAL;
 
+	BUG_ON(target == O2NM_MAX_NODES);
+
 	name = res->lockname.name;
 	namelen = res->lockname.len;
 
-	mlog(0, "%s: Migrating %.*s to %u\n", dlm->name, namelen, name, target);
-
-	/* Ensure this lockres is a proper candidate for migration */
-	spin_lock(&res->spinlock);
-	ret = dlm_is_lockres_migrateable(dlm, res);
-	spin_unlock(&res->spinlock);
-
-	/* No work to do */
-	if (!ret)
-		goto leave;
-
-	/*
-	 * preallocate up front
-	 * if this fails, abort
-	 */
+	mlog(0, "%s: Migrating %.*s to node %u\n", dlm->name, namelen, name,
+	     target);
 
+	/* preallocate up front. if this fails, abort */
 	ret = -ENOMEM;
 	mres = (struct dlm_migratable_lockres *) __get_free_page(GFP_NOFS);
 	if (!mres) {
@@ -2445,35 +2434,10 @@ static int dlm_migrate_lockres(struct dlm_ctxt *dlm,
 	ret = 0;
 
 	/*
-	 * find a node to migrate the lockres to
-	 */
-
-	spin_lock(&dlm->spinlock);
-	/* pick a new node */
-	if (!test_bit(target, dlm->domain_map) ||
-	    target >= O2NM_MAX_NODES) {
-		target = dlm_pick_migration_target(dlm, res);
-	}
-	mlog(0, "%s: res %.*s, Node %u chosen for migration\n", dlm->name,
-	     namelen, name, target);
-
-	if (target >= O2NM_MAX_NODES ||
-	    !test_bit(target, dlm->domain_map)) {
-		/* target chosen is not alive */
-		ret = -EINVAL;
-	}
-
-	if (ret) {
-		spin_unlock(&dlm->spinlock);
-		goto fail;
-	}
-
-	mlog(0, "continuing with target = %u\n", target);
-
-	/*
 	 * clear any existing master requests and
 	 * add the migration mle to the list
 	 */
+	spin_lock(&dlm->spinlock);
 	spin_lock(&dlm->master_lock);
 	ret = dlm_add_migration_mle(dlm, res, mle, &oldmle, name,
 				    namelen, target, dlm->node_num);
@@ -2514,6 +2478,7 @@ fail:
 			dlm_put_mle(mle);
 		} else if (mle) {
 			kmem_cache_free(dlm_mle_cache, mle);
+			mle = NULL;
 		}
 		goto leave;
 	}
@@ -2632,7 +2597,6 @@ leave:
 	if (wake)
 		wake_up(&res->wq);
 
-	/* TODO: cleanup */
 	if (mres)
 		free_page((unsigned long)mres);
 
@@ -2657,28 +2621,28 @@ leave:
  */
 int dlm_empty_lockres(struct dlm_ctxt *dlm, struct dlm_lock_resource *res)
 {
-	int mig, ret;
+	int ret;
 	int lock_dropped = 0;
+	u8 target = O2NM_MAX_NODES;
 
 	assert_spin_locked(&dlm->spinlock);
 
 	spin_lock(&res->spinlock);
-	mig = dlm_is_lockres_migrateable(dlm, res);
+	if (dlm_is_lockres_migrateable(dlm, res))
+		target = dlm_pick_migration_target(dlm, res);
 	spin_unlock(&res->spinlock);
-	if (!mig)
+
+	if (target == O2NM_MAX_NODES)
 		goto leave;
 
 	/* Wheee! Migrate lockres here! Will sleep so drop spinlock. */
 	spin_unlock(&dlm->spinlock);
 	lock_dropped = 1;
-	while (1) {
-		ret = dlm_migrate_lockres(dlm, res, O2NM_MAX_NODES);
-		if (ret >= 0)
-			break;
-		mlog(0, "%s: res %.*s, Migrate failed, retrying\n", dlm->name,
-		     res->lockname.len, res->lockname.name);
-		msleep(DLM_MIGRATION_RETRY_MS);
-	}
+	ret = dlm_migrate_lockres(dlm, res, target);
+	if (ret)
+		mlog(0, "%s: res %.*s, Migrate to node %u failed with %d\n",
+		     dlm->name, res->lockname.len, res->lockname.name,
+		     target, ret);
 	spin_lock(&dlm->spinlock);
 leave:
 	return lock_dropped;
@@ -2862,61 +2826,58 @@ static void dlm_remove_nonlocal_locks(struct dlm_ctxt *dlm,
 	}
 }
 
-/* for now this is not too intelligent.  we will
- * need stats to make this do the right thing.
- * this just finds the first lock on one of the
- * queues and uses that node as the target. */
+/*
+ * Pick a node to migrate the lock resource to. This function selects a
+ * potential target based first on the locks and then on refmap. It skips
+ * nodes that are in the process of exiting the domain.
+ */
 static u8 dlm_pick_migration_target(struct dlm_ctxt *dlm,
 				    struct dlm_lock_resource *res)
 {
-	int i;
+	enum dlm_lockres_list idx;
 	struct list_head *queue = &res->granted;
 	struct dlm_lock *lock;
 	int nodenum;
 
 	assert_spin_locked(&dlm->spinlock);
+	assert_spin_locked(&res->spinlock);
 
-	spin_lock(&res->spinlock);
-	for (i=0; i<3; i++) {
+	/* Go through all the locks */
+	for (idx = DLM_GRANTED_LIST; idx <= DLM_BLOCKED_LIST; idx++) {
+		queue = dlm_list_idx_to_ptr(res, idx);
 		list_for_each_entry(lock, queue, list) {
-			/* up to the caller to make sure this node
-			 * is alive */
-			if (lock->ml.node != dlm->node_num) {
-				spin_unlock(&res->spinlock);
-				return lock->ml.node;
-			}
+			if (lock->ml.node == dlm->node_num)
+				continue;
+			if (test_bit(lock->ml.node, dlm->exit_domain_map))
+				continue;
+			if (!test_bit(lock->ml.node, dlm->domain_map))
+				continue;
+			nodenum = lock->ml.node;
+			goto bail;
 		}
-		queue++;
-	}
-
-	nodenum = find_next_bit(res->refmap, O2NM_MAX_NODES, 0);
-	if (nodenum < O2NM_MAX_NODES) {
-		spin_unlock(&res->spinlock);
-		return nodenum;
 	}
-	spin_unlock(&res->spinlock);
-	mlog(0, "have not found a suitable target yet! checking domain map\n");
 
-	/* ok now we're getting desperate.  pick anyone alive. */
+	/* Go thru the refmap */
 	nodenum = -1;
 	while (1) {
-		nodenum = find_next_bit(dlm->domain_map,
-					O2NM_MAX_NODES, nodenum+1);
-		mlog(0, "found %d in domain map\n", nodenum);
+		nodenum = find_next_bit(res->refmap, O2NM_MAX_NODES,
+					nodenum + 1);
 		if (nodenum >= O2NM_MAX_NODES)
 			break;

fs/ocfs2/dlm/dlmmaster.c static void dlm_remove_nonlocal_locks(struct dlm_ctxt *dlm, · line 2872

Quoted 10 lines
fs/ocfs2/dlm/dlmmaster.c +6 -4
-		if (nodenum != dlm->node_num) {
-			mlog(0, "picking %d\n", nodenum);
-			return nodenum;
-		}
+		if (nodenum == dlm->node_num)
+			continue;
+		if (test_bit(nodenum, dlm->exit_domain_map))
+			continue;
+		if (!test_bit(lock->ml.node, dlm->domain_map))
+			continue;

If the lock's owning node isn't in the domain map, we're just ignoring it? I guess I'm not following what the last 'if (!test_bit(lock->ml.node, dlm->domain_map))' line is trying to do.

	--Mark

PS: I'm rusty on fs/ocfs2/dlm stuff as you can tell :)

--
Mark Fasheh

#5 lore raw

PATCH 3/3 ocfs2/dlm: Do not migrate resource to a node that is leaving the domain

Quoted 25 lines · Mark Fasheh
Quoted 11 lines · Sunil Mushran
fs/ocfs2/dlm/dlmmaster.c +3 -4
-	/* ok now we're getting desperate.  pick anyone alive. */
+	/* Go thru the refmap */
  	nodenum = -1;
  	while (1) {
-		nodenum = find_next_bit(dlm->domain_map,
-					O2NM_MAX_NODES, nodenum+1);
-		mlog(0, "found %d in domain map\n", nodenum);
+		nodenum = find_next_bit(res->refmap, O2NM_MAX_NODES,
+					nodenum + 1);
  		if (nodenum>= O2NM_MAX_NODES)
  			break;

fs/ocfs2/dlm/dlmmaster.c static void dlm_remove_nonlocal_locks(struct dlm_ctxt *dlm, · line 2872

Quoted 10 lines
fs/ocfs2/dlm/dlmmaster.c +6 -4
-		if (nodenum != dlm->node_num) {
-			mlog(0, "picking %d\n", nodenum);
-			return nodenum;
-		}
+		if (nodenum == dlm->node_num)
+			continue;
+		if (test_bit(nodenum, dlm->exit_domain_map))
+			continue;
+		if (!test_bit(lock->ml.node, dlm->domain_map))
+			continue;

If the lock's owning node isn't in the domain map, we're just ignoring it? I guess I'm not following what the last 'if (!test_bit(lock->ml.node, dlm->domain_map))' line is trying to do.

Quoted 3 lines
	--Mark

PS: I'm rusty on fs/ocfs2/dlm stuff as you can tell :)

oops... that's a typo. It should be nodenum. Hey, you do remember o2dlm. ;)

That check is superfluous. As in, that nodenum should always be in the domain_map. But we've had that check from before and I'm trying not to rock the boat too much.

#6 lore raw

PATCH 3/3 ocfs2/dlm: Do not migrate resource to a node that is leaving the domain

Quoted 2 lines · Sunil Mushran

oops... that's a typo. It should be nodenum. Hey, you do remember o2dlm. ;)

Lol, glad to know I haven't totally lost it ;)

Quoted 3 lines

That check is superfluous. As in, that nodenum should always be in the domain_map. But we've had that check from before and I'm trying not to rock the boat too much.

Maybe we should warn on it then?

	--Mark

--
Mark Fasheh

#7 lore raw

PATCH 3/3 ocfs2/dlm: Do not migrate resource to a node that is leaving the domain

Quoted 10 lines · Mark Fasheh
Quoted 2 lines · Sunil Mushran

oops... that's a typo. It should be nodenum. Hey, you do remember o2dlm. ;)

Lol, glad to know I haven't totally lost it ;)

Quoted 3 lines

That check is superfluous. As in, that nodenum should always be in the domain_map. But we've had that check from before and I'm trying not to rock the boat too much.

Maybe we should warn on it then?

The more I think about it, the more removal makes sense. But I may need a shot of whiskey to fortify my resolve. ;)

#8 lore raw

PATCH 3/3 ocfs2/dlm: Do not migrate resource to a node that is leaving the domain

In reply to Sunil Mushran

Hi Sunil,

Your this series of patches fix a problem in mmap(and could also other) test in ocfs2-test.
The problem is that when all the three nodes do unmount at the same time, two of them keeps migrating a lockres to each other. So the umount on these two nodes hang there. I have idea why dlm_run_purge_list have no change to purge it. Your patches fix it :)

thanks,
wengang.

Quoted 285 lines · Sunil Mushran

During dlm domain shutdown, o2dlm has to free all the lock resources. Ones that have no locks and references are freed. Ones that have locks and/or references are migrated to another node.

The first task in migration is finding a target. Currently we scan the lock resource and find one node that either has a lock or a reference. This is not very efficient in a parallel umount case as we might end up migrating the lock resource to a node which itself may have to migrate it to a third node.

The patch scans the dlm->exit_domain_map to ensure the target node is not leaving the domain. If no valid target node is found, o2dlm does not migrate the resource but instead waits for the unlock and deref messages that will allow it to free the resource.

---
 fs/ocfs2/dlm/dlmdomain.c |   10 ++-
 fs/ocfs2/dlm/dlmmaster.c |  139 ++++++++++++++++-----------------------------
 2 files changed, 57 insertions(+), 92 deletions(-)
fs/ocfs2/dlm/dlmdomain.c +7 -3
index 3aff23f..6ed6b95 100644
@@ -451,14 +451,18 @@ redo_bucket:
 			dropped = dlm_empty_lockres(dlm, res);
 
 			spin_lock(&res->spinlock);
-			__dlm_lockres_calc_usage(dlm, res);
-			iter = res->hash_node.next;
+			if (dropped)
+				__dlm_lockres_calc_usage(dlm, res);
+			else
+				iter = res->hash_node.next;
 			spin_unlock(&res->spinlock);
 
 			dlm_lockres_put(res);
 
-			if (dropped)
+			if (dropped) {
+				cond_resched_lock(&dlm->spinlock);
 				goto redo_bucket;
+			}
 		}
 		cond_resched_lock(&dlm->spinlock);
 		num += n;
fs/ocfs2/dlm/dlmmaster.c +50 -89
index 3e59ff9..4499d86 100644
@@ -2396,8 +2396,7 @@ static int dlm_is_lockres_migrateable(struct dlm_ctxt *dlm,
 
 
 static int dlm_migrate_lockres(struct dlm_ctxt *dlm,
-			       struct dlm_lock_resource *res,
-			       u8 target)
+			       struct dlm_lock_resource *res, u8 target)
 {
 	struct dlm_master_list_entry *mle = NULL;
 	struct dlm_master_list_entry *oldmle = NULL;
@@ -2411,25 +2410,15 @@ static int dlm_migrate_lockres(struct dlm_ctxt *dlm,
 	if (!dlm_grab(dlm))
 		return -EINVAL;
 
+	BUG_ON(target == O2NM_MAX_NODES);
+
 	name = res->lockname.name;
 	namelen = res->lockname.len;
 
-	mlog(0, "%s: Migrating %.*s to %u\n", dlm->name, namelen, name, target);
-
-	/* Ensure this lockres is a proper candidate for migration */
-	spin_lock(&res->spinlock);
-	ret = dlm_is_lockres_migrateable(dlm, res);
-	spin_unlock(&res->spinlock);
-
-	/* No work to do */
-	if (!ret)
-		goto leave;
-
-	/*
-	 * preallocate up front
-	 * if this fails, abort
-	 */
+	mlog(0, "%s: Migrating %.*s to node %u\n", dlm->name, namelen, name,
+	     target);
 
+	/* preallocate up front. if this fails, abort */
 	ret = -ENOMEM;
 	mres = (struct dlm_migratable_lockres *) __get_free_page(GFP_NOFS);
 	if (!mres) {
@@ -2445,35 +2434,10 @@ static int dlm_migrate_lockres(struct dlm_ctxt *dlm,
 	ret = 0;
 
 	/*
-	 * find a node to migrate the lockres to
-	 */
-
-	spin_lock(&dlm->spinlock);
-	/* pick a new node */
-	if (!test_bit(target, dlm->domain_map) ||
-	    target >= O2NM_MAX_NODES) {
-		target = dlm_pick_migration_target(dlm, res);
-	}
-	mlog(0, "%s: res %.*s, Node %u chosen for migration\n", dlm->name,
-	     namelen, name, target);
-
-	if (target >= O2NM_MAX_NODES ||
-	    !test_bit(target, dlm->domain_map)) {
-		/* target chosen is not alive */
-		ret = -EINVAL;
-	}
-
-	if (ret) {
-		spin_unlock(&dlm->spinlock);
-		goto fail;
-	}
-
-	mlog(0, "continuing with target = %u\n", target);
-
-	/*
 	 * clear any existing master requests and
 	 * add the migration mle to the list
 	 */
+	spin_lock(&dlm->spinlock);
 	spin_lock(&dlm->master_lock);
 	ret = dlm_add_migration_mle(dlm, res, mle, &oldmle, name,
 				    namelen, target, dlm->node_num);
@@ -2514,6 +2478,7 @@ fail:
 			dlm_put_mle(mle);
 		} else if (mle) {
 			kmem_cache_free(dlm_mle_cache, mle);
+			mle = NULL;
 		}
 		goto leave;
 	}
@@ -2632,7 +2597,6 @@ leave:
 	if (wake)
 		wake_up(&res->wq);
 
-	/* TODO: cleanup */
 	if (mres)
 		free_page((unsigned long)mres);
 
@@ -2657,28 +2621,28 @@ leave:
  */
 int dlm_empty_lockres(struct dlm_ctxt *dlm, struct dlm_lock_resource *res)
 {
-	int mig, ret;
+	int ret;
 	int lock_dropped = 0;
+	u8 target = O2NM_MAX_NODES;
 
 	assert_spin_locked(&dlm->spinlock);
 
 	spin_lock(&res->spinlock);
-	mig = dlm_is_lockres_migrateable(dlm, res);
+	if (dlm_is_lockres_migrateable(dlm, res))
+		target = dlm_pick_migration_target(dlm, res);
 	spin_unlock(&res->spinlock);
-	if (!mig)
+
+	if (target == O2NM_MAX_NODES)
 		goto leave;
 
 	/* Wheee! Migrate lockres here! Will sleep so drop spinlock. */
 	spin_unlock(&dlm->spinlock);
 	lock_dropped = 1;
-	while (1) {
-		ret = dlm_migrate_lockres(dlm, res, O2NM_MAX_NODES);
-		if (ret >= 0)
-			break;
-		mlog(0, "%s: res %.*s, Migrate failed, retrying\n", dlm->name,
-		     res->lockname.len, res->lockname.name);
-		msleep(DLM_MIGRATION_RETRY_MS);
-	}
+	ret = dlm_migrate_lockres(dlm, res, target);
+	if (ret)
+		mlog(0, "%s: res %.*s, Migrate to node %u failed with %d\n",
+		     dlm->name, res->lockname.len, res->lockname.name,
+		     target, ret);
 	spin_lock(&dlm->spinlock);
 leave:
 	return lock_dropped;
@@ -2862,61 +2826,58 @@ static void dlm_remove_nonlocal_locks(struct dlm_ctxt *dlm,
 	}
 }
 
-/* for now this is not too intelligent.  we will
- * need stats to make this do the right thing.
- * this just finds the first lock on one of the
- * queues and uses that node as the target. */
+/*
+ * Pick a node to migrate the lock resource to. This function selects a
+ * potential target based first on the locks and then on refmap. It skips
+ * nodes that are in the process of exiting the domain.
+ */
 static u8 dlm_pick_migration_target(struct dlm_ctxt *dlm,
 				    struct dlm_lock_resource *res)
 {
-	int i;
+	enum dlm_lockres_list idx;
 	struct list_head *queue = &res->granted;
 	struct dlm_lock *lock;
 	int nodenum;
 
 	assert_spin_locked(&dlm->spinlock);
+	assert_spin_locked(&res->spinlock);
 
-	spin_lock(&res->spinlock);
-	for (i=0; i<3; i++) {
+	/* Go through all the locks */
+	for (idx = DLM_GRANTED_LIST; idx <= DLM_BLOCKED_LIST; idx++) {
+		queue = dlm_list_idx_to_ptr(res, idx);
 		list_for_each_entry(lock, queue, list) {
-			/* up to the caller to make sure this node
-			 * is alive */
-			if (lock->ml.node != dlm->node_num) {
-				spin_unlock(&res->spinlock);
-				return lock->ml.node;
-			}
+			if (lock->ml.node == dlm->node_num)
+				continue;
+			if (test_bit(lock->ml.node, dlm->exit_domain_map))
+				continue;
+			if (!test_bit(lock->ml.node, dlm->domain_map))
+				continue;
+			nodenum = lock->ml.node;
+			goto bail;
 		}
-		queue++;
-	}
-
-	nodenum = find_next_bit(res->refmap, O2NM_MAX_NODES, 0);
-	if (nodenum < O2NM_MAX_NODES) {
-		spin_unlock(&res->spinlock);
-		return nodenum;
 	}
-	spin_unlock(&res->spinlock);
-	mlog(0, "have not found a suitable target yet! checking domain map\n");
 
-	/* ok now we're getting desperate.  pick anyone alive. */
+	/* Go thru the refmap */
 	nodenum = -1;
 	while (1) {
-		nodenum = find_next_bit(dlm->domain_map,
-					O2NM_MAX_NODES, nodenum+1);
-		mlog(0, "found %d in domain map\n", nodenum);
+		nodenum = find_next_bit(res->refmap, O2NM_MAX_NODES,
+					nodenum + 1);
 		if (nodenum >= O2NM_MAX_NODES)
 			break;
-		if (nodenum != dlm->node_num) {
-			mlog(0, "picking %d\n", nodenum);
-			return nodenum;
-		}
+		if (nodenum == dlm->node_num)
+			continue;
+		if (test_bit(nodenum, dlm->exit_domain_map))
+			continue;
+		if (!test_bit(lock->ml.node, dlm->domain_map))
+			continue;
+		goto bail;
 	}
 
-	mlog(0, "giving up.  no master to migrate to\n");
-	return DLM_LOCK_RES_OWNER_UNKNOWN;
+	nodenum = O2NM_MAX_NODES;
+bail:
+	return nodenum;
 }
 
-
-
 /* this is called by the new master once all lockres
  * data has been received */
 static int dlm_do_migrate_request(struct dlm_ctxt *dlm,
Signature · 7 lines
1.7.1


_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-devel
#9 lore raw

PATCH 3/3 ocfs2/dlm: Do not migrate resource to a node that is leaving the domain

Yes, this series addresses that problem.

Quoted 298 lines · Wengang Wang

Hi Sunil,

Your this series of patches fix a problem in mmap(and could also other) test in ocfs2-test.
The problem is that when all the three nodes do unmount at the same time, two of them keeps migrating a lockres to each other. So the umount on these two nodes hang there. I have idea why dlm_run_purge_list have no change to purge it. Your patches fix it :)

thanks,
wengang.

Quoted 285 lines · Sunil Mushran

During dlm domain shutdown, o2dlm has to free all the lock resources. Ones that have no locks and references are freed. Ones that have locks and/or references are migrated to another node.

The first task in migration is finding a target. Currently we scan the lock resource and find one node that either has a lock or a reference. This is not very efficient in a parallel umount case as we might end up migrating the lock resource to a node which itself may have to migrate it to a third node.

The patch scans the dlm->exit_domain_map to ensure the target node is not leaving the domain. If no valid target node is found, o2dlm does not migrate the resource but instead waits for the unlock and deref messages that will allow it to free the resource.

---
  fs/ocfs2/dlm/dlmdomain.c |   10 ++-
  fs/ocfs2/dlm/dlmmaster.c |  139 ++++++++++++++++-----------------------------
  2 files changed, 57 insertions(+), 92 deletions(-)
fs/ocfs2/dlm/dlmdomain.c +7 -3
index 3aff23f..6ed6b95 100644
@@ -451,14 +451,18 @@ redo_bucket:
  			dropped = dlm_empty_lockres(dlm, res);
 
  			spin_lock(&res->spinlock);
-			__dlm_lockres_calc_usage(dlm, res);
-			iter = res->hash_node.next;
+			if (dropped)
+				__dlm_lockres_calc_usage(dlm, res);
+			else
+				iter = res->hash_node.next;
  			spin_unlock(&res->spinlock);
 
  			dlm_lockres_put(res);
 
-			if (dropped)
+			if (dropped) {
+				cond_resched_lock(&dlm->spinlock);
  				goto redo_bucket;
+			}
  		}
  		cond_resched_lock(&dlm->spinlock);
  		num += n;
fs/ocfs2/dlm/dlmmaster.c +50 -89
index 3e59ff9..4499d86 100644
@@ -2396,8 +2396,7 @@ static int dlm_is_lockres_migrateable(struct dlm_ctxt *dlm,
 
 
  static int dlm_migrate_lockres(struct dlm_ctxt *dlm,
-			       struct dlm_lock_resource *res,
-			       u8 target)
+			       struct dlm_lock_resource *res, u8 target)
  {
  	struct dlm_master_list_entry *mle = NULL;
  	struct dlm_master_list_entry *oldmle = NULL;
@@ -2411,25 +2410,15 @@ static int dlm_migrate_lockres(struct dlm_ctxt *dlm,
  	if (!dlm_grab(dlm))
  		return -EINVAL;
 
+	BUG_ON(target == O2NM_MAX_NODES);
+
  	name = res->lockname.name;
  	namelen = res->lockname.len;
 
-	mlog(0, "%s: Migrating %.*s to %u\n", dlm->name, namelen, name, target);
-
-	/* Ensure this lockres is a proper candidate for migration */
-	spin_lock(&res->spinlock);
-	ret = dlm_is_lockres_migrateable(dlm, res);
-	spin_unlock(&res->spinlock);
-
-	/* No work to do */
-	if (!ret)
-		goto leave;
-
-	/*
-	 * preallocate up front
-	 * if this fails, abort
-	 */
+	mlog(0, "%s: Migrating %.*s to node %u\n", dlm->name, namelen, name,
+	     target);
 
+	/* preallocate up front. if this fails, abort */
  	ret = -ENOMEM;
  	mres = (struct dlm_migratable_lockres *) __get_free_page(GFP_NOFS);
  	if (!mres) {
@@ -2445,35 +2434,10 @@ static int dlm_migrate_lockres(struct dlm_ctxt *dlm,
  	ret = 0;
 
  	/*
-	 * find a node to migrate the lockres to
-	 */
-
-	spin_lock(&dlm->spinlock);
-	/* pick a new node */
-	if (!test_bit(target, dlm->domain_map) ||
-	    target>= O2NM_MAX_NODES) {
-		target = dlm_pick_migration_target(dlm, res);
-	}
-	mlog(0, "%s: res %.*s, Node %u chosen for migration\n", dlm->name,
-	     namelen, name, target);
-
-	if (target>= O2NM_MAX_NODES ||
-	    !test_bit(target, dlm->domain_map)) {
-		/* target chosen is not alive */
-		ret = -EINVAL;
-	}
-
-	if (ret) {
-		spin_unlock(&dlm->spinlock);
-		goto fail;
-	}
-
-	mlog(0, "continuing with target = %u\n", target);
-
-	/*
  	 * clear any existing master requests and
  	 * add the migration mle to the list
  	 */
+	spin_lock(&dlm->spinlock);
  	spin_lock(&dlm->master_lock);
  	ret = dlm_add_migration_mle(dlm, res, mle,&oldmle, name,
  				    namelen, target, dlm->node_num);
@@ -2514,6 +2478,7 @@ fail:
  			dlm_put_mle(mle);
  		} else if (mle) {
  			kmem_cache_free(dlm_mle_cache, mle);
+			mle = NULL;
  		}
  		goto leave;
  	}
@@ -2632,7 +2597,6 @@ leave:
  	if (wake)
  		wake_up(&res->wq);
 
-	/* TODO: cleanup */
  	if (mres)
  		free_page((unsigned long)mres);
 
@@ -2657,28 +2621,28 @@ leave:
   */
  int dlm_empty_lockres(struct dlm_ctxt *dlm, struct dlm_lock_resource *res)
  {
-	int mig, ret;
+	int ret;
  	int lock_dropped = 0;
+	u8 target = O2NM_MAX_NODES;
 
  	assert_spin_locked(&dlm->spinlock);
 
  	spin_lock(&res->spinlock);
-	mig = dlm_is_lockres_migrateable(dlm, res);
+	if (dlm_is_lockres_migrateable(dlm, res))
+		target = dlm_pick_migration_target(dlm, res);
  	spin_unlock(&res->spinlock);
-	if (!mig)
+
+	if (target == O2NM_MAX_NODES)
  		goto leave;
 
  	/* Wheee! Migrate lockres here! Will sleep so drop spinlock. */
  	spin_unlock(&dlm->spinlock);
  	lock_dropped = 1;
-	while (1) {
-		ret = dlm_migrate_lockres(dlm, res, O2NM_MAX_NODES);
-		if (ret>= 0)
-			break;
-		mlog(0, "%s: res %.*s, Migrate failed, retrying\n", dlm->name,
-		     res->lockname.len, res->lockname.name);
-		msleep(DLM_MIGRATION_RETRY_MS);
-	}
+	ret = dlm_migrate_lockres(dlm, res, target);
+	if (ret)
+		mlog(0, "%s: res %.*s, Migrate to node %u failed with %d\n",
+		     dlm->name, res->lockname.len, res->lockname.name,
+		     target, ret);
  	spin_lock(&dlm->spinlock);
  leave:
  	return lock_dropped;
@@ -2862,61 +2826,58 @@ static void dlm_remove_nonlocal_locks(struct dlm_ctxt *dlm,
  	}
  }
 
-/* for now this is not too intelligent.  we will
- * need stats to make this do the right thing.
- * this just finds the first lock on one of the
- * queues and uses that node as the target. */
+/*
+ * Pick a node to migrate the lock resource to. This function selects a
+ * potential target based first on the locks and then on refmap. It skips
+ * nodes that are in the process of exiting the domain.
+ */
  static u8 dlm_pick_migration_target(struct dlm_ctxt *dlm,
  				    struct dlm_lock_resource *res)
  {
-	int i;
+	enum dlm_lockres_list idx;
  	struct list_head *queue =&res->granted;
  	struct dlm_lock *lock;
  	int nodenum;
 
  	assert_spin_locked(&dlm->spinlock);
+	assert_spin_locked(&res->spinlock);
 
-	spin_lock(&res->spinlock);
-	for (i=0; i<3; i++) {
+	/* Go through all the locks */
+	for (idx = DLM_GRANTED_LIST; idx<= DLM_BLOCKED_LIST; idx++) {
+		queue = dlm_list_idx_to_ptr(res, idx);
  		list_for_each_entry(lock, queue, list) {
-			/* up to the caller to make sure this node
-			 * is alive */
-			if (lock->ml.node != dlm->node_num) {
-				spin_unlock(&res->spinlock);
-				return lock->ml.node;
-			}
+			if (lock->ml.node == dlm->node_num)
+				continue;
+			if (test_bit(lock->ml.node, dlm->exit_domain_map))
+				continue;
+			if (!test_bit(lock->ml.node, dlm->domain_map))
+				continue;
+			nodenum = lock->ml.node;
+			goto bail;
  		}
-		queue++;
-	}
-
-	nodenum = find_next_bit(res->refmap, O2NM_MAX_NODES, 0);
-	if (nodenum<  O2NM_MAX_NODES) {
-		spin_unlock(&res->spinlock);
-		return nodenum;
  	}
-	spin_unlock(&res->spinlock);
-	mlog(0, "have not found a suitable target yet! checking domain map\n");
 
-	/* ok now we're getting desperate.  pick anyone alive. */
+	/* Go thru the refmap */
  	nodenum = -1;
  	while (1) {
-		nodenum = find_next_bit(dlm->domain_map,
-					O2NM_MAX_NODES, nodenum+1);
-		mlog(0, "found %d in domain map\n", nodenum);
+		nodenum = find_next_bit(res->refmap, O2NM_MAX_NODES,
+					nodenum + 1);
  		if (nodenum>= O2NM_MAX_NODES)
  			break;
-		if (nodenum != dlm->node_num) {
-			mlog(0, "picking %d\n", nodenum);
-			return nodenum;
-		}
+		if (nodenum == dlm->node_num)
+			continue;
+		if (test_bit(nodenum, dlm->exit_domain_map))
+			continue;
+		if (!test_bit(lock->ml.node, dlm->domain_map))
+			continue;
+		goto bail;
  	}
 
-	mlog(0, "giving up.  no master to migrate to\n");
-	return DLM_LOCK_RES_OWNER_UNKNOWN;
+	nodenum = O2NM_MAX_NODES;
+bail:
+	return nodenum;
  }
 
-
-
  /* this is called by the new master once all lockres
   * data has been received */
  static int dlm_do_migrate_request(struct dlm_ctxt *dlm,
Signature · 7 lines
1.7.1


_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-devel
#10 lore raw

PATCH 1/3 ocfs2/dlm: dlm_is_lockres_migrateable() returns boolean

In reply to Sunil Mushran
Quoted 113 lines · Sunil Mushran

Patch cleans up the gunk added by commit 388c4bcb4e63e88fb1f312a2f5f9eb2623afcf5b. dlm_is_lockres_migrateable() now returns 1 if lockresource is deemed migrateable and 0 if not.

---
 fs/ocfs2/dlm/dlmcommon.h |   12 ++++
 fs/ocfs2/dlm/dlmmaster.c |  135 +++++++++++++++++----------------------------
 2 files changed, 63 insertions(+), 84 deletions(-)
fs/ocfs2/dlm/dlmcommon.h +12 -0
index 4bdf7ba..1aac42a 100644
@@ -401,6 +401,18 @@ static inline int dlm_lvb_is_empty(char *lvb)
 	return 1;
 }
 
+static inline char *dlm_list_in_text(enum dlm_lockres_list idx)
+{
+	if (idx == DLM_GRANTED_LIST)
+		return "granted";
+	else if (idx == DLM_CONVERTING_LIST)
+		return "converting";
+	else if (idx == DLM_BLOCKED_LIST)
+		return "blocked";
+	else
+		return "unknown";
+}
+
 static inline struct list_head *
 dlm_list_idx_to_ptr(struct dlm_lock_resource *res, enum dlm_lockres_list idx)
 {
fs/ocfs2/dlm/dlmmaster.c +25 -34
index 9d67610..3e59ff9 100644
@@ -2339,65 +2339,55 @@ static void dlm_deref_lockres_worker(struct dlm_work_item *item, void *data)
 	dlm_lockres_put(res);
 }
 
-/* Checks whether the lockres can be migrated. Returns 0 if yes, < 0
- * if not. If 0, numlocks is set to the number of locks in the lockres.
+/*
+ * A migrateable resource is one that is :
+ * 1. locally mastered, and,
+ * 2. zero local locks, and,
+ * 3. one or more non-local locks, or, one or more references
+ * Returns 1 if yes, 0 if not.
  */
 static int dlm_is_lockres_migrateable(struct dlm_ctxt *dlm,
-				      struct dlm_lock_resource *res,
-				      int *numlocks,
-				      int *hasrefs)
+				      struct dlm_lock_resource *res)
 {
-	int ret;
-	int i;
-	int count = 0;
+	enum dlm_lockres_list idx;
+	int nonlocal = 0;
 	struct list_head *queue;
 	struct dlm_lock *lock;
+	u64 cookie;
 
 	assert_spin_locked(&res->spinlock);
 
-	*numlocks = 0;
-	*hasrefs = 0;
-
-	ret = -EINVAL;
-	if (res->owner == DLM_LOCK_RES_OWNER_UNKNOWN) {
-		mlog(0, "cannot migrate lockres with unknown owner!\n");
-		goto leave;
-	}
-
-	if (res->owner != dlm->node_num) {
-		mlog(0, "cannot migrate lockres this node doesn't own!\n");
-		goto leave;
-	}
+	if (res->owner != dlm->node_num)
+		return 0;
 
-	ret = 0;
-	queue = &res->granted;
-	for (i = 0; i < 3; i++) {
+        for (idx = DLM_GRANTED_LIST; idx <= DLM_BLOCKED_LIST; idx++) {
+		queue = dlm_list_idx_to_ptr(res, idx);
 		list_for_each_entry(lock, queue, list) {
-			++count;
-			if (lock->ml.node == dlm->node_num) {
-				mlog(0, "found a lock owned by this node still "
-				     "on the %s queue!  will not migrate this "
-				     "lockres\n", (i == 0 ? "granted" :
-						   (i == 1 ? "converting" :
-						    "blocked")));
-				ret = -ENOTEMPTY;
-				goto leave;
+			if (lock->ml.node != dlm->node_num) {
+				nonlocal++;
+				continue;
 			}
+			cookie = be64_to_cpu(lock->ml.cookie);
+			mlog(0, "%s: Not migrateable res %.*s, lock %u:%llu on "
+			     "%s list\n", dlm->name, res->lockname.len,
+			     res->lockname.name,
+			     dlm_get_lock_cookie_node(cookie),
+			     dlm_get_lock_cookie_seq(cookie),
+			     dlm_list_in_text(idx));
+			return 0;
 		}
-		queue++;
 	}

fs/ocfs2/dlm/dlmmaster.c static void dlm_deref_lockres_worker(struct dlm_work_item *item, void *data) · line 2384

Quoted 10 lines
fs/ocfs2/dlm/dlmmaster.c +4 -5
 
-	*numlocks = count;
-
-	count = find_next_bit(res->refmap, O2NM_MAX_NODES, 0);
-	if (count < O2NM_MAX_NODES)
-		*hasrefs = 1;
+	if (!nonlocal) {
+		nonlocal = find_next_bit(res->refmap, O2NM_MAX_NODES, 0);
+		if (nonlocal >= O2NM_MAX_NODES)
+			return 0;

Minor quibble, but can you use a new variable for these two lines? It took me a minute to realize that what you were doing was gettting a refcount on the lockres. Maybe:

if (!nonlocal) {
	/*
	 * We have no locks on the resource (local or remote). Check for
	 * references now.
	 */
	node_ref = find_next_bit(res->refmap, O2NM_MAX_NODES, 0);
	if (node_ref >= O2NM_MAX_NODES)
		return 0;
	}
}

Otherwise this looks much nicer than before, thanks.

	--Mark

--
Mark Fasheh

#11 lore raw

PATCH 1/3 ocfs2/dlm: dlm_is_lockres_migrateable() returns boolean

Quoted 15 lines · Mark Fasheh

fs/ocfs2/dlm/dlmmaster.c static void dlm_deref_lockres_worker(struct dlm_work_item *item, void *data) · line 2384

Quoted 10 lines · Sunil Mushran
fs/ocfs2/dlm/dlmmaster.c +4 -5
-	*numlocks = count;
-
-	count = find_next_bit(res->refmap, O2NM_MAX_NODES, 0);
-	if (count<  O2NM_MAX_NODES)
-		*hasrefs = 1;
+	if (!nonlocal) {
+		nonlocal = find_next_bit(res->refmap, O2NM_MAX_NODES, 0);
+		if (nonlocal>= O2NM_MAX_NODES)
+			return 0;

Minor quibble, but can you use a new variable for these two lines? It took me a minute to realize that what you were doing was gettting a refcount on the lockres. Maybe:

Quoted 10 lines
if (!nonlocal) {
	/*
	 * We have no locks on the resource (local or remote). Check for
	 * references now.
	 */
	node_ref = find_next_bit(res->refmap, O2NM_MAX_NODES, 0);
	if (node_ref>= O2NM_MAX_NODES)
		return 0;
	}
}

ok.

maintainer
#12 lore raw

PATCH 1/3 ocfs2/dlm: dlm_is_lockres_migrateable() returns boolean

Quoted 1 line · Sunil Mushran

ok.

I don't have your original email to reply to, but I got it off of your patch directory. It's now in the merge-window branch of ocfs2.git.

Joel

Signature · 7 lines
"When I am working on a problem I never think about beauty. I
 only think about how to solve the problem. But when I have finished, if
 the solution is not beautiful, I know it is wrong."
         - Buckminster Fuller

			http://www.jlbec.org/
			jlbec at evilplan.org