nodemanager, ocfs2, dlm

Some of the comments about the dlm concerned how it's configured (from user space.) In particular, there was interest in seeing the dlm and ocfs2 use common methods for their configuration.

The first area I'm looking at is how we get addresses/ids of other nodes. Currently, the dlm uses an ioctl on a misc device and ocfs2 uses a separate kernel module called "ocfs2_nodemanager" that's based on configfs.

I've taken a stab at generalizing ocfs2_nodemanager so the dlm could use it (removing ocfs-specific stuff). It still needs some work, but I'd like to know if this appeals to the ocfs group and to others who were interested in seeing some similarity in dlm/ocfs configuration.

The nodeid, I thought, was relative to a given DLM namespace, no? This concept seems to be missing here, or are you suggesting the nodeid to be global across namespaces?

Also, eventually we obviously need to have state for the nodes - up/down et cetera. I think the node manager also ought to track this.

How would kernel components use this and be notified about changes to the configuration / membership state?

Quoted 2 lines

Also, eventually we obviously need to have state for the nodes - up/down et cetera. I think the node manager also ought to track this.

We don't have a need for that information yet; I'm hoping we won't ever need it in the kernel, but we'll see.

There are at least two ways to handle this:

1. Pass cluster events and data into the kernel (this sounds like what you're talking about above), notify the effected kernel components, each kernel component takes the cluster data and does whatever it needs to with it (internal adjustments, recovery, etc).

2. Each kernel component "foo-kernel" has an associated user space component "foo-user". Cluster events (from userland clustering infrastructure) are passed to foo-user -- not into the kernel. foo-user determines what the specific consequences are for foo-kernel. foo-user then manipulates foo-kernel accordingly, through user/kernel hooks (sysfs, configfs, etc). These control hooks would largely be specific to foo.

We're following option 2 with the dlm and gfs and have been for quite a while, which means we don't need 1. I think ocfs2 is moving that way, too. Someone could still try 1, of course, but it would be of no use or interest to me. I'm not aware of any actual projects pushing forward with something like 1, so the persistent reference to it is somewhat baffling.

I've taken a stab at generalizing ocfs2_nodemanager so the dlm could use it (removing ocfs-specific stuff). It still needs some work, but I'd like to know if this appeals to the ocfs group and to others who were interested in seeing some similarity in dlm/ocfs configuration.

Some of the comments about the dlm concerned how it's configured (from user space.) In particular, there was interest in seeing the dlm and ocfs2 use common methods for their configuration.

The first area I'm looking at is how we get addresses/ids of other nodes.

I've taken a stab at generalizing ocfs2_nodemanager so the dlm could use it (removing ocfs-specific stuff). It still needs some work, but I'd like to know if this appeals to the ocfs group and to others who were interested in seeing some similarity in dlm/ocfs configuration.

drivers/nodemanager/nodemanager.c +13 -0

+static ssize_t node_local_write(struct node *node, const char *page,
+				size_t count)
+{
+	struct cluster *cluster = node_to_cluster(node);
+	unsigned long tmp;
+	char *p = (char *)page;
+
+	tmp = simple_strtoul(p, &p, 0);
+	if (!p || (*p && (*p != '\n')))
+		return -EINVAL;
+
+	tmp = !!tmp; /* boolean of whether this node wants to be local */
+

diff -urN a/drivers/nodemanager/nodemanager.h b/drivers/nodemanager/nodemanager.h

drivers/nodemanager/nodemanager.h +26 -0

@@ -0,0 +1,37 @@
+/*
+ * nodemanager.h
+ *
+ * Copyright (C) 2004 Oracle.  All rights reserved.
+ * Copyright (C) 2005 Red Hat, Inc.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ * 
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ * 
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ *
+ */
+
+#ifndef NODEMANAGER_H
+#define NODEMANAGER_H
+

For OCFS2 that would mean that an ocfs2_nodemanager would still exist, but as a much smaller module sitting on top of 'nodemanager'.

So no port attribute. The OCFS2 network code normally takes port from the node manager in order to determine how to talk to a given node. We'll have to figure out how to resolve that. The easiest would be to add 'port' back, but I think that might be problematic if we have multiple cluster network infrastructures as we do today.

Another way to handle this would be to have userspace symlink to the node items as an attribute on an ocfs2_tcp item. We could store 'port' as a second attribute. This would have the added benefit of pinning node information while OCFS2 uses it.

drivers/nodemanager/nodemanager.c line 185

Quoted 1 line

drivers/nodemanager/nodemanager.c +1 -0

+	char			nd_name[NODEMANAGER_MAX_NAME_LEN+1];

An accessor function for this would be nice for pretty prints - maybe strcpy into a passed string.

drivers/nodemanager/nodemanager.c line 186

Quoted 1 line

drivers/nodemanager/nodemanager.c +1 -0

+	int			nd_nodeid;

This definitely won't work with OCFS2... Nodeid (what used to be called node_num) needs to be unsigned. Otherwise this will break all our nodemap stuff which uses a bitmap to represent cluster state.

drivers/nodemanager/nodemanager.c line 192

Quoted 1 line

drivers/nodemanager/nodemanager.c +1 -0

+	struct list_head	nd_status_list;

What are these two for? They don't seem to be referenced elsewhere...

drivers/nodemanager/nodemanager.c line 442

Quoted 3 lines

drivers/nodemanager/nodemanager.c +3 -0

+	if (!tmp && cluster->cl_has_local &&
+	    cluster->cl_local_node == node->nd_nodeid) {
+		cluster->cl_local_node = 0;

I think we might want to be setting cl_local_node to NODEMANAGER_MAX_NODES here. It seems that ocfs2_nodemanager also does this so we might have just caught a bug you inherited :)

You removed o2nm_configured_node_map but we need some sort of method for enumerating over the set of configured nodes.

Also we need a method for querying the existence of a node.
The OCFS2 code usually uses o2nm_get_node_by_num(..) != NULL for this but a simple boolean api call would be cleaner and would avoid exposing the node structure.