Skip to contentSkip to conversation

OCFS2: allow the ocfs2 heartbeat thread to prioritize I/O

PATCH New Zhen Wei 2 participants 2 messages Dec 6, 2006 – Dec 7, 2006 OCFS2

lore
Share & tools
lore

Zhen Wei

author
#0 lore raw

PATCH 2/2 OCFS2: allow the ocfs2 heartbeat thread to prioritize I/O

From: Zhen Wei <[email protected]>
Subject: allow the ocfs2 heartbeat thread to prioritize I/O
Patch-mainline: ocfs2-1.2.2

The patch allows the ocfs2 heartbeat thread to prioritize I/O which may help cut down on spurious fencing. Most of this will be in the tools - we can have a pid configfs attribute and let userspace (ocfs2_hb_ctl) calls the ioprio_set syscall after starting heartbeat, but only cfq scheduler supports I/O priorities now.

zhen wei
[email protected]
+86 10 65339225
Novell, Inc.

-------------- next part --------------

ocfs2_err.et renamed from tmp/ocfs2-tools-1.2.2/libocfs2/ocfs2_err.et +3 -0
@@ -159,4 +159,7 @@ ec	OCFS2_ET_BLOCK_SIZE_TOO_SMALL_FOR_HAR
 ec	OCFS2_ET_INVALID_LOCKRES,
 	"The lock name is invalid"
 
+ec	OCFS2_ET_UNSUPPORTED_SYSCALL,
+	"Unsupported system call"
+
 	end
o2cb_abi.c renamed from tmp/ocfs2-tools-1.2.2/libo2cb/o2cb_abi.c +30 -0
@@ -1257,6 +1257,36 @@ void o2cb_free_nodes_list(char **nodes)
 	o2cb_free_dir_list(nodes);
 }
 
+errcode_t o2cb_get_hb_thread_pid (const char *cluster_name, const char *region_name,
+			   pid_t *pid)
+{
+	char attr_path[PATH_MAX];
+	char _fake_cluster_name[NAME_MAX];
+	char attr_value[16];
+	errcode_t ret;
+
+	if (!cluster_name) {
+		ret = _fake_default_cluster(_fake_cluster_name);
+		if (ret)
+			return ret;
+		cluster_name = _fake_cluster_name;
+	}
+
+	ret = snprintf(attr_path, PATH_MAX - 1,
+		       O2CB_FORMAT_HEARTBEAT_REGION_ATTR,
+		       configfs_path, cluster_name, region_name,
+		       "pid");
+
+	if ((ret <= 0) || (ret == (PATH_MAX - 1)))
+		return O2CB_ET_INTERNAL_FAILURE;
+
+	ret = o2cb_get_attribute(attr_path, attr_value, sizeof(attr_value) - 1);
+	if (ret == 0)
+		*pid = atoi (attr_value);
+
+	return ret;
+}
+
 errcode_t o2cb_get_node_num(const char *cluster_name, const char *node_name,
 			    uint16_t *node_num)
 {
ocfs2_hb_ctl.c renamed from tmp/ocfs2-tools-1.2.2/ocfs2_hb_ctl/ocfs2_hb_ctl.c +82 -2

156 lines not on this page · raw message

#1 lore raw

PATCH 2/2 OCFS2: allow the ocfs2 heartbeat thread to prioritize I/O

Hi Zwei,

Quoted 3 lines · Zhen Wei

From: Zhen Wei <[email protected]>
Subject: allow the ocfs2 heartbeat thread to prioritize I/O
Patch-mainline: ocfs2-1.2.2

Ok - the kernel patch looks good thus far.

Quoted 4 lines
o2cb_abi.c renamed from tmp/ocfs2-tools-1.2.2/libo2cb/o2cb_abi.c +4 -0
+errcode_t o2cb_get_hb_thread_pid (const char *cluster_name, const char *region_name,
+			   pid_t *pid)
+{
+	char attr_path[PATH_MAX];

o2cb_abi.c void o2cb_free_nodes_list(char **nodes) · line 1273

Quoted 10 lines
o2cb_abi.c renamed from tmp/ocfs2-tools-1.2.2/libo2cb/o2cb_abi.c +10 -0
+	char _fake_cluster_name[NAME_MAX];
+	char attr_value[16];
+	errcode_t ret;
+
+	if (!cluster_name) {
+		ret = _fake_default_cluster(_fake_cluster_name);
+		if (ret)
+			return ret;
+		cluster_name = _fake_cluster_name;
+	}

Hmm, I don't see any other test for cluster_name == NULL in similar code paths... Is there any particular reason why you added one?

Quoted 40 lines
ocfs2_hb_ctl.c renamed from tmp/ocfs2-tools-1.2.2/ocfs2_hb_ctl/ocfs2_hb_ctl.c +27 -0
@@ -29,6 +29,7 @@
 #define _GNU_SOURCE /* Because libc really doesn't want us using O_DIRECT? */
 
 #include <sys/types.h>
+#include <asm/unistd.h>
 #include <inttypes.h>
 
 #include <stdio.h>
@@ -47,18 +48,56 @@
 #define DEV_PREFIX      "/dev/"
 #define PROC_IDE_FORMAT "/proc/ide/%s/media"
 
+extern int ioprio_set(int, int, int);
+extern int ioprio_get(int, int);
+
+#if defined(__i386__)
+#define __NR_ioprio_set         289
+#define __NR_ioprio_get         290
+#elif defined(__ppc__)
+#define __NR_ioprio_set         273
+#define __NR_ioprio_get         274
+#elif defined(__x86_64__)
+#define __NR_ioprio_set         251
+#define __NR_ioprio_get         252
+#elif defined(__ia64__)
+#define __NR_ioprio_set         1274
+#define __NR_ioprio_get         1275
+#else
+#error "Unsupported arch"
+#endif
+
+_syscall3(int, ioprio_set, int, which, int, who, int, ioprio);
+_syscall2(int, ioprio_get, int, which, int, who);
+
+enum {
+        IOPRIO_CLASS_NONE,
+        IOPRIO_CLASS_RT,
+        IOPRIO_CLASS_BE,

ocfs2_hb_ctl.c line 86

Quoted 10 lines
ocfs2_hb_ctl.c renamed from tmp/ocfs2-tools-1.2.2/ocfs2_hb_ctl/ocfs2_hb_ctl.c +10 -0
+        IOPRIO_CLASS_IDLE,
+};
+
+enum {
+        IOPRIO_WHO_PROCESS = 1,
+        IOPRIO_WHO_PGRP,
+        IOPRIO_WHO_USER,
+};
+#define IOPRIO_CLASS_SHIFT      13
+

After looking at all this, and realizing that the proper syscall definitions don't exist anywhere on a modern distro, I'm begining to think that perhaps we want to just fork and exec /usr/bin/ionice.

It seems more future maintainable than defining the syscall interface ourselves.

ocfs2_hb_ctl.c int main(int argc, char **argv) · line 562

Quoted 5 lines
ocfs2_hb_ctl.c renamed from tmp/ocfs2-tools-1.2.2/ocfs2_hb_ctl/ocfs2_hb_ctl.c +5 -0
+	case HB_ACTION_IONICE:
+		err = adjust_priority(&hbo);
+		if (err) {
+			com_err(progname, err, "while adjusting heartbeat thread I/O priority");
+			ret = -EINVAL;

So, one thing I don't want to see is spurious errors if the set_ioprio or ionice stuff doesn't exist - this should work just fine without confusing users if they're still on an old kernel. You probably then want to filter out any errors pertaining to the existence (or not) of that api.

Otherwise most of your patch looks pretty reasonable. I guess a follow up patch would be to get mount.ocfs2.c to call ocfs2_hb_ctl to set a higher-than-the-default priority.

Thanks!

	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
[email protected]