OCFS2: allow the ocfs2 heartbeat thread to prioritize I/O
PATCH 2/2 OCFS2: allow the ocfs2 heartbeat thread to prioritize I/O
From: Zhen Wei <[email protected]>
Subject: allow the ocfs2 heartbeat thread to prioritize I/O
Patch-mainline: ocfs2-1.2.2
The patch allows the ocfs2 heartbeat thread to prioritize I/O which may help cut down on spurious fencing. Most of this will be in the tools - we can have a pid configfs attribute and let userspace (ocfs2_hb_ctl) calls the ioprio_set syscall after starting heartbeat, but only cfq scheduler supports I/O priorities now.
zhen wei
[email protected]
+86 10 65339225
Novell, Inc.
-------------- next part --------------
ocfs2_err.et renamed from tmp/ocfs2-tools-1.2.2/libocfs2/ocfs2_err.et +3 -0
@@ -159,4 +159,7 @@ ec OCFS2_ET_BLOCK_SIZE_TOO_SMALL_FOR_HAR ec OCFS2_ET_INVALID_LOCKRES, "The lock name is invalid" +ec OCFS2_ET_UNSUPPORTED_SYSCALL, + "Unsupported system call" + end
o2cb_abi.c renamed from tmp/ocfs2-tools-1.2.2/libo2cb/o2cb_abi.c +30 -0
@@ -1257,6 +1257,36 @@ void o2cb_free_nodes_list(char **nodes)o2cb_free_dir_list (nodes); } +errcode_t o2cb_get_hb_thread_pid (const char *cluster_name,const char *region_name, +pid_t *pid) +{ +char attr_path[PATH_MAX ]; +char _fake_cluster_name[NAME_MAX ]; +char attr_value[16 ]; +errcode_t ret; + +if (!cluster_name) { + ret =_fake_default_cluster (_fake_cluster_name); +if (ret) +return ret; + cluster_name = _fake_cluster_name; + } + + ret =snprintf (attr_path,PATH_MAX -1 , +O2CB_FORMAT_HEARTBEAT_REGION_ATTR , + configfs_path, cluster_name, region_name, +"pid" ); + +if ((ret <=0 ) || (ret == (PATH_MAX -1 ))) +return O2CB_ET_INTERNAL_FAILURE ; + + ret =o2cb_get_attribute (attr_path, attr_value,sizeof (attr_value) -1 ); +if (ret ==0 ) + *pid =atoi (attr_value); + +return ret; +} +errcode_t o2cb_get_node_num (const char *cluster_name,const char *node_name,uint16_t *node_num) {
ocfs2_hb_ctl.c renamed from tmp/ocfs2-tools-1.2.2/ocfs2_hb_ctl/ocfs2_hb_ctl.c +82 -2
156 lines not on this page · raw message
PATCH 2/2 OCFS2: allow the ocfs2 heartbeat thread to prioritize I/O
Hi Zwei,
Quoted 3 lines · Zhen Wei
From: Zhen Wei <[email protected]>
Subject: allow the ocfs2 heartbeat thread to prioritize I/O
Patch-mainline: ocfs2-1.2.2
Ok - the kernel patch looks good thus far.
Quoted 4 lines
o2cb_abi.c renamed from tmp/ocfs2-tools-1.2.2/libo2cb/o2cb_abi.c +4
-0+errcode_t o2cb_get_hb_thread_pid (const char *cluster_name,const char *region_name, +pid_t *pid) +{ +char attr_path[PATH_MAX ];
o2cb_abi.c void o2cb_free_nodes_list(char **nodes) · line 1273
Quoted 10 lines
o2cb_abi.c renamed from tmp/ocfs2-tools-1.2.2/libo2cb/o2cb_abi.c +10
-0+char _fake_cluster_name[NAME_MAX ]; +char attr_value[16 ]; +errcode_t ret; + +if (!cluster_name) { + ret =_fake_default_cluster (_fake_cluster_name); +if (ret) +return ret; + cluster_name = _fake_cluster_name; + }
Hmm, I don't see any other test for cluster_name == NULL in similar code paths... Is there any particular reason why you added one?
Quoted 40 lines
ocfs2_hb_ctl.c renamed from tmp/ocfs2-tools-1.2.2/ocfs2_hb_ctl/ocfs2_hb_ctl.c +27
-0@@ -29,6 +29,7 @@#define _GNU_SOURCE/* Because libc really doesn't want us using O_DIRECT? */ #include <sys/types.h> +#include <asm/unistd.h> #include <inttypes.h> #include <stdio.h> @@ -47,18 +48,56 @@#define DEV_PREFIX "/dev/"#define PROC_IDE_FORMAT "/proc/ide/%s/media" +extern int ioprio_set (int ,int ,int ); +extern int ioprio_get (int ,int ); + +#if defined(__i386__) +#define __NR_ioprio_set 289 +#define __NR_ioprio_get 290 +#elif defined(__ppc__) +#define __NR_ioprio_set 273 +#define __NR_ioprio_get 274 +#elif defined(__x86_64__) +#define __NR_ioprio_set 251 +#define __NR_ioprio_get 252 +#elif defined(__ia64__) +#define __NR_ioprio_set 1274 +#define __NR_ioprio_get 1275 +#else +#error "Unsupported arch" +#endif + +_syscall3 (int, ioprio_set, int, which, int, who, int, ioprio); +_syscall2 (int, ioprio_get, int, which, int, who); + +enum { +IOPRIO_CLASS_NONE , +IOPRIO_CLASS_RT , +IOPRIO_CLASS_BE ,
ocfs2_hb_ctl.c line 86
Quoted 10 lines
ocfs2_hb_ctl.c renamed from tmp/ocfs2-tools-1.2.2/ocfs2_hb_ctl/ocfs2_hb_ctl.c +10
-0+ IOPRIO_CLASS_IDLE, +}; + +enum { +IOPRIO_WHO_PROCESS =1 , +IOPRIO_WHO_PGRP , +IOPRIO_WHO_USER , +}; +#define IOPRIO_CLASS_SHIFT 13 +
After looking at all this, and realizing that the proper syscall definitions don't exist anywhere on a modern distro, I'm begining to think that perhaps we want to just fork and exec /usr/bin/ionice.
It seems more future maintainable than defining the syscall interface ourselves.
ocfs2_hb_ctl.c int main(int argc, char **argv) · line 562
Quoted 5 lines
ocfs2_hb_ctl.c renamed from tmp/ocfs2-tools-1.2.2/ocfs2_hb_ctl/ocfs2_hb_ctl.c +5
-0+case HB_ACTION_IONICE : + err =adjust_priority (&hbo); +if (err) { +com_err (progname, err,"while adjusting heartbeat thread I/O priority" ); + ret = -EINVAL ;
So, one thing I don't want to see is spurious errors if the set_ioprio or ionice stuff doesn't exist - this should work just fine without confusing users if they're still on an old kernel. You probably then want to filter out any errors pertaining to the existence (or not) of that api.
Otherwise most of your patch looks pretty reasonable. I guess a follow up patch would be to get mount.ocfs2.c to call ocfs2_hb_ctl to set a higher-than-the-default priority.
Thanks!
--Mark
--
Mark Fasheh
Senior Software Developer, Oracle
[email protected]