Skip to contentSkip to conversation

Improve mirror DSO's failure logging

PATCH New Malahal Naineni 3 participants 5 messages Nov 20, 2009 – Nov 30, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Improve mirror DSO's failure logging

The mirror target has the following device states. The mirror DSO (daemons/dmeventd/plugins/mirror/dmeventd_mirror.c) doesn't know any of these states. This patchs adds these states to the DSO for better error reporting.

A => Alive - No failures
D => Dead - A write failure occurred leaving mirror out-of-sync S => Sync - A sychronization failure occurred, mirror out-of-sync R => Read - A read failure occurred, mirror data unaffected

daemons/dmeventd/plugins/mirror/dmeventd_mirror.c +36 -10
diff -r fff61ad560ad -r c107c082a3a5 daemons/dmeventd/plugins/mirror/dmeventd_mirror.c
@@ -28,9 +28,13 @@
 #include <syslog.h> /* FIXME Replace syslog with multilog */
 /* FIXME Missing openlog? */
 
-#define ME_IGNORE    0
-#define ME_INSYNC    1
-#define ME_FAILURE   2
+#define ME_IGNORE    			0
+#define ME_INSYNC    			1
+#define ME_SYNC_FAILURE   		2
+#define ME_LOG_FAILURE 			3
+#define ME_READ_FAILURE   		4
+#define ME_PRIMARY_WRITE_FAILURE	5
+#define ME_SECONDARY_WRITE_FAILURE	6
 
 /*
  * register_device() is called first and performs initialisation.
@@ -53,7 +57,7 @@ static pthread_mutex_t _event_mutex = PT
 
 static int _get_mirror_event(char *params)
 {
-	int i, r = ME_INSYNC;
+	int i, r;
 	char **args = NULL;
 	char *dev_status_str;
 	char *log_status_str;
@@ -89,22 +93,40 @@ static int _get_mirror_event(char *param
 	sync_str = args[num_devs];
 
 	/* Check for bad mirror devices */
-	for (i = 0; i < num_devs; i++)
-		if (dev_status_str[i] == 'D') {
+	r = -1;
+	for (i = 0; i < num_devs && r == -1; i++) {
+		switch (dev_status_str[i]) {
+		case 'D':
 			syslog(LOG_ERR, "Mirror device, %s, has failed.\n", args[i]);
-			r = ME_FAILURE;
+			if (i == 0)
+				r = ME_PRIMARY_WRITE_FAILURE;
+			else 
+				r = ME_SECONDARY_WRITE_FAILURE;
+			break;
+		case 'S':
+			syslog(LOG_ERR, "Mirror synchronization failed. "
+					"device, %s, failed.\n", args[i]);
+			r = ME_SYNC_FAILURE;
+			break;
+		case 'R':
+			syslog(LOG_ERR, "Mirror device, %s, read failed.\n",
+					args[i]);
+			r = ME_READ_FAILURE;
+			break;
 		}
+	}
 
 	/* Check for bad disk log device */
 	if (log_argc > 1 && log_status_str[0] == 'D') {
 		syslog(LOG_ERR, "Log device, %s, has failed.\n",
 		       args[2 + num_devs + log_argc]);
-		r = ME_FAILURE;
+		r = ME_LOG_FAILURE;
 	}
 
-	if (r == ME_FAILURE)
+	if (r != -1)	/* if a failure occurred */
 		goto out;
 
+	r = ME_INSYNC;	/* assume INSYNC event */
 	p = strstr(sync_str, "/");
 	if (p) {
 		p[0] = '\0';
@@ -210,7 +232,9 @@ void process_event(struct dm_task *dmt,
 			*/
 			syslog(LOG_NOTICE, "%s is now in-sync\n", device);
 			break;
-		case ME_FAILURE:
+		case ME_LOG_FAILURE:
+		case ME_PRIMARY_WRITE_FAILURE:
+		case ME_SECONDARY_WRITE_FAILURE:
 			syslog(LOG_ERR, "Device failure in %s\n", device);
 			if (_remove_failed_devices(device))
 				/* FIXME Why are all the error return codes unused? Get rid of them? */
@@ -222,6 +246,8 @@ void process_event(struct dm_task *dmt,
 					device);
 			*/
 			break;
+		case ME_READ_FAILURE: /* we just ignore for now */
+		case ME_SYNC_FAILURE:
 		case ME_IGNORE:
 			break;
 		default:
#1 lore raw

PATCH 1/1 Improve mirror DSO's failure logging

Quoted 8 lines · Malahal Naineni

The mirror target has the following device states. The mirror DSO (daemons/dmeventd/plugins/mirror/dmeventd_mirror.c) doesn't know any of these states. This patchs adds these states to the DSO for better error reporting.

A => Alive - No failures
D => Dead - A write failure occurred leaving mirror out-of-sync S => Sync - A sychronization failure occurred, mirror out-of-sync R => Read - A read failure occurred, mirror data unaffected

I think that the idea to improve an error reporting is very good.

Quoted 26 lines

static int _get_mirror_event(char *params)
{

daemons/dmeventd/plugins/mirror/dmeventd_mirror.c +12 -4
-	int i, r = ME_INSYNC;
+	int i, r;
 	char **args = NULL;
 	char *dev_status_str;
 	char *log_status_str;
@@ -89,22 +93,40 @@ static int _get_mirror_event(char *param
 	sync_str = args[num_devs];
 
 	/* Check for bad mirror devices */
-	for (i = 0; i < num_devs; i++)
-		if (dev_status_str[i] == 'D') {
+	r = -1;
+	for (i = 0; i < num_devs && r == -1; i++) {
+		switch (dev_status_str[i]) {
+		case 'D':
 			syslog(LOG_ERR, "Mirror device, %s, has failed.\n", args[i]);
-			r = ME_FAILURE;
+			if (i == 0)
+				r = ME_PRIMARY_WRITE_FAILURE;
+			else 
+				r = ME_SECONDARY_WRITE_FAILURE;
+			break;
+		case 'S':
+			syslog(LOG_ERR, "Mirror synchronization failed. "

daemons/dmeventd/plugins/mirror/dmeventd_mirror.c static int _get_mirror_event(char *param · line 117

Quoted 10 lines
daemons/dmeventd/plugins/mirror/dmeventd_mirror.c +9 -0
+					"device, %s, failed.\n", args[i]);
+			r = ME_SYNC_FAILURE;
+			break;
+		case 'R':
+			syslog(LOG_ERR, "Mirror device, %s, read failed.\n",
+					args[i]);
+			r = ME_READ_FAILURE;
+			break;
 		}
+	}

I'm afraid that only an error status of the device with the smallest index of the dev_status_str[] array will be reported. For example, if the status is "ARD," "r" will have a value, ME_READ_FAILURE. Then, a write failure event (ME_PRIMARY_WRITE_FAILURE and ME_SECONDARY_WRITE_FAILURE) won't be detected in process_event(). I think that "WRITE_FAILURE" events should have priority.

Thanks,
Taka

#2 lore raw

PATCH 1/1 Improve mirror DSO's failure logging

Quoted 12 lines · Takahiro Yasui

I'm afraid that only an error status of the device with the smallest index
of the dev_status_str[] array will be reported. For example, if the status
is "ARD," "r" will have a value, ME_READ_FAILURE. Then, a write failure
event (ME_PRIMARY_WRITE_FAILURE and ME_SECONDARY_WRITE_FAILURE) won't be
detected in process_event(). I think that "WRITE_FAILURE" events should
have priority.

Quoted 2 lines

Thanks,
Taka

Good catch Taka, I agree. Otherwise, I welcome this patch.

brassow

#3 lore raw

PATCH 1/1 Improve mirror DSO's failure logging

In reply to Takahiro Yasui
Quoted 6 lines · Takahiro Yasui [tyasui at redhat.com]

I'm afraid that only an error status of the device with the smallest index of the dev_status_str[] array will be reported. For example, if the status is "ARD," "r" will have a value, ME_READ_FAILURE. Then, a write failure event (ME_PRIMARY_WRITE_FAILURE and ME_SECONDARY_WRITE_FAILURE) won't be detected in process_event(). I think that "WRITE_FAILURE" events should have priority.

Thank you, Takahiro. Just posted a patch fixing the above issue. I used them as flags to fix it. Let me know what you think.

See
http://permalink.gmane.org/gmane.linux.kernel.device-mapper.devel/10675
for details.

Thanks, Malahal.

#4 lore raw

PATCH 1/1 Improve mirror DSO's failure logging

Quoted 8 lines · malahal at us.ibm.com
Quoted 6 lines · Takahiro Yasui [tyasui at redhat.com]

I'm afraid that only an error status of the device with the smallest index of the dev_status_str[] array will be reported. For example, if the status is "ARD," "r" will have a value, ME_READ_FAILURE. Then, a write failure event (ME_PRIMARY_WRITE_FAILURE and ME_SECONDARY_WRITE_FAILURE) won't be detected in process_event(). I think that "WRITE_FAILURE" events should have priority.

Quoted 6 lines

Thank you, Takahiro. Just posted a patch fixing the above issue. I used them as flags to fix it. Let me know what you think.

See
http://permalink.gmane.org/gmane.linux.kernel.device-mapper.devel/10675
for details.

I'm afraid that you need to refresh this patch set on top of the recent lvm2 source. The patch similar to your patch #1 has been merged recently.

Please see this link.
http://permalink.gmane.org/gmane.linux.lvm.devel/2312

Thanks,
Taka