Skip to contentSkip to conversation

Making "verbose" mode a little less verbose

PATCH Leonardo Chiquitto 3 participants 5 messages Jul 6, 2010 – Jul 23, 2010 Kernel automounter

lore
Share & tools
lore
#0 lore raw

Hello AutoFS developers,

I'm trying to improve the usefulness of "verbose" logs for a customer. The problem they currently face can be summarized as: they need some messages that are only logged with DEFAULT_LOGGING="verbose" (or debug), but enabling the verbose mode on busy servers (hundreds of users and AutoFS mount points) causes a significant traffic increase on their syslog infrastructure.

My first idea was to introduce a new log level ("normal") and move the most important messages from "verbose" to "normal", but I realized later that this would require some redesign to allow some info() messages to be printed only on "verbose" mode. Since this would be more intrusive, I decided to go with a simpler proposal: move some "verbose" messages to "debug" level. Here are some examples of messages that would be demoted to debug (prefixed with -):

Successful mount:

14:49:13 n1 automount[1013]: attempting to mount entry /v/temp
-14:49:13 n1 automount[1013]: mount(nfs): mounted libre:/temp on /v/temp
14:49:13 n1 automount[1013]: mounted /v/temp

Invalid map:

14:51:15 n1 automount[1013]: attempting to mount entry /v/no
-14:51:15 n1 automount[1013]: key "no" not found in map source(s).
14:51:15 n1 automount[1013]: failed to mount /v/no

Expiring mount:

15:00:23 lotus automount[1013]: expiring path /v/temp
-15:00:23 lotus automount[1013]: unmounting dir = /v/temp
15:00:23 lotus automount[1013]: expired /v/temp

Please, could you consider the patch below for upstream inclusion?

Thanks,
Leonardo

daemon/lookup.c +3 -3
Index: autofs/daemon/lookup.c
===================================================================
@@ -688,7 +688,7 @@ static int lookup_name_file_source_insta
 	char *type, *format;
 
 	if (stat(map->argv[0], &st) == -1) {
-		warn(ap->logopt, "file map not found");
+		debug(ap->logopt, "file map not found");
 		return NSS_STATUS_NOTFOUND;
 	}
 
@@ -826,8 +826,8 @@ static void update_negative_cache(struct
 		 */
 		cache_unlock(me->mc);
 	else {
-		/* Notify only once after fail */
-		logmsg("key \"%s\" not found in map source(s).", name);
+		debug(ap->logopt,
+		      "key \"%s\" not found in map source(s).", name);
 
 		/* Doesn't exist in any source, just add it somewhere */
 		if (source)
daemon/automount.c +1 -1
Index: autofs/daemon/automount.c
===================================================================
@@ -512,7 +512,7 @@ static int umount_subtree_mounts(struct
 	 * it already to ensure it's ok to remove any offset triggers.
 	 */
 	if (!is_mm_root && is_mounted(_PATH_MOUNTED, path, MNTS_REAL)) {
-		info(ap->logopt, "unmounting dir = %s", path);
+		debug(ap->logopt, "unmounting dir = %s", path);
 		if (umount_ent(ap, path)) {
 			warn(ap->logopt, "could not umount dir %s", path);
 			left++;
modules/mount_changer.c +1 -1
Index: autofs/modules/mount_changer.c
===================================================================
@@ -129,7 +129,7 @@ int mount_mount(struct autofs_point *ap,
 
 		return 1;
 	} else {
-		info(ap->logopt, MODPREFIX "mounted %s type %s on %s",
+		debug(ap->logopt, MODPREFIX "mounted %s type %s on %s",
 		    what, fstype, fullpath);
 		return 0;
 	}
modules/mount_ext2.c +1 -1
Index: autofs/modules/mount_ext2.c
===================================================================
@@ -140,7 +140,7 @@ int mount_mount(struct autofs_point *ap,
 
 		return 1;
 	} else {
-		info(ap->logopt,
+		debug(ap->logopt,
 		      MODPREFIX "mounted %s type %s on %s",
 		      what, fstype, fullpath);
 		return 0;
modules/mount_generic.c +1 -1
Index: autofs/modules/mount_generic.c
===================================================================
@@ -122,7 +122,7 @@ int mount_mount(struct autofs_point *ap,
 
 		return 1;
 	} else {
-		info(ap->logopt, MODPREFIX "mounted %s type %s on %s",
+		debug(ap->logopt, MODPREFIX "mounted %s type %s on %s",
 		     loc, fstype, fullpath);
 		free(loc);
 		return 0;
modules/mount_nfs.c +1 -1
Index: autofs/modules/mount_nfs.c
===================================================================
@@ -251,7 +251,7 @@ int mount_mount(struct autofs_point *ap,
 		}
 
 		if (!err) {
-			info(ap->logopt, MODPREFIX "mounted %s on %s", loc, fullpath);
+			debug(ap->logopt, MODPREFIX "mounted %s on %s", loc, fullpath);
 			free(loc);
 			free_host_list(&hosts);
 			return 0;
#1 lore raw
Quoted 9 lines · Leonardo Chiquitto

Hello AutoFS developers,

I'm trying to improve the usefulness of "verbose" logs for a customer. The problem they currently face can be summarized as: they need some messages that are only logged with DEFAULT_LOGGING="verbose" (or debug), but enabling the verbose mode on busy servers (hundreds of users and AutoFS mount points) causes a significant traffic increase on their syslog infrastructure.

Quoted 7 lines

My first idea was to introduce a new log level ("normal") and move the most important messages from "verbose" to "normal", but I realized later that this would require some redesign to allow some info() messages to be printed only on "verbose" mode. Since this would be more intrusive, I decided to go with a simpler proposal: move some "verbose" messages to "debug" level. Here are some examples of messages that would be demoted to debug (prefixed with -):

Hi Ian and Leonardo,

I am seeing a similar issue with my clients as well. Autofs logs quite a bit of data and I would like to reduce it. Turning off logging isn't an option since it is useful for debugging sporadic issues.

Would this patch be considered? Is there some other way to reduce logging while still retaining enough to debug problems ( where running in debug mode isn't possible )?

Michael

maintainer
#2 lore raw
In reply to Leonardo Chiquitto
Quoted 9 lines · Leonardo Chiquitto

Hello AutoFS developers,

I'm trying to improve the usefulness of "verbose" logs for a customer. The problem they currently face can be summarized as: they need some messages that are only logged with DEFAULT_LOGGING="verbose" (or debug), but enabling the verbose mode on busy servers (hundreds of users and AutoFS mount points) causes a significant traffic increase on their syslog infrastructure.

Quoted 7 lines

My first idea was to introduce a new log level ("normal") and move the most important messages from "verbose" to "normal", but I realized later that this would require some redesign to allow some info() messages to be printed only on "verbose" mode. Since this would be more intrusive, I decided to go with a simpler proposal: move some "verbose" messages to "debug" level. Here are some examples of messages that would be demoted to debug (prefixed with -):

Leonardo and Michael, as you've probably guessed, I've been avoiding replying to this for a while now.

The problem is that there are two different views on this, one saying there isn't enough logging and others saying there is too much, *sigh*.

Quoted 6 lines

Successful mount:

14:49:13 n1 automount[1013]: attempting to mount entry /v/temp
-14:49:13 n1 automount[1013]: mount(nfs): mounted libre:/temp on /v/temp
14:49:13 n1 automount[1013]: mounted /v/temp

This is a good opportunity to eliminate some extra logging, good.

Quoted 6 lines

Invalid map:

14:51:15 n1 automount[1013]: attempting to mount entry /v/no
-14:51:15 n1 automount[1013]: key "no" not found in map source(s).
14:51:15 n1 automount[1013]: failed to mount /v/no

This is the problem for me.

This annoying constant key not found logging was the result of a customer complaining that they couldn't work out if a problem was due to a non-existent key or some other issue, so I'd rather not remove it. It should only be logged once on the first lookup of a non-existent key and once again after the negative cache entry times out. There was however a bug with the negative caching of non-existent keys, so we would need to check that the source in use is up to date wrt. this issue, and that it is working the way it is supposed to and then see how the logging goes.

Quoted 6 lines

Expiring mount:

15:00:23 lotus automount[1013]: expiring path /v/temp
-15:00:23 lotus automount[1013]: unmounting dir = /v/temp
15:00:23 lotus automount[1013]: expired /v/temp

And this is probably an extra entry we can do without, so probably good.

Quoted 100 lines

Please, could you consider the patch below for upstream inclusion?

Thanks,
Leonardo

Index: autofs/daemon/lookup.c
===================================================================

daemon/lookup.c +3 -3
@@ -688,7 +688,7 @@ static int lookup_name_file_source_insta
 	char *type, *format;
 
 	if (stat(map->argv[0], &st) == -1) {
-		warn(ap->logopt, "file map not found");
+		debug(ap->logopt, "file map not found");
 		return NSS_STATUS_NOTFOUND;
 	}
 
@@ -826,8 +826,8 @@ static void update_negative_cache(struct
 		 */
 		cache_unlock(me->mc);
 	else {
-		/* Notify only once after fail */
-		logmsg("key \"%s\" not found in map source(s).", name);
+		debug(ap->logopt,
+		      "key \"%s\" not found in map source(s).", name);
 
 		/* Doesn't exist in any source, just add it somewhere */
 		if (source)

Index: autofs/daemon/automount.c
===================================================================

daemon/automount.c +1 -1
@@ -512,7 +512,7 @@ static int umount_subtree_mounts(struct
 	 * it already to ensure it's ok to remove any offset triggers.
 	 */
 	if (!is_mm_root && is_mounted(_PATH_MOUNTED, path, MNTS_REAL)) {
-		info(ap->logopt, "unmounting dir = %s", path);
+		debug(ap->logopt, "unmounting dir = %s", path);
 		if (umount_ent(ap, path)) {
 			warn(ap->logopt, "could not umount dir %s", path);
 			left++;

Index: autofs/modules/mount_changer.c
===================================================================

modules/mount_changer.c +1 -1
@@ -129,7 +129,7 @@ int mount_mount(struct autofs_point *ap,
 
 		return 1;
 	} else {
-		info(ap->logopt, MODPREFIX "mounted %s type %s on %s",
+		debug(ap->logopt, MODPREFIX "mounted %s type %s on %s",
 		    what, fstype, fullpath);
 		return 0;
 	}

Index: autofs/modules/mount_ext2.c
===================================================================

modules/mount_ext2.c +1 -1
@@ -140,7 +140,7 @@ int mount_mount(struct autofs_point *ap,
 
 		return 1;
 	} else {
-		info(ap->logopt,
+		debug(ap->logopt,
 		      MODPREFIX "mounted %s type %s on %s",
 		      what, fstype, fullpath);
 		return 0;

Index: autofs/modules/mount_generic.c
===================================================================

modules/mount_generic.c +1 -1
@@ -122,7 +122,7 @@ int mount_mount(struct autofs_point *ap,
 
 		return 1;
 	} else {
-		info(ap->logopt, MODPREFIX "mounted %s type %s on %s",
+		debug(ap->logopt, MODPREFIX "mounted %s type %s on %s",
 		     loc, fstype, fullpath);
 		free(loc);
 		return 0;

Index: autofs/modules/mount_nfs.c
===================================================================

modules/mount_nfs.c +1 -1
@@ -251,7 +251,7 @@ int mount_mount(struct autofs_point *ap,
 		}
 
 		if (!err) {
-			info(ap->logopt, MODPREFIX "mounted %s on %s", loc, fullpath);
+			debug(ap->logopt, MODPREFIX "mounted %s on %s", loc, fullpath);
 			free(loc);
 			free_host_list(&hosts);
 			return 0;

_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs

#3 lore raw
Quoted 42 lines · Ian Kent
Quoted 9 lines · Leonardo Chiquitto

Hello AutoFS developers,

I'm trying to improve the usefulness of "verbose" logs for a customer. The problem they currently face can be summarized as: they need some messages that are only logged with DEFAULT_LOGGING="verbose" (or debug), but enabling the verbose mode on busy servers (hundreds of users and AutoFS mount points) causes a significant traffic increase on their syslog infrastructure.

Quoted 7 lines

My first idea was to introduce a new log level ("normal") and move the most important messages from "verbose" to "normal", but I realized later that this would require some redesign to allow some info() messages to be printed only on "verbose" mode. Since this would be more intrusive, I decided to go with a simpler proposal: move some "verbose" messages to "debug" level. Here are some examples of messages that would be demoted to debug (prefixed with -):

Leonardo and Michael, as you've probably guessed, I've been avoiding replying to this for a while now.

The problem is that there are two different views on this, one saying there isn't enough logging and others saying there is too much, *sigh*.

Quoted 6 lines

Successful mount:

14:49:13 n1 automount[1013]: attempting to mount entry /v/temp
-14:49:13 n1 automount[1013]: mount(nfs): mounted libre:/temp on /v/temp
14:49:13 n1 automount[1013]: mounted /v/temp

This is a good opportunity to eliminate some extra logging, good.

Quoted 6 lines

Invalid map:

14:51:15 n1 automount[1013]: attempting to mount entry /v/no
-14:51:15 n1 automount[1013]: key "no" not found in map source(s).
14:51:15 n1 automount[1013]: failed to mount /v/no

This is the problem for me.

Quoted 8 lines

This annoying constant key not found logging was the result of a customer complaining that they couldn't work out if a problem was due to a non-existent key or some other issue, so I'd rather not remove it. It should only be logged once on the first lookup of a non-existent key and once again after the negative cache entry times out. There was however a bug with the negative caching of non-existent keys, so we would need to check that the source in use is up to date wrt. this issue, and that it is working the way it is supposed to and then see how the logging goes.

Ian,

I believe the customer's complain makes sense. I missed this use case. Here's an updated patch without the "key not found" chunk.

Thanks,
Leonardo

Move some informational logs to debug level to avoid redundancy.

---
 daemon/lookup.c         |    2 +-
 daemon/automount.c      |    2 +-
 modules/mount_changer.c |    2 +-
 modules/mount_ext2.c    |    2 +-
 modules/mount_generic.c |    2 +-
 modules/mount_nfs.c     |    2 +-
 6 files changed, 6 insertions(+), 6 deletions(-)
daemon/lookup.c +1 -1
Index: autofs/daemon/lookup.c
===================================================================
@@ -688,7 +688,7 @@ static int lookup_name_file_source_insta
 	char *type, *format;
 
 	if (stat(map->argv[0], &st) == -1) {
-		warn(ap->logopt, "file map not found");
+		debug(ap->logopt, "file map not found");
 		return NSS_STATUS_NOTFOUND;
 	}
 
daemon/automount.c +1 -1
Index: autofs/daemon/automount.c
===================================================================
@@ -512,7 +512,7 @@ static int umount_subtree_mounts(struct
 	 * it already to ensure it's ok to remove any offset triggers.
 	 */
 	if (!is_mm_root && is_mounted(_PATH_MOUNTED, path, MNTS_REAL)) {
-		info(ap->logopt, "unmounting dir = %s", path);
+		debug(ap->logopt, "unmounting dir = %s", path);
 		if (umount_ent(ap, path)) {
 			warn(ap->logopt, "could not umount dir %s", path);
 			left++;
modules/mount_changer.c +1 -1
Index: autofs/modules/mount_changer.c
===================================================================
@@ -129,7 +129,7 @@ int mount_mount(struct autofs_point *ap,
 
 		return 1;
 	} else {
-		info(ap->logopt, MODPREFIX "mounted %s type %s on %s",
+		debug(ap->logopt, MODPREFIX "mounted %s type %s on %s",
 		    what, fstype, fullpath);
 		return 0;
 	}
modules/mount_ext2.c +1 -1
Index: autofs/modules/mount_ext2.c
===================================================================
@@ -140,7 +140,7 @@ int mount_mount(struct autofs_point *ap,
 
 		return 1;
 	} else {
-		info(ap->logopt,
+		debug(ap->logopt,
 		      MODPREFIX "mounted %s type %s on %s",
 		      what, fstype, fullpath);
 		return 0;
modules/mount_generic.c +1 -1
Index: autofs/modules/mount_generic.c
===================================================================
@@ -122,7 +122,7 @@ int mount_mount(struct autofs_point *ap,
 
 		return 1;
 	} else {
-		info(ap->logopt, MODPREFIX "mounted %s type %s on %s",
+		debug(ap->logopt, MODPREFIX "mounted %s type %s on %s",
 		     loc, fstype, fullpath);
 		free(loc);
 		return 0;
modules/mount_nfs.c +1 -1
Index: autofs/modules/mount_nfs.c
===================================================================
@@ -251,7 +251,7 @@ int mount_mount(struct autofs_point *ap,
 		}
 
 		if (!err) {
-			info(ap->logopt, MODPREFIX "mounted %s on %s", loc, fullpath);
+			debug(ap->logopt, MODPREFIX "mounted %s on %s", loc, fullpath);
 			free(loc);
 			free_host_list(&hosts);
 			return 0;
maintainer
#4 lore raw
Quoted 52 lines · Leonardo Chiquitto
Quoted 42 lines · Ian Kent
Quoted 9 lines · Leonardo Chiquitto

Hello AutoFS developers,

I'm trying to improve the usefulness of "verbose" logs for a customer. The problem they currently face can be summarized as: they need some messages that are only logged with DEFAULT_LOGGING="verbose" (or debug), but enabling the verbose mode on busy servers (hundreds of users and AutoFS mount points) causes a significant traffic increase on their syslog infrastructure.

Quoted 7 lines

My first idea was to introduce a new log level ("normal") and move the most important messages from "verbose" to "normal", but I realized later that this would require some redesign to allow some info() messages to be printed only on "verbose" mode. Since this would be more intrusive, I decided to go with a simpler proposal: move some "verbose" messages to "debug" level. Here are some examples of messages that would be demoted to debug (prefixed with -):

Leonardo and Michael, as you've probably guessed, I've been avoiding replying to this for a while now.

The problem is that there are two different views on this, one saying there isn't enough logging and others saying there is too much, *sigh*.

Quoted 6 lines

Successful mount:

14:49:13 n1 automount[1013]: attempting to mount entry /v/temp
-14:49:13 n1 automount[1013]: mount(nfs): mounted libre:/temp on /v/temp
14:49:13 n1 automount[1013]: mounted /v/temp

This is a good opportunity to eliminate some extra logging, good.

Quoted 6 lines

Invalid map:

14:51:15 n1 automount[1013]: attempting to mount entry /v/no
-14:51:15 n1 automount[1013]: key "no" not found in map source(s).
14:51:15 n1 automount[1013]: failed to mount /v/no

This is the problem for me.

Quoted 8 lines

This annoying constant key not found logging was the result of a customer complaining that they couldn't work out if a problem was due to a non-existent key or some other issue, so I'd rather not remove it. It should only be logged once on the first lookup of a non-existent key and once again after the negative cache entry times out. There was however a bug with the negative caching of non-existent keys, so we would need to check that the source in use is up to date wrt. this issue, and that it is working the way it is supposed to and then see how the logging goes.

Quoted 4 lines

Ian,

I believe the customer's complain makes sense. I missed this use case. Here's an updated patch without the "key not found" chunk.

Sure, I've had a quick look and it looks fine.
I'll get to it as soon as I can.

Quoted 94 lines

Thanks,
Leonardo

Move some informational logs to debug level to avoid redundancy.

---
 daemon/lookup.c         |    2 +-
 daemon/automount.c      |    2 +-
 modules/mount_changer.c |    2 +-
 modules/mount_ext2.c    |    2 +-
 modules/mount_generic.c |    2 +-
 modules/mount_nfs.c     |    2 +-
 6 files changed, 6 insertions(+), 6 deletions(-)

Index: autofs/daemon/lookup.c
===================================================================

daemon/lookup.c +1 -1
@@ -688,7 +688,7 @@ static int lookup_name_file_source_insta
 	char *type, *format;
 
 	if (stat(map->argv[0], &st) == -1) {
-		warn(ap->logopt, "file map not found");
+		debug(ap->logopt, "file map not found");
 		return NSS_STATUS_NOTFOUND;
 	}
 

Index: autofs/daemon/automount.c
===================================================================

daemon/automount.c +1 -1
@@ -512,7 +512,7 @@ static int umount_subtree_mounts(struct
 	 * it already to ensure it's ok to remove any offset triggers.
 	 */
 	if (!is_mm_root && is_mounted(_PATH_MOUNTED, path, MNTS_REAL)) {
-		info(ap->logopt, "unmounting dir = %s", path);
+		debug(ap->logopt, "unmounting dir = %s", path);
 		if (umount_ent(ap, path)) {
 			warn(ap->logopt, "could not umount dir %s", path);
 			left++;

Index: autofs/modules/mount_changer.c
===================================================================

modules/mount_changer.c +1 -1
@@ -129,7 +129,7 @@ int mount_mount(struct autofs_point *ap,
 
 		return 1;
 	} else {
-		info(ap->logopt, MODPREFIX "mounted %s type %s on %s",
+		debug(ap->logopt, MODPREFIX "mounted %s type %s on %s",
 		    what, fstype, fullpath);
 		return 0;
 	}

Index: autofs/modules/mount_ext2.c
===================================================================

modules/mount_ext2.c +1 -1
@@ -140,7 +140,7 @@ int mount_mount(struct autofs_point *ap,
 
 		return 1;
 	} else {
-		info(ap->logopt,
+		debug(ap->logopt,
 		      MODPREFIX "mounted %s type %s on %s",
 		      what, fstype, fullpath);
 		return 0;

Index: autofs/modules/mount_generic.c
===================================================================

modules/mount_generic.c +1 -1
@@ -122,7 +122,7 @@ int mount_mount(struct autofs_point *ap,
 
 		return 1;
 	} else {
-		info(ap->logopt, MODPREFIX "mounted %s type %s on %s",
+		debug(ap->logopt, MODPREFIX "mounted %s type %s on %s",
 		     loc, fstype, fullpath);
 		free(loc);
 		return 0;

Index: autofs/modules/mount_nfs.c
===================================================================

modules/mount_nfs.c +1 -1
@@ -251,7 +251,7 @@ int mount_mount(struct autofs_point *ap,
 		}
 
 		if (!err) {
-			info(ap->logopt, MODPREFIX "mounted %s on %s", loc, fullpath);
+			debug(ap->logopt, MODPREFIX "mounted %s on %s", loc, fullpath);
 			free(loc);
 			free_host_list(&hosts);
 			return 0;