Skip to contentSkip to conversation

printk: add console_msg_format command line option

PATCH Superseded Sergey Senozhatsky 4 participants 5 messages Dec 19, 2017 – Dec 22, 2017

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 printk: add console_msg_format command line option

0day and kernelCI automatically parse kernel log - basically some sort of grepping using the pre-defined text patterns - in order to detect and report regressions/errors. There are several sources they get the kernel logs from:

a) dmesg or /proc/ksmg

This is the preferred way. Because `dmesg --raw' (see later Note) and /proc/kmsg output contains facility and log level, which greatly simplifies grepping for EMERG/ALERT/CRIT/ERR messages.

b) serial consoles

This option is harder to maintain, because serial console messages don't contain facility and log level.

This patch introduces a `console_msg_format=' command line option, to switch between different message formatting on serial consoles. For the time being we have just two options - default and syslog. The "default" option just keeps the existing format. While the "syslog" option makes serial console messages to appear in syslog format [syslog() syscall], matching the dmesg -S --raw' and cat /proc/kmsg' output formats:

- facility and log level
- time stamp (depends on printk_time/PRINTK_TIME)
- message

	<%u>[time stamp] text\n

NOTE: while Kevin and Fengguang talk about "dmesg --raw", it's actually "dmesg -S --raw" that always prints messages in syslog format [per Petr Mladek]. Running "dmesg --raw" may produce output in non-syslog format sometimes. console_msg_format=syslog enables syslog format, thus in documentation we mention "dmesg -S --raw", not "dmesg --raw".

Per Kevin Hilman:

: Right now we can get this info from a "dmesg --raw" after bootup, : but it would be really nice in certain automation frameworks to : have a kernel command-line option to enable printing of loglevels : in default boot log.
:
: This is especially useful when ingesting kernel logs into advanced : search/analytics frameworks (I'm playing with and ELK stack: Elastic : Search, Logstash, Kibana).
:
: The other important reason for having this on the command line is that : for testing linux-next (and other bleeding edge developer branches), : it's common that we never make it to userspace, so can't even run : "dmesg --raw" (or equivalent.) So we really want this on the primary : boot (serial) console.

Per Fengguang Wu, 0day scripts should quickly benefit from that feature, because they will be able to switch to a more reliable parsing, based on messages' facility and log levels [1]:

`#{grep} -a -E -e '^<[0123]>' -e '^kern :(err |crit |alert |emerg )'

instead of doing text pattern matching

`#{grep} -a -F -f /lkp/printk-error-messages #{kmsg_file} |

      grep -a -v -E -f #{LKP_SRC}/etc/oops-pattern |
      grep -a -v -F -f #{LKP_SRC}/etc/kmsg-blacklist`

[1] https://github.com/fengguang/lkp-tests/blob/master/lib/dmesg.rb

---

v2: addressed comments made by Petr Mladek

 Documentation/admin-guide/kernel-parameters.txt | 14 ++++++++++++++
 kernel/printk/printk.c                          | 23 ++++++++++++++++++++++-
 2 files changed, 36 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +14 -0
index 8a61d1eda677..543de386e01e 100644
@@ -642,6 +642,20 @@
 			console=brl,ttyS0
 		For now, only VisioBraille is supported.
 
+	console_msg_format=
+			[KNL] Change console messages format
+		default
+			By default we print messages on consoles in
+			"[time stamp] text\n" format (time stamp may not be
+			printed, depending on CONFIG_PRINTK_TIME or
+			`printk_time' param).
+		syslog
+			Switch to syslog format: "<%u>[time stamp] text\n"
+			IOW, each message will have a facility and loglevel
+			prefix. The format is similar to one used by syslog()
+			syscall, or to executing "dmesg -S --raw" or to reading
+			from /proc/kmsg.
+
 	consoleblank=	[KNL] The console blank (screen saver) timeout in
 			seconds. A value of 0 disables the blank timer.
                        Defaults to 0.
kernel/printk/printk.c +22 -1
index c2e713f6ae2e..16dd91260273 100644
@@ -277,6 +277,13 @@ EXPORT_SYMBOL(console_set_on_cmdline);
 /* Flag: console code may call schedule() */
 static int console_may_schedule;
 
+enum con_msg_format {
+	MSG_FORMAT_DEFAULT	= 0,
+	MSG_FORMAT_SYSLOG	= (1 << 0),
+};
+
+static int console_msg_format = MSG_FORMAT_DEFAULT;
+
 /*
  * The printk log buffer consists of a chain of concatenated variable
  * length records. Every record starts with a record header, containing
@@ -1913,6 +1920,17 @@ static int __add_preferred_console(char *name, int idx, char *options,
 	c->index = idx;
 	return 0;
 }
+
+static int __init console_msg_format_setup(char *str)
+{
+	if (!strncmp(str, "syslog", 6))
+		console_msg_format = MSG_FORMAT_SYSLOG;
+	if (!strncmp(str, "default", 7))
+		console_msg_format = MSG_FORMAT_DEFAULT;
+	return 1;
+}
+__setup("console_msg_format=", console_msg_format_setup);
+
 /*
  * Set up a console.  Called via do_early_param() in init/main.c
  * for each "console=" parameter in the boot command line.
@@ -2215,7 +2233,10 @@ void console_unlock(void)
 			goto skip;
 		}
 
-		len += msg_print_text(msg, false, text + len, sizeof(text) - len);
+		len += msg_print_text(msg,
+				console_msg_format & MSG_FORMAT_SYSLOG,
+				text + len,
+				sizeof(text) - len);
 		if (nr_ext_console_drivers) {
 			ext_len = msg_print_ext_header(ext_text,
 						sizeof(ext_text),
Signature · 1 line
2.15.1
#1 lore raw
Quoted 11 lines · On Tue, 19 Dec 2017 17:42:43 +0900 Sergey Senozhatsky…
Documentation/admin-guide/kernel-parameters.txt +3 -0
index 8a61d1eda677..543de386e01e 100644
@@ -642,6 +642,20 @@
 			console=brl,ttyS0
 		For now, only VisioBraille is supported.
 
+	console_msg_format=
+			[KNL] Change console messages format
+		default

Documentation/admin-guide/kernel-parameters.txt line 657

Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +10 -0
+			By default we print messages on consoles in
+			"[time stamp] text\n" format (time stamp may not be
+			printed, depending on CONFIG_PRINTK_TIME or
+			`printk_time' param).
+		syslog
+			Switch to syslog format: "<%u>[time stamp] text\n"
+			IOW, each message will have a facility and loglevel
+			prefix. The format is similar to one used by syslog()
+			syscall, or to executing "dmesg -S --raw" or to reading
+			from /proc/kmsg.

Just to clarify. This affects all consoles, not just serial?

Quoted 9 lines
Documentation/admin-guide/kernel-parameters.txt +1 -0
+
 	consoleblank=	[KNL] The console blank (screen saver) timeout in
 			seconds. A value of 0 disables the blank timer.
                        Defaults to 0.
kernel/printk/printk.c +0 -0
index c2e713f6ae2e..16dd91260273 100644

kernel/printk/printk.c EXPORT_SYMBOL(console_set_on_cmdline); · line 283

Quoted 7 lines
kernel/printk/printk.c +4 -0
@@ -277,6 +277,13 @@ EXPORT_SYMBOL(console_set_on_cmdline);
 /* Flag: console code may call schedule() */
 static int console_may_schedule;
 
+enum con_msg_format {
+	MSG_FORMAT_DEFAULT	= 0,
+	MSG_FORMAT_SYSLOG	= (1 << 0),
+};

So the con_msg_format will be a flag (as denoted with the (1<<0)). Should we call it "con_msg_format_flags" to make it more obvious that these are treated as flags and not a simple number.

Quoted 7 lines
kernel/printk/printk.c +3 -0
+
+static int console_msg_format = MSG_FORMAT_DEFAULT;
+
 /*
  * The printk log buffer consists of a chain of concatenated variable
  * length records. Every record starts with a record header, containing

kernel/printk/printk.c static int __add_preferred_console(char *name, int idx, char *options, · line 1928

Quoted 9 lines
kernel/printk/printk.c +6 -0
@@ -1913,6 +1920,17 @@ static int __add_preferred_console(char *name, int idx, char *options,
 	c->index = idx;
 	return 0;
 }
+
+static int __init console_msg_format_setup(char *str)
+{
+	if (!strncmp(str, "syslog", 6))
+		console_msg_format = MSG_FORMAT_SYSLOG;
+	if (!strncmp(str, "default", 7))

You can use strcmp() instead of strncmp(), the init code will nul terminate the string for you before calling this function. Otherwise if we add "syslog_special" or "default_extra" they will be confused with the above.

-- Steve

Quoted 20 lines
kernel/printk/printk.c +9 -1
+		console_msg_format = MSG_FORMAT_DEFAULT;
+	return 1;
+}
+__setup("console_msg_format=", console_msg_format_setup);
+
 /*
  * Set up a console.  Called via do_early_param() in init/main.c
  * for each "console=" parameter in the boot command line.
@@ -2215,7 +2233,10 @@ void console_unlock(void)
 			goto skip;
 		}
 
-		len += msg_print_text(msg, false, text + len, sizeof(text) - len);
+		len += msg_print_text(msg,
+				console_msg_format & MSG_FORMAT_SYSLOG,
+				text + len,
+				sizeof(text) - len);
 		if (nr_ext_console_drivers) {
 			ext_len = msg_print_ext_header(ext_text,
 						sizeof(ext_text),
#2 lore raw

Hi,

On (12/19/17 10:17), Steven Rostedt wrote:
[..]

Quoted 7 lines
Quoted 7 lines
Documentation/admin-guide/kernel-parameters.txt +7 -0
+	console_msg_format=
+			[KNL] Change console messages format
+		default
+			By default we print messages on consoles in
+			"[time stamp] text\n" format (time stamp may not be
+			printed, depending on CONFIG_PRINTK_TIME or
+			`printk_time' param).
Quoted 8 lines

Documentation/admin-guide/kernel-parameters.txt line 657

Quoted 6 lines
Documentation/admin-guide/kernel-parameters.txt +6 -0
+		syslog
+			Switch to syslog format: "<%u>[time stamp] text\n"
+			IOW, each message will have a facility and loglevel
+			prefix. The format is similar to one used by syslog()
+			syscall, or to executing "dmesg -S --raw" or to reading
+			from /proc/kmsg.

Just to clarify. This affects all consoles, not just serial?

yes.

just like printk_time param/config option affects all the consoles, console_msg_format affects all of them. we msg_print_text() once, that makes messages look the same on every console. so I kept the existing behaviour.

[..]

Quoted 8 lines

kernel/printk/printk.c EXPORT_SYMBOL(console_set_on_cmdline); · line 283

Quoted 4 lines
kernel/printk/printk.c +4 -0
+enum con_msg_format {
+	MSG_FORMAT_DEFAULT	= 0,
+	MSG_FORMAT_SYSLOG	= (1 << 0),
+};

So the con_msg_format will be a flag (as denoted with the (1<<0)). Should we call it "con_msg_format_flags" to make it more obvious that these are treated as flags and not a simple number.

ok, can do.

[..]

Quoted 10 lines

kernel/printk/printk.c static int __add_preferred_console(char *name, int idx, char *options, · line 1928

Quoted 5 lines
kernel/printk/printk.c +5 -0
+static int __init console_msg_format_setup(char *str)
+{
+	if (!strncmp(str, "syslog", 6))
+		console_msg_format = MSG_FORMAT_SYSLOG;
+	if (!strncmp(str, "default", 7))

You can use strcmp() instead of strncmp(), the init code will nul terminate the string for you before calling this function. Otherwise if we add "syslog_special" or "default_extra" they will be confused with the above.

ok, that's a minor thing, but can change.

	-ss
#3 lore raw

PATCH v3 1/1 printk: add console_msg_format command line option

In reply to Sergey Senozhatsky

0day and kernelCI automatically parse kernel log - basically some sort of grepping using the pre-defined text patterns - in order to detect and report regressions/errors. There are several sources they get the kernel logs from:

a) dmesg or /proc/ksmg

This is the preferred way. Because `dmesg --raw' (see later Note) and /proc/kmsg output contains facility and log level, which greatly simplifies grepping for EMERG/ALERT/CRIT/ERR messages.

b) serial consoles

This option is harder to maintain, because serial console messages don't contain facility and log level.

This patch introduces a `console_msg_format=' command line option, to switch between different message formatting on serial consoles. For the time being we have just two options - default and syslog. The "default" option just keeps the existing format. While the "syslog" option makes serial console messages to appear in syslog format [syslog() syscall], matching the dmesg -S --raw' and cat /proc/kmsg' output formats:

- facility and log level
- time stamp (depends on printk_time/PRINTK_TIME)
- message

	<%u>[time stamp] text\n

NOTE: while Kevin and Fengguang talk about "dmesg --raw", it's actually "dmesg -S --raw" that always prints messages in syslog format [per Petr Mladek]. Running "dmesg --raw" may produce output in non-syslog format sometimes. console_msg_format=syslog enables syslog format, thus in documentation we mention "dmesg -S --raw", not "dmesg --raw".

Per Kevin Hilman:

: Right now we can get this info from a "dmesg --raw" after bootup, : but it would be really nice in certain automation frameworks to : have a kernel command-line option to enable printing of loglevels : in default boot log.
:
: This is especially useful when ingesting kernel logs into advanced : search/analytics frameworks (I'm playing with and ELK stack: Elastic : Search, Logstash, Kibana).
:
: The other important reason for having this on the command line is that : for testing linux-next (and other bleeding edge developer branches), : it's common that we never make it to userspace, so can't even run : "dmesg --raw" (or equivalent.) So we really want this on the primary : boot (serial) console.

Per Fengguang Wu, 0day scripts should quickly benefit from that feature, because they will be able to switch to a more reliable parsing, based on messages' facility and log levels [1]:

`#{grep} -a -E -e '^<[0123]>' -e '^kern :(err |crit |alert |emerg )'

instead of doing text pattern matching

`#{grep} -a -F -f /lkp/printk-error-messages #{kmsg_file} |

      grep -a -v -E -f #{LKP_SRC}/etc/oops-pattern |
      grep -a -v -F -f #{LKP_SRC}/etc/kmsg-blacklist`

[1] https://github.com/fengguang/lkp-tests/blob/master/lib/dmesg.rb

---

v3: addressed comments made by Steven Rostedt
v2: addressed comments made by Petr Mladek

 Documentation/admin-guide/kernel-parameters.txt | 14 ++++++++++++++
 kernel/printk/printk.c                          | 23 ++++++++++++++++++++++-
 2 files changed, 36 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +14 -0
index 8a61d1eda677..543de386e01e 100644
@@ -642,6 +642,20 @@
 			console=brl,ttyS0
 		For now, only VisioBraille is supported.
 
+	console_msg_format=
+			[KNL] Change console messages format
+		default
+			By default we print messages on consoles in
+			"[time stamp] text\n" format (time stamp may not be
+			printed, depending on CONFIG_PRINTK_TIME or
+			`printk_time' param).
+		syslog
+			Switch to syslog format: "<%u>[time stamp] text\n"
+			IOW, each message will have a facility and loglevel
+			prefix. The format is similar to one used by syslog()
+			syscall, or to executing "dmesg -S --raw" or to reading
+			from /proc/kmsg.
+
 	consoleblank=	[KNL] The console blank (screen saver) timeout in
 			seconds. A value of 0 disables the blank timer.
                        Defaults to 0.
kernel/printk/printk.c +22 -1
index c2e713f6ae2e..a214759b0fc1 100644
@@ -277,6 +277,13 @@ EXPORT_SYMBOL(console_set_on_cmdline);
 /* Flag: console code may call schedule() */
 static int console_may_schedule;
 
+enum con_msg_format_flags {
+	MSG_FORMAT_DEFAULT	= 0,
+	MSG_FORMAT_SYSLOG	= (1 << 0),
+};
+
+static int console_msg_format = MSG_FORMAT_DEFAULT;
+
 /*
  * The printk log buffer consists of a chain of concatenated variable
  * length records. Every record starts with a record header, containing
@@ -1913,6 +1920,17 @@ static int __add_preferred_console(char *name, int idx, char *options,
 	c->index = idx;
 	return 0;
 }
+
+static int __init console_msg_format_setup(char *str)
+{
+	if (!strcmp(str, "syslog"))
+		console_msg_format = MSG_FORMAT_SYSLOG;
+	if (!strcmp(str, "default"))
+		console_msg_format = MSG_FORMAT_DEFAULT;
+	return 1;
+}
+__setup("console_msg_format=", console_msg_format_setup);
+
 /*
  * Set up a console.  Called via do_early_param() in init/main.c
  * for each "console=" parameter in the boot command line.
@@ -2215,7 +2233,10 @@ void console_unlock(void)
 			goto skip;
 		}
 
-		len += msg_print_text(msg, false, text + len, sizeof(text) - len);
+		len += msg_print_text(msg,
+				console_msg_format & MSG_FORMAT_SYSLOG,
+				text + len,
+				sizeof(text) - len);
 		if (nr_ext_console_drivers) {
 			ext_len = msg_print_ext_header(ext_text,
 						sizeof(ext_text),
Signature · 1 line
2.15.1
maintainer Reviewed-by on v3 1/1
#4 lore raw
Quoted 12 lines · Sergey Senozhatsky

This patch introduces a `console_msg_format=' command line option, to switch between different message formatting on serial consoles. For the time being we have just two options - default and syslog. The "default" option just keeps the existing format. While the "syslog" option makes serial console messages to appear in syslog format [syslog() syscall], matching the dmesg -S --raw' and cat /proc/kmsg' output formats:

- facility and log level
- time stamp (depends on printk_time/PRINTK_TIME)
- message

Quoted 7 lines
	<%u>[time stamp] text\n

The patch looks and works good for me.

I am going to add it into for-4.16 branch when I am back in January. The patch is basically innocent. But I do not want to eventually screw up printk.git with a last minute commit.

Best Regards,
Petr