Skip to contentSkip to conversation

small scheduler debug stat fixups

PATCH In review Joe Lawrence 3 participants 8 messages Mar 19, 2018 – Mar 20, 2018 Scheduler

lore
Share & tools
lore
#0 lore raw

cover small scheduler debug stat fixups

A customer noticed some misalignment issues in the scheduler debug stats caused by line break/continuations. While the kernel isn't responsible for log rendering, these trivial changes can help clean up the stats display with default settings for dmesg/rsyslog/etc.

Note that /proc/sched_debug didn't suffer from these issues and its output remains unchanged by this patchset.

Joe Lawrence (2):
sched/debug: fix per-task line continuation for console sched/debug: adjust newlines for better alignment

 kernel/sched/debug.c | 29 +++++++++++++++++------------
 1 file changed, 17 insertions(+), 12 deletions(-)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH 1/2 sched/debug: fix per-task line continuation for console

When the SEQ_printf() macro prints to the console, it runs a simple printk() without KERN_CONT "continued" line printing. The result of this is oddly wrapped task info, for example:

% echo t > /proc/sysrq-trigger
% dmesg
...
runnable tasks:
...

  [   29.608611]  I
  [   29.608613]       rcu_sched     8      3252.013846      4087   120
  [   29.608614]         0.000000        29.090111         0.000000
  [   29.608615]  0 0
  [   29.608616]  /

Modify SEQ_printf to use pr_cont() for expected one-line results:

% echo t > /proc/sysrq-trigger
% dmesg
...
runnable tasks:
...

  [  106.716329]  S        cpuhp/5    37      2006.315026        14   120         0.000000         0.496893         0.000000 0 0 /
---
 kernel/sched/debug.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/debug.c +1 -1
index 1ca0130ed4f9..50026aa2d81e 100644
@@ -32,7 +32,7 @@
 	if (m)					\
 		seq_printf(m, x);		\
 	else					\
-		printk(x);			\
+		pr_cont(x);			\
  } while (0)
 
 /*
Signature · 1 line
1.8.3.1
maintainer
#2 lore raw

Re: [PATCH 1/2] sched/debug: fix per-task line continuation for console

Quoted 34 lines · Joe Lawrence

When the SEQ_printf() macro prints to the console, it runs a simple printk() without KERN_CONT "continued" line printing. The result of this is oddly wrapped task info, for example:

% echo t > /proc/sysrq-trigger
% dmesg
...
runnable tasks:
...

  [   29.608611]  I
  [   29.608613]       rcu_sched     8      3252.013846      4087   120
  [   29.608614]         0.000000        29.090111         0.000000
  [   29.608615]  0 0
  [   29.608616]  /

Modify SEQ_printf to use pr_cont() for expected one-line results:

% echo t > /proc/sysrq-trigger
% dmesg
...
runnable tasks:
...

  [  106.716329]  S        cpuhp/5    37      2006.315026        14   120         0.000000         0.496893         0.000000 0 0 /
---
 kernel/sched/debug.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/debug.c +0 -0
index 1ca0130ed4f9..50026aa2d81e 100644

kernel/sched/debug.c line 35

Quoted 5 lines
kernel/sched/debug.c +1 -1
@@ -32,7 +32,7 @@
 	if (m)					\
 		seq_printf(m, x);		\
 	else					\
-		printk(x);			\
+		pr_cont(x);			\

That used to work I think.. I think someone changed how printk() behaves somewhere along the lines.

Does pr_cont("\n") DTRT? it seems like something weird.

#3 lore raw

Re: [PATCH 1/2] sched/debug: fix per-task line continuation for console

Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Joe Lawrence

When the SEQ_printf() macro prints to the console, it runs a simple printk() without KERN_CONT "continued" line printing. The result of this is oddly wrapped task info, for example:

% echo t > /proc/sysrq-trigger
% dmesg
...
runnable tasks:
...

  [   29.608611]  I
  [   29.608613]       rcu_sched     8      3252.013846      4087   120
  [   29.608614]         0.000000        29.090111         0.000000
  [   29.608615]  0 0
  [   29.608616]  /

Modify SEQ_printf to use pr_cont() for expected one-line results:

% echo t > /proc/sysrq-trigger
% dmesg
...
runnable tasks:
...

  [  106.716329]  S        cpuhp/5    37      2006.315026        14   120         0.000000         0.496893         0.000000 0 0 /
---
 kernel/sched/debug.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines

kernel/sched/debug.c line 35

Quoted 10 lines
kernel/sched/debug.c +1 -1
index 1ca0130ed4f9..50026aa2d81e 100644
@@ -32,7 +32,7 @@
 	if (m)					\
 		seq_printf(m, x);		\
 	else					\
-		printk(x);			\
+		pr_cont(x);			\

That used to work I think.. I think someone changed how printk() behaves somewhere along the lines.

Hi Peter,

This code:

        printk("printk one");
        printk("printk two\n");

        pr_cont("\n");
        pr_cont("pr_cont one");
        pr_cont("pr_cont two\n");

        pr_cont("pr_cont first line\n");
        pr_cont("\n");
        pr_cont("\n");
        pr_cont("pr_cont next line");

Creates this output:

	%  uname -r
	4.16.0-rc5+

	% dmesg
	[  575.221280] printk one
	[  575.221281] printk two
	[  575.221282] pr_cont onepr_cont two
	[  575.221283] pr_cont first line

I don't have the commit offhand that changed the printk behavior, but from observation:

1 - printk implies a trailing newline:

      https://lwn.net/Articles/732420/

2 - pr_cont seems to eat redundant newlines

Quoted 2 lines

Does pr_cont("\n") DTRT? it seems like something weird.

Yeah, pr_cont() is kind of a hack. It will terminate a line if that's the first newline, but as demonstrated above, I don't think it likes extra newline chars.

A better fix would be to marshal the text into temp buffer then dump it out. Dunno if you prefer that kind of churn for these stats.

-- Joe

maintainer
#4 lore raw

Re: [PATCH 1/2] sched/debug: fix per-task line continuation for console

Quoted 4 lines · Joe Lawrence

On 03/19/2018 04:17 PM, Peter Zijlstra wrote:

I don't have the commit offhand that changed the printk behavior, but from observation:

4bcc595ccd80 ("printk: reinstate KERN_CONT for printing continuation lines")

and thereabout I think.

Quoted 6 lines
Quoted 2 lines

Does pr_cont("\n") DTRT? it seems like something weird.

Yeah, pr_cont() is kind of a hack. It will terminate a line if that's the first newline, but as demonstrated above, I don't think it likes extra newline chars.

Fair enough I suppose, just a bit weird.

Quoted 2 lines

A better fix would be to marshal the text into temp buffer then dump it out. Dunno if you prefer that kind of churn for these stats.

Yeah, not worth it, simple is better here.

#5 lore raw

[tip:sched/urgent] sched/debug: Fix per-task line continuation for console output

In reply to Joe Lawrence
sched/debug: Fix per-task line continuation for console output
Commit-ID:  a8c024cd9b9683d25ae1f459525dd2c6bec75e79
Gitweb:     https://git.kernel.org/tip/a8c024cd9b9683d25ae1f459525dd2c6bec75e79
Author:     Joe Lawrence <[email protected]>
AuthorDate: Mon, 19 Mar 2018 14:35:54 -0400
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 20 Mar 2018 09:30:09 +0100

sched/debug: Fix per-task line continuation for console output

When the SEQ_printf() macro prints to the console, it runs a simple printk() without KERN_CONT "continued" line printing. The result of this is oddly wrapped task info, for example:

% echo t > /proc/sysrq-trigger
% dmesg
...
runnable tasks:
...

  [   29.608611]  I
  [   29.608613]       rcu_sched     8      3252.013846      4087   120
  [   29.608614]         0.000000        29.090111         0.000000
  [   29.608615]  0 0
  [   29.608616]  /

Modify SEQ_printf to use pr_cont() for expected one-line results:

% echo t > /proc/sysrq-trigger
% dmesg
...
runnable tasks:
...

  [  106.716329]  S        cpuhp/5    37      2006.315026        14   120         0.000000         0.496893         0.000000 0 0 /
---
 kernel/sched/debug.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/debug.c +1 -1
index 1ca0130ed4f9..50026aa2d81e 100644
@@ -32,7 +32,7 @@ static DEFINE_SPINLOCK(sched_debug_lock);
 	if (m)					\
 		seq_printf(m, x);		\
 	else					\
-		printk(x);			\
+		pr_cont(x);			\
  } while (0)
 
 /*
#6 lore raw

PATCH 2/2 sched/debug: adjust newlines for better alignment

In reply to Joe Lawrence

Scheduler debug stats include newlines that display out of alignment when prefixed by timestamps. For example, the dmesg utility:

% echo t > /proc/sysrq-trigger
% dmesg
...

  [   83.124251]
  runnable tasks:
   S           task   PID         tree-key  switches  prio     wait-time
  sum-exec        sum-sleep
  -----------------------------------------------------------------------------------------------------------

At the same time, some syslog utilities (like rsyslog by default) don't like the additional newlines control characters, saving lines like this to /var/log/messages:

  Mar 16 16:02:29 localhost kernel: #012runnable tasks:#012 S           task   PID         tree-key ...
                                    ^^^^               ^^^^

Clean these up by moving newline characters to their own SEQ_printf invocation. This leaves the /proc/sched_debug unchanged, but brings the entire output into alignment when prefixed:

% echo t > /proc/sysrq-trigger
% dmesg
...

  [   62.410368] runnable tasks:
  [   62.410368]  S           task   PID         tree-key  switches  prio     wait-time             sum-exec        sum-sleep
  [   62.410369] -----------------------------------------------------------------------------------------------------------
  [   62.410369]  I  kworker/u12:0     5      1932.215593       332   120         0.000000         3.621252         0.000000 0 0 /

and no escaped control characters from rsyslog in /var/log/messages:

  Mar 16 16:15:06 localhost kernel: runnable tasks:
  Mar 16 16:15:06 localhost kernel: S           task   PID         tree-key  ...
---
 kernel/sched/debug.c | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)
kernel/sched/debug.c +16 -11
index 50026aa2d81e..72c401b3b15c 100644
@@ -501,12 +501,12 @@ static void print_rq(struct seq_file *m, struct rq *rq, int rq_cpu)
 {
 	struct task_struct *g, *p;
 
-	SEQ_printf(m,
-	"\nrunnable tasks:\n"
-	" S           task   PID         tree-key  switches  prio"
-	"     wait-time             sum-exec        sum-sleep\n"
-	"-------------------------------------------------------"
-	"----------------------------------------------------\n");
+	SEQ_printf(m, "\n");
+	SEQ_printf(m, "runnable tasks:\n");
+	SEQ_printf(m, " S           task   PID         tree-key  switches  prio"
+		   "     wait-time             sum-exec        sum-sleep\n");
+	SEQ_printf(m, "-------------------------------------------------------"
+		   "----------------------------------------------------\n");
 
 	rcu_read_lock();
 	for_each_process_thread(g, p) {
@@ -527,9 +527,11 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 	unsigned long flags;
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-	SEQ_printf(m, "\ncfs_rq[%d]:%s\n", cpu, task_group_path(cfs_rq->tg));
+	SEQ_printf(m, "\n");
+	SEQ_printf(m, "cfs_rq[%d]:%s\n", cpu, task_group_path(cfs_rq->tg));
 #else
-	SEQ_printf(m, "\ncfs_rq[%d]:\n", cpu);
+	SEQ_printf(m, "\n");
+	SEQ_printf(m, "cfs_rq[%d]:\n", cpu);
 #endif
 	SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "exec_clock",
 			SPLIT_NS(cfs_rq->exec_clock));
@@ -595,9 +597,11 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 void print_rt_rq(struct seq_file *m, int cpu, struct rt_rq *rt_rq)
 {
 #ifdef CONFIG_RT_GROUP_SCHED
-	SEQ_printf(m, "\nrt_rq[%d]:%s\n", cpu, task_group_path(rt_rq->tg));
+	SEQ_printf(m, "\n");
+	SEQ_printf(m, "rt_rq[%d]:%s\n", cpu, task_group_path(rt_rq->tg));
 #else
-	SEQ_printf(m, "\nrt_rq[%d]:\n", cpu);
+	SEQ_printf(m, "\n");
+	SEQ_printf(m, "rt_rq[%d]:\n", cpu);
 #endif
 
 #define P(x) \
@@ -624,7 +628,8 @@ void print_dl_rq(struct seq_file *m, int cpu, struct dl_rq *dl_rq)
 {
 	struct dl_bw *dl_bw;
 
-	SEQ_printf(m, "\ndl_rq[%d]:\n", cpu);
+	SEQ_printf(m, "\n");
+	SEQ_printf(m, "dl_rq[%d]:\n", cpu);
 
 #define PU(x) \
 	SEQ_printf(m, "  .%-30s: %lu\n", #x, (unsigned long)(dl_rq->x))
Signature · 1 line
1.8.3.1
#7 lore raw

[tip:sched/urgent] sched/debug: Adjust newlines for better alignment

sched/debug: Adjust newlines for better alignment
Commit-ID:  e9ca267096674eadd1fd479279bcb58df1486049
Gitweb:     https://git.kernel.org/tip/e9ca267096674eadd1fd479279bcb58df1486049
Author:     Joe Lawrence <[email protected]>
AuthorDate: Mon, 19 Mar 2018 14:35:55 -0400
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 20 Mar 2018 09:30:09 +0100

sched/debug: Adjust newlines for better alignment

Scheduler debug stats include newlines that display out of alignment when prefixed by timestamps. For example, the dmesg utility:

% echo t > /proc/sysrq-trigger
% dmesg
...

  [   83.124251]
  runnable tasks:
   S           task   PID         tree-key  switches  prio     wait-time
  sum-exec        sum-sleep
  -----------------------------------------------------------------------------------------------------------

At the same time, some syslog utilities (like rsyslog by default) don't like the additional newlines control characters, saving lines like this to /var/log/messages:

  Mar 16 16:02:29 localhost kernel: #012runnable tasks:#012 S           task   PID         tree-key ...
                                    ^^^^               ^^^^

Clean these up by moving newline characters to their own SEQ_printf invocation. This leaves the /proc/sched_debug unchanged, but brings the entire output into alignment when prefixed:

% echo t > /proc/sysrq-trigger
% dmesg
...

  [   62.410368] runnable tasks:
  [   62.410368]  S           task   PID         tree-key  switches  prio     wait-time             sum-exec        sum-sleep
  [   62.410369] -----------------------------------------------------------------------------------------------------------
  [   62.410369]  I  kworker/u12:0     5      1932.215593       332   120         0.000000         3.621252         0.000000 0 0 /

and no escaped control characters from rsyslog in /var/log/messages:

  Mar 16 16:15:06 localhost kernel: runnable tasks:
  Mar 16 16:15:06 localhost kernel: S           task   PID         tree-key  ...
---
 kernel/sched/debug.c | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)
kernel/sched/debug.c +16 -11
index 50026aa2d81e..72c401b3b15c 100644
@@ -501,12 +501,12 @@ static void print_rq(struct seq_file *m, struct rq *rq, int rq_cpu)
 {
 	struct task_struct *g, *p;
 
-	SEQ_printf(m,
-	"\nrunnable tasks:\n"
-	" S           task   PID         tree-key  switches  prio"
-	"     wait-time             sum-exec        sum-sleep\n"
-	"-------------------------------------------------------"
-	"----------------------------------------------------\n");
+	SEQ_printf(m, "\n");
+	SEQ_printf(m, "runnable tasks:\n");
+	SEQ_printf(m, " S           task   PID         tree-key  switches  prio"
+		   "     wait-time             sum-exec        sum-sleep\n");
+	SEQ_printf(m, "-------------------------------------------------------"
+		   "----------------------------------------------------\n");
 
 	rcu_read_lock();
 	for_each_process_thread(g, p) {
@@ -527,9 +527,11 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 	unsigned long flags;
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-	SEQ_printf(m, "\ncfs_rq[%d]:%s\n", cpu, task_group_path(cfs_rq->tg));
+	SEQ_printf(m, "\n");
+	SEQ_printf(m, "cfs_rq[%d]:%s\n", cpu, task_group_path(cfs_rq->tg));
 #else
-	SEQ_printf(m, "\ncfs_rq[%d]:\n", cpu);
+	SEQ_printf(m, "\n");
+	SEQ_printf(m, "cfs_rq[%d]:\n", cpu);
 #endif
 	SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "exec_clock",
 			SPLIT_NS(cfs_rq->exec_clock));
@@ -595,9 +597,11 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 void print_rt_rq(struct seq_file *m, int cpu, struct rt_rq *rt_rq)
 {
 #ifdef CONFIG_RT_GROUP_SCHED
-	SEQ_printf(m, "\nrt_rq[%d]:%s\n", cpu, task_group_path(rt_rq->tg));
+	SEQ_printf(m, "\n");
+	SEQ_printf(m, "rt_rq[%d]:%s\n", cpu, task_group_path(rt_rq->tg));
 #else
-	SEQ_printf(m, "\nrt_rq[%d]:\n", cpu);
+	SEQ_printf(m, "\n");
+	SEQ_printf(m, "rt_rq[%d]:\n", cpu);
 #endif
 
 #define P(x) \
@@ -624,7 +628,8 @@ void print_dl_rq(struct seq_file *m, int cpu, struct dl_rq *dl_rq)
 {
 	struct dl_bw *dl_bw;
 
-	SEQ_printf(m, "\ndl_rq[%d]:\n", cpu);
+	SEQ_printf(m, "\n");
+	SEQ_printf(m, "dl_rq[%d]:\n", cpu);
 
 #define PU(x) \
 	SEQ_printf(m, "  .%-30s: %lu\n", #x, (unsigned long)(dl_rq->x))