Skip to contentSkip to conversation

tracing/splice/ringbuffer: updates for tip

GIT PULL In review Steven Rostedt 4 participants 23 messages Apr 29, 2009

Subsystems (2)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

Ingo,

I was doing a bit of tracing that was stressing the splice code and I found lots of badness. Mostly, the second patch is the killer, and should most likely go upstream ASAP. The ref counting on the splice page is wrong and is never freed!

The rest are fixes to the read side. While debugging the memory leak I pulled out my old logdev and used it to monitor what was happening inside of ftrace (tracing the tracer). I did not like the amount of copying that was happening in splice. This was mainly because splice was faster than the writer (and this was with function tracer running!). I added some enhancements to make the splice code perform a little nicer.

Here's the logdev patch I was using to trace. As well as adding my lfcnprint statements (similar to trace_printk).

http://rostedt.homelinux.com/logdev/logdev-0.6.0-2.6.30-rc1.patch

Please pull the latest tip/tracing/ftrace-1 tree, which can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/tracing/ftrace-1

Steven Rostedt (5):
tracing: convert ftrace_dump spinlocks to raw
tracing: fix ref count in splice pages
tracing: only add splice page if entries exist
tracing: have splice only copy full pages
ring-buffer: fix printk output

----

 kernel/trace/ring_buffer.c |    2 +-
 kernel/trace/trace.c       |   20 ++++++++++++++------
 2 files changed, 15 insertions(+), 7 deletions(-)
author maintainer
#4 lore raw

PATCH 2/5 tracing: fix ref count in splice pages

From: Steven Rostedt <[email protected]>

The pages allocated for the splice binary buffer did not initialize the ref count correctly. This caused pages not to be freed and causes a drastic memory leak.

Thanks to logdev I was able to trace the tracer to find where the leak was.

[ Impact: stop memory leak when using splice ]

---
 kernel/trace/trace.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
kernel/trace/trace.c +1 -0
index 5d704a4..9058240 100644
@@ -3531,6 +3531,7 @@ tracing_buffers_splice_read(struct file *file, loff_t *ppos,
 		if (!ref)
 			break;
 
+		ref->ref = 1;
 		ref->buffer = info->tr->buffer;
 		ref->page = ring_buffer_alloc_read_page(ref->buffer);
 		if (!ref->page) {
Signature · 3 lines
1.6.2.1

-- 
author maintainer
#1 lore raw

PATCH 1/5 tracing: convert ftrace_dump spinlocks to raw

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

ftrace_dump is used for printing out the contents of the ftrace ring buffer to the console on failure. Currently it uses a spinlock to synchronize the output from multiple failures on different CPUs. This spin lock currently is a normal spinlock and can cause issues with lockdep and lock tracing.

This patch converts it to raw since it is for error handling only. The lock is local to the ftrace_dump and is not used by any other infrastructure.

[ Impact: prevent ftrace_dump from locking up by internal tracing ]

---
 kernel/trace/trace.c |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)
kernel/trace/trace.c +6 -3
index b6183bc..5d704a4 100644
@@ -4114,7 +4114,8 @@ trace_printk_seq(struct trace_seq *s)
 
 static void __ftrace_dump(bool disable_tracing)
 {
-	static DEFINE_SPINLOCK(ftrace_dump_lock);
+	static raw_spinlock_t ftrace_dump_lock =
+		(raw_spinlock_t)__RAW_SPIN_LOCK_UNLOCKED;
 	/* use static because iter can be a bit big for the stack */
 	static struct trace_iterator iter;
 	unsigned int old_userobj;
@@ -4123,7 +4124,8 @@ static void __ftrace_dump(bool disable_tracing)
 	int cnt = 0, cpu;
 
 	/* only one dump */
-	spin_lock_irqsave(&ftrace_dump_lock, flags);
+	local_irq_save(flags);
+	__raw_spin_lock(&ftrace_dump_lock);
 	if (dump_ran)
 		goto out;
 
@@ -4195,7 +4197,8 @@ static void __ftrace_dump(bool disable_tracing)
 	}
 
  out:
-	spin_unlock_irqrestore(&ftrace_dump_lock, flags);
+	__raw_spin_unlock(&ftrace_dump_lock);
+	local_irq_restore(flags);
 }
 
 /* By default: disable tracing after the dump */
Signature · 3 lines
1.6.2.1

-- 
author maintainer
#5 lore raw

PATCH 3/5 tracing: only add splice page if entries exist

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

The splice code allocates a page even when the ring buffer is empty. It detects the ring buffer being empty when it it fails to copy anything from the ring buffer into the page.

This patch adds a check to see if there is anything in the ring buffer before allocating a page.

Thanks to logdev for letting me trace the tracer to find this.

[ Impact: speed up due to removing unnecessary allocation ]

---
 kernel/trace/trace.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)
kernel/trace/trace.c +6 -2
index 9058240..0aeb3b9 100644
@@ -3508,7 +3508,7 @@ tracing_buffers_splice_read(struct file *file, loff_t *ppos,
 		.spd_release	= buffer_spd_release,
 	};
 	struct buffer_ref *ref;
-	int size, i;
+	int entries, size, i;
 	size_t ret;
 
 	if (*ppos & (PAGE_SIZE - 1)) {
@@ -3523,7 +3523,9 @@ tracing_buffers_splice_read(struct file *file, loff_t *ppos,
 		len &= PAGE_MASK;
 	}
 
-	for (i = 0; i < PIPE_BUFFERS && len; i++, len -= PAGE_SIZE) {
+	entries = ring_buffer_entries_cpu(info->tr->buffer, info->cpu);
+
+	for (i = 0; i < PIPE_BUFFERS && len && entries; i++, len -= PAGE_SIZE) {
 		struct page *page;
 		int r;
 
@@ -3564,6 +3566,8 @@ tracing_buffers_splice_read(struct file *file, loff_t *ppos,
 		spd.partial[i].private = (unsigned long)ref;
 		spd.nr_pages++;
 		*ppos += PAGE_SIZE;
+
+		entries = ring_buffer_entries_cpu(info->tr->buffer, info->cpu);
 	}
 
 	spd.nr_pages = i;
Signature · 3 lines
1.6.2.1

-- 
author maintainer
#6 lore raw

PATCH 4/5 tracing: have splice only copy full pages

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

Splice works with pages, it is much more effecient to use an entire page than to copy bits over several pages.

Using logdev to trace the internals of the splice mechanism, I was able to see that splice can be very aggressive. When tracing is occurring, and the reader caught up to the writer, and the writer is on the reader page, the reader will copy what is there into the splice page. Splice may iterate over several pages and if the writer is still writing to the page, the reader will keep copying bits to new pages to pass to userspace.

This patch changes it to only pass data to userspace if the page is full (the writer has left the page). This has a small side effect that splice can not read a partial page, and must wait for the page to fill. This should not be an issue. If tracing has stopped, then a use of "read" will still read all of the page.

[ Impact: better performance for ring buffer splice code ]

---
 kernel/trace/trace.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/trace/trace.c +1 -1
index 0aeb3b9..f5427e0 100644
@@ -3542,7 +3542,7 @@ tracing_buffers_splice_read(struct file *file, loff_t *ppos,
 		}
 
 		r = ring_buffer_read_page(ref->buffer, &ref->page,
-					  len, info->cpu, 0);
+					  len, info->cpu, 1);
 		if (r < 0) {
 			ring_buffer_free_read_page(ref->buffer,
 						   ref->page);
Signature · 3 lines
1.6.2.1

-- 
author maintainer
#7 lore raw

PATCH 5/5 ring-buffer: fix printk output

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

The warning output in trace_recursive_lock uses %d for a long when it should be %ld.

[ Impact: fix compile warning ]

---
 kernel/trace/ring_buffer.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/trace/ring_buffer.c +1 -1
index 9692f10..f4cc590 100644
@@ -1491,7 +1491,7 @@ static int trace_recursive_lock(void)
 	/* Disable all tracing before we do anything else */
 	tracing_off_permanent();
 
-	printk_once(KERN_WARNING "Tracing recursion: depth[%d]:"
+	printk_once(KERN_WARNING "Tracing recursion: depth[%ld]:"
 		    "HC[%lu]:SC[%lu]:NMI[%lu]\n",
 		    current->trace_recursion,
 		    hardirq_count() >> HARDIRQ_SHIFT,
Signature · 3 lines
1.6.2.1

-- 
#2 lore raw

Re: [PATCH 1/5] tracing: convert ftrace_dump spinlocks to raw

In reply to Steven Rostedt

kernel/trace/trace.c static void __ftrace_dump(bool disable_tracing) · line 4128

Quoted 2 lines · On Wed, 29 Apr 2009 00:48:15 -0400 Steven Rostedt <[email protected]> wrote:
kernel/trace/trace.c +2 -0
+	local_irq_save(flags);
+	__raw_spin_lock(&ftrace_dump_lock);

I'm counting twentyish instances of this. Maybe it's time to give up and implement raw_spin_lock_irqsave().

#8 lore raw

Re: [PATCH 5/5] ring-buffer: fix printk output

In reply to Steven Rostedt

kernel/trace/ring_buffer.c static int trace_recursive_lock(void) · line 1494

Quoted 2 lines · On Wed, 29 Apr 2009 00:48:19 -0400 Steven Rostedt <[email protected]> wrote:
kernel/trace/ring_buffer.c +1 -1
-	printk_once(KERN_WARNING "Tracing recursion: depth[%d]:"
+	printk_once(KERN_WARNING "Tracing recursion: depth[%ld]:"

hrmph. I didn't know that printk_once() existed, and I should have known. I wonder how many other people don't know.

Should it use unlikely()?

#9 lore raw

Re: [PATCH 5/5] ring-buffer: fix printk output

Quoted 7 lines · Andrew Morton

kernel/trace/ring_buffer.c static int trace_recursive_lock(void) · line 1494

Quoted 2 lines · On Wed, 29 Apr 2009 00:48:19 -0400 Steven Rostedt <[email protected]> wrote:
kernel/trace/ring_buffer.c +1 -1
-	printk_once(KERN_WARNING "Tracing recursion: depth[%d]:"
+	printk_once(KERN_WARNING "Tracing recursion: depth[%ld]:"

hrmph. I didn't know that printk_once() existed, and I should have known. I wonder how many other people don't know.

Was posted to [email protected].

Quoted 1 line

Should it use unlikely()?

Not really. Any printk_once() usage site is an 'unlikely' branch already in most cases. Double unlikely just spreads the instructions needlessly. The condition that _leads_ to a printk_once() should be an unlikely().

Ingo

#3 lore raw

Re: [PATCH 1/5] tracing: convert ftrace_dump spinlocks to raw

In reply to Andrew Morton
Quoted 6 lines · Andrew Morton

kernel/trace/trace.c static void __ftrace_dump(bool disable_tracing) · line 4128

Quoted 2 lines · On Wed, 29 Apr 2009 00:48:15 -0400 Steven Rostedt <[email protected]> wrote:
kernel/trace/trace.c +2 -0
+	local_irq_save(flags);
+	__raw_spin_lock(&ftrace_dump_lock);

I'm counting twentyish instances of this. [...]

there's 9 in the tracing code, two more in the rest of the kernel.

Quoted 2 lines

[...] Maybe it's time to give up and implement
raw_spin_lock_irqsave().

There's a difference between:

	local_irq_save(flags);
	__raw_spin_lock(&ftrace_dump_lock);

and:

	raw_local_irq_save(flags);
	__raw_spin_lock(&ftrace_dump_lock);

raw_spin_lock_irqsave() would map to the latter logic-wise - but in the tracing code we want the former.

But the main reason is that using raw locks has its costs - it skips lockdep coverage, is preempt unsafe, etc. The APIs are not complete and intentionally so: they are minimal building blocks for the real APIs. Instrumentation goes deep in the guts of the kernel so using very low level primitives is justified there - but we dont want to make it _too_ easy to use a raw-anything locking facility.

Ingo

#10 lore raw

Re: [PATCH 5/5] ring-buffer: fix printk output

In reply to Ingo Molnar
Quoted 5 lines · On Wed, 29 Apr 2009 07:43:59 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 2 lines · Andrew Morton

On Wed, 29 Apr 2009 00:48:19 -0400 Steven Rostedt <[email protected]> wrote:

Quoted 7 lines
Quoted 5 lines

kernel/trace/ring_buffer.c static int trace_recursive_lock(void) · line 1494

Quoted 2 lines
kernel/trace/ring_buffer.c +1 -1
-	printk_once(KERN_WARNING "Tracing recursion: depth[%d]:"
+	printk_once(KERN_WARNING "Tracing recursion: depth[%ld]:"

hrmph. I didn't know that printk_once() existed, and I should have known. I wonder how many other people don't know.

Was posted to [email protected].

Well no wonder I didn't know about it.

Quoted 7 lines
Quoted 1 line

Should it use unlikely()?

Not really. Any printk_once() usage site is an 'unlikely' branch already in most cases. Double unlikely just spreads the instructions needlessly. The condition that _leads_ to a printk_once() should be an unlikely().

OK.

#22 lore raw
In reply to Steven Rostedt
Quoted 6 lines · Steven Rostedt

Ingo,

I was doing a bit of tracing that was stressing the splice code and I found lots of badness. Mostly, the second patch is the killer, and should most likely go upstream ASAP. The ref counting on the splice page is wrong and is never freed!

ah, this probably explains what i reported some time ago: the ftrace-analyzer OOM hang. I've cherry-picked this into tracing/urgent as well.

Quoted 27 lines

The rest are fixes to the read side. While debugging the memory leak I pulled out my old logdev and used it to monitor what was happening inside of ftrace (tracing the tracer). I did not like the amount of copying that was happening in splice. This was mainly because splice was faster than the writer (and this was with function tracer running!). I added some enhancements to make the splice code perform a little nicer.

Here's the logdev patch I was using to trace. As well as adding my lfcnprint statements (similar to trace_printk).

http://rostedt.homelinux.com/logdev/logdev-0.6.0-2.6.30-rc1.patch

Please pull the latest tip/tracing/ftrace-1 tree, which can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/tracing/ftrace-1

Steven Rostedt (5):
tracing: convert ftrace_dump spinlocks to raw
tracing: fix ref count in splice pages
tracing: only add splice page if entries exist
tracing: have splice only copy full pages
ring-buffer: fix printk output

Quoted 4 lines

----

 kernel/trace/ring_buffer.c |    2 +-
 kernel/trace/trace.c       |   20 ++++++++++++++------
 2 files changed, 15 insertions(+), 7 deletions(-)

Pulled, thanks a lot Steve!

Ingo

#11 lore raw

Re: [PATCH 5/5] ring-buffer: fix printk output

In reply to Andrew Morton
Quoted 13 lines · Andrew Morton
Quoted 11 lines · On Wed, 29 Apr 2009 07:43:59 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 7 lines · Andrew Morton

kernel/trace/ring_buffer.c static int trace_recursive_lock(void) · line 1494

Quoted 2 lines · On Wed, 29 Apr 2009 00:48:19 -0400 Steven Rostedt <[email protected]> wrote:
kernel/trace/ring_buffer.c +1 -1
-	printk_once(KERN_WARNING "Tracing recursion: depth[%d]:"
+	printk_once(KERN_WARNING "Tracing recursion: depth[%ld]:"

hrmph. I didn't know that printk_once() existed, and I should have known. I wonder how many other people don't know.

Quoted 3 lines
Quoted 1 line

Was posted to [email protected].

Well no wonder I didn't know about it.

I (too?) think it's counter-productive that the linux-next list is split out of lkml. I constantly fall into that trap: i get a bugreport against one of our trees, i see that there's vger in the
Cc: list and mistake it for being Cc:-ed to lkml (all our trees are developed on lkml and most of the bugreports come Cc:-ed to lkml) but in reality it's Cc:-ed to linux-next which has a much smaller audience. (which audience apparently does not even include you)

If this email list fragmentation and the resoluting loss of information bothers you too then please ask Stephen to move linux-next mails to lkml (i've Cc:-ed Stephen) - it's not like it's actually something separate ... today's linux-next messages are tomorrow's lkml messages. Moving linux-next mails to lkml would nicely improve the S/N ratio on lkml.

Ingo

#12 lore raw

Re: [PATCH 5/5] ring-buffer: fix printk output

Quoted 28 lines · On Wed, 29 Apr 2009 08:09:55 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 13 lines · Andrew Morton
Quoted 11 lines · On Wed, 29 Apr 2009 07:43:59 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 7 lines · Andrew Morton

kernel/trace/ring_buffer.c static int trace_recursive_lock(void) · line 1494

Quoted 2 lines · On Wed, 29 Apr 2009 00:48:19 -0400 Steven Rostedt <[email protected]> wrote:
kernel/trace/ring_buffer.c +1 -1
-	printk_once(KERN_WARNING "Tracing recursion: depth[%d]:"
+	printk_once(KERN_WARNING "Tracing recursion: depth[%ld]:"

hrmph. I didn't know that printk_once() existed, and I should have known. I wonder how many other people don't know.

Quoted 3 lines
Quoted 1 line

Was posted to [email protected].

Well no wonder I didn't know about it.

I (too?) think it's counter-productive that the linux-next list is split out of lkml. I constantly fall into that trap: i get a bugreport against one of our trees, i see that there's vger in the
Cc: list and mistake it for being Cc:-ed to lkml (all our trees are developed on lkml and most of the bugreports come Cc:-ed to lkml) but in reality it's Cc:-ed to linux-next which has a much smaller audience. (which audience apparently does not even include you)

Quoted 7 lines

If this email list fragmentation and the resoluting loss of information bothers you too then please ask Stephen to move linux-next mails to lkml (i've Cc:-ed Stephen) - it's not like it's actually something separate ... today's linux-next messages are tomorrow's lkml messages. Moving linux-next mails to lkml would nicely improve the S/N ratio on lkml.

eh. Just auto-add lkml to tipbot-commit emails?

printk_once() is racy on smp and preempt btw ;)

#13 lore raw

Re: [PATCH 5/5] ring-buffer: fix printk output

Quoted 38 lines · Andrew Morton
Quoted 28 lines · On Wed, 29 Apr 2009 08:09:55 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 13 lines · Andrew Morton
Quoted 11 lines · On Wed, 29 Apr 2009 07:43:59 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 7 lines · Andrew Morton

kernel/trace/ring_buffer.c static int trace_recursive_lock(void) · line 1494

Quoted 2 lines · On Wed, 29 Apr 2009 00:48:19 -0400 Steven Rostedt <[email protected]> wrote:
kernel/trace/ring_buffer.c +1 -1
-	printk_once(KERN_WARNING "Tracing recursion: depth[%d]:"
+	printk_once(KERN_WARNING "Tracing recursion: depth[%ld]:"

hrmph. I didn't know that printk_once() existed, and I should have known. I wonder how many other people don't know.

Quoted 3 lines
Quoted 1 line

Was posted to [email protected].

Well no wonder I didn't know about it.

I (too?) think it's counter-productive that the linux-next list is split out of lkml. I constantly fall into that trap: i get a bugreport against one of our trees, i see that there's vger in the
Cc: list and mistake it for being Cc:-ed to lkml (all our trees are developed on lkml and most of the bugreports come Cc:-ed to lkml) but in reality it's Cc:-ed to linux-next which has a much smaller audience. (which audience apparently does not even include you)

Quoted 7 lines

If this email list fragmentation and the resoluting loss of information bothers you too then please ask Stephen to move linux-next mails to lkml (i've Cc:-ed Stephen) - it's not like it's actually something separate ... today's linux-next messages are tomorrow's lkml messages. Moving linux-next mails to lkml would nicely improve the S/N ratio on lkml.

Quoted 1 line

eh. Just auto-add lkml to tipbot-commit emails?

Yeah - the tip-commits-bot already has such a feature and we are making use of it - but this commit predates it. So it was posted to linux-next and the tip-commits list.

My larger point remains, about possibly embedding linux-next into lkml. I couldnt think of a single linux-next mail that isnt relevant to lkml. It's all about commits that are destined for upstream in 0-2.5 months.

Quoted 1 line

printk_once() is racy on smp and preempt btw ;)

Like WARN_ONCE() and WARN_ON_ONCE(). It's really an "oh crap" facility, not for normal kernel messages.

Do we want to complicate them with locking and preemption - or should we just concentrate on getting the "oh crap" message out to the syslog (before it's possibly too late to get anything out)?

I have no strong opinion about it - but i tend to like the simpler method most. printk + stack dumps themselves arent atomic to begin with.

Ingo

#14 lore raw

Re: [PATCH 5/5] ring-buffer: fix printk output

Quoted 41 lines · On Wed, 29 Apr 2009 09:22:08 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 38 lines · Andrew Morton
Quoted 35 lines · On Wed, 29 Apr 2009 08:09:55 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 13 lines · Andrew Morton
Quoted 11 lines · On Wed, 29 Apr 2009 07:43:59 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 7 lines · Andrew Morton

Review

Quoted 2 lines · On Wed, 29 Apr 2009 00:48:19 -0400 Steven Rostedt <[email protected]> wrote:
(quoted diff) +1 -1
-	printk_once(KERN_WARNING "Tracing recursion: depth[%d]:"
+	printk_once(KERN_WARNING "Tracing recursion: depth[%ld]:"

hrmph. I didn't know that printk_once() existed, and I should have known. I wonder how many other people don't know.

Quoted 3 lines
Quoted 1 line

Was posted to [email protected].

Well no wonder I didn't know about it.

I (too?) think it's counter-productive that the linux-next list is split out of lkml. I constantly fall into that trap: i get a bugreport against one of our trees, i see that there's vger in the
Cc: list and mistake it for being Cc:-ed to lkml (all our trees are developed on lkml and most of the bugreports come Cc:-ed to lkml) but in reality it's Cc:-ed to linux-next which has a much smaller audience. (which audience apparently does not even include you)

If this email list fragmentation and the resoluting loss of information bothers you too then please ask Stephen to move linux-next mails to lkml (i've Cc:-ed Stephen) - it's not like it's actually something separate ... today's linux-next messages are tomorrow's lkml messages. Moving linux-next mails to lkml would nicely improve the S/N ratio on lkml.

Quoted 5 lines
Quoted 1 line

eh. Just auto-add lkml to tipbot-commit emails?

Yeah - the tip-commits-bot already has such a feature and we are making use of it - but this commit predates it. So it was posted to linux-next and the tip-commits list.

there's a tip-commits list?

Quoted 4 lines

My larger point remains, about possibly embedding linux-next into lkml. I couldnt think of a single linux-next mail that isnt relevant to lkml. It's all about commits that are destined for upstream in 0-2.5 months.

Sure, I'd be OK with zapping the linux-next list.

Quoted 5 lines
Quoted 1 line

printk_once() is racy on smp and preempt btw ;)

Like WARN_ONCE() and WARN_ON_ONCE(). It's really an "oh crap" facility, not for normal kernel messages.

Quoted 7 lines

Do we want to complicate them with locking and preemption - or should we just concentrate on getting the "oh crap" message out to the syslog (before it's possibly too late to get anything out)?

I have no strong opinion about it - but i tend to like the simpler method most. printk + stack dumps themselves arent atomic to begin with.

Well, it's hardly likely to be a problem. otoh, if two CPUs _do_ hit the thing at the same time, the resulting output will be all messed up and we'd really like to see it.

Easily fixed with test_and_set_bit()?

#15 lore raw

Re: [PATCH 5/5] ring-buffer: fix printk output

Quoted 43 lines · Andrew Morton
Quoted 41 lines · On Wed, 29 Apr 2009 09:22:08 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 38 lines · Andrew Morton
Quoted 35 lines · On Wed, 29 Apr 2009 08:09:55 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 13 lines · Andrew Morton
Quoted 11 lines · On Wed, 29 Apr 2009 07:43:59 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 7 lines · Andrew Morton

Review

Quoted 2 lines · On Wed, 29 Apr 2009 00:48:19 -0400 Steven Rostedt <[email protected]> wrote:
(quoted diff) +1 -1
-	printk_once(KERN_WARNING "Tracing recursion: depth[%d]:"
+	printk_once(KERN_WARNING "Tracing recursion: depth[%ld]:"

hrmph. I didn't know that printk_once() existed, and I should have known. I wonder how many other people don't know.

Quoted 3 lines
Quoted 1 line

Was posted to [email protected].

Well no wonder I didn't know about it.

I (too?) think it's counter-productive that the linux-next list is split out of lkml. I constantly fall into that trap: i get a bugreport against one of our trees, i see that there's vger in the
Cc: list and mistake it for being Cc:-ed to lkml (all our trees are developed on lkml and most of the bugreports come Cc:-ed to lkml) but in reality it's Cc:-ed to linux-next which has a much smaller audience. (which audience apparently does not even include you)

If this email list fragmentation and the resoluting loss of information bothers you too then please ask Stephen to move linux-next mails to lkml (i've Cc:-ed Stephen) - it's not like it's actually something separate ... today's linux-next messages are tomorrow's lkml messages. Moving linux-next mails to lkml would nicely improve the S/N ratio on lkml.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

eh. Just auto-add lkml to tipbot-commit emails?

Yeah - the tip-commits-bot already has such a feature and we are making use of it - but this commit predates it. So it was posted to linux-next and the tip-commits list.

there's a tip-commits list?

has been for months, see:

http://vger.kernel.org/vger-lists.html
http://vger.kernel.org/vger-lists.html#linux-tip-commits

Quoted 6 lines
Quoted 4 lines

My larger point remains, about possibly embedding linux-next into lkml. I couldnt think of a single linux-next mail that isnt relevant to lkml. It's all about commits that are destined for upstream in 0-2.5 months.

Sure, I'd be OK with zapping the linux-next list.

Another, less drastic solution would be to keep it as an _alias_ list. All mails posted to it also go to lkml, but it would still be subscribe-able separately.

( This has come up before and this would be useful for a number of
other things - such as tracing/instrumentation. Someone who is only interested in instrumentation related discussions could subscribe to that list. )

Quoted 14 lines
Quoted 5 lines
Quoted 1 line

printk_once() is racy on smp and preempt btw ;)

Like WARN_ONCE() and WARN_ON_ONCE(). It's really an "oh crap" facility, not for normal kernel messages.

Quoted 8 lines

Do we want to complicate them with locking and preemption - or should we just concentrate on getting the "oh crap" message out to the syslog (before it's possibly too late to get anything out)?

I have no strong opinion about it - but i tend to like the simpler method most. printk + stack dumps themselves arent atomic to begin with.

Quoted 5 lines

Well, it's hardly likely to be a problem. otoh, if two CPUs _do_ hit the thing at the same time, the resulting output will be all messed up and we'd really like to see it.

Easily fixed with test_and_set_bit()?

but if two CPUs hit it at once then the printk+stack-dump itself is already mixed up. So if we do any atomicity it should be done for all the print-once APIs. (note, lockdep does such message-atomicity already, in its own facility)

Ingo

#16 lore raw

Re: [PATCH 5/5] ring-buffer: fix printk output

Quoted 11 lines · On Wed, 29 Apr 2009 11:56:25 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 7 lines
Quoted 5 lines

My larger point remains, about possibly embedding linux-next into lkml. I couldnt think of a single linux-next mail that isnt relevant to lkml. It's all about commits that are destined for upstream in 0-2.5 months.

Sure, I'd be OK with zapping the linux-next list.

Another, less drastic solution would be to keep it as an _alias_ list. All mails posted to it also go to lkml, but it would still be subscribe-able separately.

That would work, although I wonder about the potential for duplicates turning up somewhere.

Quoted 23 lines

( This has come up before and this would be useful for a number of
other things - such as tracing/instrumentation. Someone who is only interested in instrumentation related discussions could subscribe to that list. )

Quoted 18 lines
Quoted 5 lines
Quoted 1 line

printk_once() is racy on smp and preempt btw ;)

Like WARN_ONCE() and WARN_ON_ONCE(). It's really an "oh crap" facility, not for normal kernel messages.

Quoted 8 lines

Do we want to complicate them with locking and preemption - or should we just concentrate on getting the "oh crap" message out to the syslog (before it's possibly too late to get anything out)?

I have no strong opinion about it - but i tend to like the simpler method most. printk + stack dumps themselves arent atomic to begin with.

Well, it's hardly likely to be a problem. otoh, if two CPUs _do_ hit the thing at the same time, the resulting output will be all messed up and we'd really like to see it.

Quoted 6 lines
Quoted 1 line

Easily fixed with test_and_set_bit()?

but if two CPUs hit it at once then the printk+stack-dump itself is already mixed up. So if we do any atomicity it should be done for all the print-once APIs. (note, lockdep does such message-atomicity already, in its own facility)

Confused.

<gets distracted by FW_BUG and friends. ytf are they in kernel.h?>

#define printk_once(x...) ({				\
	static unsigned long __print_once;		\
							\
	if (!test_and_set_bit(0, &__print_once))	\
		printk(x);				\
})

How can two CPUs do the printk(x)?

#20 lore raw

Re: [PATCH 5/5] ring-buffer: fix printk output

In reply to Andrew Morton
Quoted 1 line · Andrew Morton

printk_once() is racy on smp and preempt btw ;)

Looks like at worst, it's prone to the usual printk racing, plus a possible misfire causing it to print twice before shutting up? Or is there something I'm missing here?

#17 lore raw

Re: [PATCH 5/5] ring-buffer: fix printk output

In reply to Andrew Morton
Quoted 8 lines · Andrew Morton
Quoted 6 lines · On Wed, 29 Apr 2009 11:56:25 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 6 lines
Quoted 5 lines

My larger point remains, about possibly embedding linux-next into lkml. I couldnt think of a single linux-next mail that isnt relevant to lkml. It's all about commits that are destined for upstream in 0-2.5 months.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Sure, I'd be OK with zapping the linux-next list.

Another, less drastic solution would be to keep it as an _alias_ list. All mails posted to it also go to lkml, but it would still be subscribe-able separately.

That would work, although I wonder about the potential for duplicates turning up somewhere.

The potential for duplicates is inherent in Cc: lines to begin with.

Quoted 30 lines
Quoted 23 lines

( This has come up before and this would be useful for a number of
other things - such as tracing/instrumentation. Someone who is only interested in instrumentation related discussions could subscribe to that list. )

Quoted 18 lines
Quoted 5 lines
Quoted 1 line

printk_once() is racy on smp and preempt btw ;)

Like WARN_ONCE() and WARN_ON_ONCE(). It's really an "oh crap" facility, not for normal kernel messages.

Quoted 8 lines

Do we want to complicate them with locking and preemption - or should we just concentrate on getting the "oh crap" message out to the syslog (before it's possibly too late to get anything out)?

I have no strong opinion about it - but i tend to like the simpler method most. printk + stack dumps themselves arent atomic to begin with.

Well, it's hardly likely to be a problem. otoh, if two CPUs _do_ hit the thing at the same time, the resulting output will be all messed up and we'd really like to see it.

Quoted 6 lines
Quoted 1 line

Easily fixed with test_and_set_bit()?

but if two CPUs hit it at once then the printk+stack-dump itself is already mixed up. So if we do any atomicity it should be done for all the print-once APIs. (note, lockdep does such message-atomicity already, in its own facility)

Quoted 7 lines

Confused.

<gets distracted by FW_BUG and friends. ytf are they in kernel.h?>

#define printk_once(x...) ({				\
	static unsigned long __print_once;		\
							\

hm, this doubles the flag size on 32-bit kernels.

Quoted 5 lines
	if (!test_and_set_bit(0, &__print_once))	\
		printk(x);				\
})

How can two CPUs do the printk(x)?

they cannot. Did i say they do? What i said (or thought to have said ;-) was that all the print-once APIs need similar treatment - or neither should.

Ingo

#18 lore raw

Re: [PATCH 5/5] ring-buffer: fix printk output

Quoted 13 lines · On Wed, 29 Apr 2009 21:45:46 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 10 lines · Andrew Morton
Quoted 8 lines · On Wed, 29 Apr 2009 11:56:25 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 7 lines
Quoted 5 lines

My larger point remains, about possibly embedding linux-next into lkml. I couldnt think of a single linux-next mail that isnt relevant to lkml. It's all about commits that are destined for upstream in 0-2.5 months.

Sure, I'd be OK with zapping the linux-next list.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Another, less drastic solution would be to keep it as an _alias_ list. All mails posted to it also go to lkml, but it would still be subscribe-able separately.

That would work, although I wonder about the potential for duplicates turning up somewhere.

The potential for duplicates is inherent in Cc: lines to begin with.

But if someone does reply-to-all to linux-next and linux-kernel, and the linux-next email gets redirected to linux-kernel, badness might happen. Bearing in mind the screwiness of the mail clients whcih some people use..

Quoted 32 lines
Quoted 32 lines
Quoted 23 lines

( This has come up before and this would be useful for a number of
other things - such as tracing/instrumentation. Someone who is only interested in instrumentation related discussions could subscribe to that list. )

Quoted 18 lines
Quoted 5 lines
Quoted 1 line

printk_once() is racy on smp and preempt btw ;)

Like WARN_ONCE() and WARN_ON_ONCE(). It's really an "oh crap" facility, not for normal kernel messages.

Quoted 8 lines

Do we want to complicate them with locking and preemption - or should we just concentrate on getting the "oh crap" message out to the syslog (before it's possibly too late to get anything out)?

I have no strong opinion about it - but i tend to like the simpler method most. printk + stack dumps themselves arent atomic to begin with.

Well, it's hardly likely to be a problem. otoh, if two CPUs _do_ hit the thing at the same time, the resulting output will be all messed up and we'd really like to see it.

Quoted 6 lines
Quoted 1 line

Easily fixed with test_and_set_bit()?

but if two CPUs hit it at once then the printk+stack-dump itself is already mixed up. So if we do any atomicity it should be done for all the print-once APIs. (note, lockdep does such message-atomicity already, in its own facility)

Confused.

Quoted 7 lines
Quoted 5 lines

<gets distracted by FW_BUG and friends. ytf are they in kernel.h?>

#define printk_once(x...) ({				\
	static unsigned long __print_once;		\
							\

hm, this doubles the flag size on 32-bit kernels.

Well yeah. I was wondering whether __print_once should be a char anyway. Will that hurt text size on any arch? Will gcc dtrt with such things? It might go and 4-byte align the chars anwyay.

#19 lore raw

Re: [PATCH 5/5] ring-buffer: fix printk output

Quoted 23 lines · Andrew Morton
Quoted 20 lines · On Wed, 29 Apr 2009 21:45:46 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 16 lines · Andrew Morton
Quoted 11 lines · On Wed, 29 Apr 2009 11:56:25 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 7 lines
Quoted 5 lines

My larger point remains, about possibly embedding linux-next into lkml. I couldnt think of a single linux-next mail that isnt relevant to lkml. It's all about commits that are destined for upstream in 0-2.5 months.

Sure, I'd be OK with zapping the linux-next list.

Another, less drastic solution would be to keep it as an _alias_ list. All mails posted to it also go to lkml, but it would still be subscribe-able separately.

That would work, although I wonder about the potential for duplicates turning up somewhere.

Quoted 6 lines
Quoted 1 line

The potential for duplicates is inherent in Cc: lines to begin with.

But if someone does reply-to-all to linux-next and linux-kernel, and the linux-next email gets redirected to linux-kernel, badness might happen. Bearing in mind the screwiness of the mail clients whcih some people use..

Hm, what badness would happen? Today, if i send a mail to linux-next and lkml, it shows up on both lists. If i'm subscribed to both, i get two mails - one from lkml and one from linux-next.

Auto-aliasing the lists means adding an implicit Cc: lkml to all mails that the linux-next list server gets. Two mails are generated
- and if someone is subscribed to both lists, two mails are received. How would mail clients have a problem with that? It's already happening.

Quoted 38 lines
Quoted 38 lines
Quoted 37 lines
Quoted 23 lines

( This has come up before and this would be useful for a number of
other things - such as tracing/instrumentation. Someone who is only interested in instrumentation related discussions could subscribe to that list. )

Quoted 18 lines
Quoted 5 lines
Quoted 1 line

printk_once() is racy on smp and preempt btw ;)

Like WARN_ONCE() and WARN_ON_ONCE(). It's really an "oh crap" facility, not for normal kernel messages.

Quoted 8 lines

Do we want to complicate them with locking and preemption - or should we just concentrate on getting the "oh crap" message out to the syslog (before it's possibly too late to get anything out)?

I have no strong opinion about it - but i tend to like the simpler method most. printk + stack dumps themselves arent atomic to begin with.

Well, it's hardly likely to be a problem. otoh, if two CPUs _do_ hit the thing at the same time, the resulting output will be all messed up and we'd really like to see it.

Quoted 6 lines
Quoted 1 line

Easily fixed with test_and_set_bit()?

but if two CPUs hit it at once then the printk+stack-dump itself is already mixed up. So if we do any atomicity it should be done for all the print-once APIs. (note, lockdep does such message-atomicity already, in its own facility)

Confused.

<gets distracted by FW_BUG and friends. ytf are they in kernel.h?>

#define printk_once(x...) ({				\
	static unsigned long __print_once;		\
							\
Quoted 5 lines
Quoted 1 line

hm, this doubles the flag size on 32-bit kernels.

Well yeah. I was wondering whether __print_once should be a char anyway. Will that hurt text size on any arch? Will gcc dtrt with such things? It might go and 4-byte align the chars anwyay.

a char would make sense - since they are all rare codepaths compression is important - and this should be done for the other _ONCE APIs as well.

gcc does the right thing and will compress adjacent char's - but the problem is that these char's are unlikely to be adjacent - te adjacent variables will likely be larger so the chars will take up 4 or 8 bytes in practice.

So to achieve compression we'd have to put them into a separate data section and have a per arch linker script detail for that. Is that worth the trouble? How many flags are we talking about?

Ingo

#21 lore raw

Re: [PATCH 5/5] ring-buffer: fix printk output

In reply to [email protected]
Quoted 6 lines · * [email protected] <[email protected]> wrote:
Quoted 1 line · Andrew Morton

printk_once() is racy on smp and preempt btw ;)

Looks like at worst, it's prone to the usual printk racing, plus a possible misfire causing it to print twice before shutting up? Or is there something I'm missing here?

Yeah, that's the worst-case scenario.

Still worth improving if we can do it without bloating the code or data.

[ If not, we can just assume the kernel is perfect and ignore all
the printk_once() and WARN_ONCE() / WARN_ON_ONCE() callsites as obviously dead code ;-) ]

Ingo