Skip to contentSkip to conversation

Self nomination

Ksummit-discuss Sasha Levin 9 participants 22 messages Jul 31, 2015 – Aug 3, 2015 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hi folks,

I'd like to nominate myself to this year's kernel summit.

Mainly I'd like to talk about improving testing around the kernel, both by catching bugs and by improving the quality of debug output that comes out of the kernel.

As the maintainer of the 3.18 LTS line I'd also like to discuss various -stable kernel topics that came up previously on the mailing list.

Thanks,
Sasha

#1 lore raw
Quoted 2 lines · Sasha Levin

Mainly I'd like to talk about improving testing around the kernel, both by catching bugs and by improving the quality of debug output that comes out of the kernel.

What sort of debug output improvements are you interested in? I spend a fair amount of time converting to dev_printk and %pR. They make the kernel more consistent and approachable, but they're minor and I'm sure you have more substantive things in mind.

#2 lore raw

Hi Bjorn,

Quoted 6 lines · Bjorn Helgaas
Quoted 2 lines · Sasha Levin

Mainly I'd like to talk about improving testing around the kernel, both by catching bugs and by improving the quality of debug output that comes out of the kernel.

What sort of debug output improvements are you interested in? I spend a fair amount of time converting to dev_printk and %pR. They make the

I have been wondering about that, especially since dev_dbg() and 'dev_printk(KERN_DEBUG, ...)' are semantically different. Any reason for preferring dev_printk() over dev_dbg() ?

Thank,
Guenter

#3 lore raw
Quoted 7 lines · Guenter Roeck

Hi Bjorn,

Quoted 4 lines · Bjorn Helgaas
Quoted 2 lines · Sasha Levin

Mainly I'd like to talk about improving testing around the kernel, both by catching bugs and by improving the quality of debug output that comes out of the kernel.

Quoted 6 lines
Quoted 2 lines

What sort of debug output improvements are you interested in? I spend a fair amount of time converting to dev_printk and %pR. They make the

I have been wondering about that, especially since dev_dbg() and 'dev_printk(KERN_DEBUG, ...)' are semantically different. Any reason for preferring dev_printk() over dev_dbg() ?

Personally I think that difference is a hassle. When I convert printk(KERN_DEBUG) to use dev_printk, I use dev_printk(KERN_DEBUG) to preserve the fact that the printk happens unconditionally. If I were smarter, I would figure out how to use dev_dbg() effectively, but usually I want my patch to be "convert to dev_printk", not "convert to dev_dbg and only print when some magic incantation is given".

#4 lore raw
In reply to Guenter Roeck
Quoted 7 lines · Guenter Roeck

Hi Bjorn,

Quoted 4 lines · Bjorn Helgaas
Quoted 2 lines · Sasha Levin

Mainly I'd like to talk about improving testing around the kernel, both by catching bugs and by improving the quality of debug output that comes out of the kernel.

Quoted 6 lines
Quoted 2 lines

What sort of debug output improvements are you interested in? I spend a fair amount of time converting to dev_printk and %pR. They make the

I have been wondering about that, especially since dev_dbg() and 'dev_printk(KERN_DEBUG, ...)' are semantically different. Any reason for preferring dev_printk() over dev_dbg() ?

The opposite is true, please always use dev_dbg() as it properly ties into the overall kernel-wide dynamic debug infrastructure, providing a unified way to enable/disable debug messages, or even, compiling them out if none are wanted due to size constraints.

thanks,

greg k-h

#5 lore raw
Quoted 8 lines · Greg KH
Quoted 7 lines · Guenter Roeck

Hi Bjorn,

Quoted 4 lines · Bjorn Helgaas
Quoted 2 lines · Sasha Levin

Mainly I'd like to talk about improving testing around the kernel, both by catching bugs and by improving the quality of debug output that comes out of the kernel.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

What sort of debug output improvements are you interested in? I spend a fair amount of time converting to dev_printk and %pR. They make the

I have been wondering about that, especially since dev_dbg() and 'dev_printk(KERN_DEBUG, ...)' are semantically different. Any reason for preferring dev_printk() over dev_dbg() ?

The opposite is true, please always use dev_dbg() as it properly ties

Hi Greg,

maybe my semantics wasn't perfect - my question was why Bjorn prefers dev_printk(KERN_DEBUG, ..) over dev_dbg(), which he answered.

I did not (want to) make the claim or even suggest that dev_printk() would be preferred over dev_dbg() in general.

Thanks,
Guenter

#12 lore raw
In reply to Greg KH
Quoted 11 lines · Greg KH
Quoted 10 lines · Guenter Roeck

Hi Bjorn,

Quoted 6 lines · Bjorn Helgaas
Quoted 2 lines · Sasha Levin

Mainly I'd like to talk about improving testing around the kernel, both by catching bugs and by improving the quality of debug output that comes out of the kernel.

What sort of debug output improvements are you interested in? I spend a fair amount of time converting to dev_printk and %pR. They make the

Quoted 8 lines
Quoted 3 lines

I have been wondering about that, especially since dev_dbg() and 'dev_printk(KERN_DEBUG, ...)' are semantically different. Any reason for preferring dev_printk() over dev_dbg() ?

The opposite is true, please always use dev_dbg() as it properly ties into the overall kernel-wide dynamic debug infrastructure, providing a unified way to enable/disable debug messages, or even, compiling them out if none are wanted due to size constraints.

Other than to confuse us all over minute details, is there a reason for this difference to exist? I realise right at the moment dev_dbg can be configured off by a variety of symbols and dev_printk(KERN_DEBUG) can't, conversely dev_printk doesn't pick up the location information and dev_dbg does, but there's no reason all that couldn't be harmonised so we didn't have to have arguments about which to use (and endless patches switching from one to the other)?

James

#13 lore raw
Quoted 16 lines · James Bottomley
Quoted 15 lines · Greg KH
Quoted 13 lines · Guenter Roeck

Hi Bjorn,

Quoted 6 lines · Bjorn Helgaas
Quoted 2 lines · Sasha Levin

Mainly I'd like to talk about improving testing around the kernel, both by catching bugs and by improving the quality of debug output that comes out of the kernel.

What sort of debug output improvements are you interested in? I spend a fair amount of time converting to dev_printk and %pR. They make the

I have been wondering about that, especially since dev_dbg() and 'dev_printk(KERN_DEBUG, ...)' are semantically different. Any reason for preferring dev_printk() over dev_dbg() ?

Quoted 7 lines
Quoted 4 lines

The opposite is true, please always use dev_dbg() as it properly ties into the overall kernel-wide dynamic debug infrastructure, providing a unified way to enable/disable debug messages, or even, compiling them out if none are wanted due to size constraints.

Other than to confuse us all over minute details, is there a reason for this difference to exist?

dev_dbg() uses dev_printk(KERN_DEBUG) to build on, we can't delete dev_printk() from the tree, so we are stuck with both. If you can figure out a way to prevent this, that would be great.

thanks,

greg k-h

#16 lore raw
In reply to a message not in the archive
Quoted 9 lines · Julia Lawall
Quoted 4 lines

I'd like to nominate myself to this year's kernel summit.

Quoted 3 lines

Mainly I'd like to talk about improving testing around the kernel, both by catching bugs and by improving the quality of debug output that comes out of the kernel.

Improving it in what way? We are doing some research related to logging at the moment, and so I would be interested to hear what you think are the problems.

From my perspective the biggest issue is making a given error state more reproducible. Quite

often I am able to trigger a bug on my testing box, but the information the kernel spits out (backtrace + registers) aren't too useful if the problem is complicated.

The result is that finding out what happens depends on few people who are able to "magically" trigger bugs and result in a long and inefficient back and forth while the bug manages to sneak in upstream and to users hands.

To sum it up, I think we need to figure out a way to produce enough information to make bugs more reproducible, but not too much to avoid a needle in a haystack situation.

In this regard, the new Intel PT technology is interesting and it's also worth figuring out a good way to integrate it with our current infrastructure/tools.

Among other, less urgent ideas I'd like to discuss are:

- Using KASan for poison fields. Quite a few structures have embedded poison fields. Rather
than detecting a corruption after it happens we are able to detect it as it happens - making bugs more obvious.

- Encouraging folks who add new sysctls (or features to existing sysctls) to contribute
testing code to the various testing projects around (trinity and such).

- Improving userspace tooling to make transfer of information simpler (along the lines of
scripts/decode_stacktrace.sh that no one seems to be using).

I suppose it'll be interesting to discuss the idea of making the kernel dump a "black box" on panic. Not just the core memory, but also various parameters regarding configuration and such, to enable folks to provide us with more than just backtraces.

Thanks,
Sasha

#14 lore raw
In reply to James Bottomley
Quoted 7 lines · James Bottomley

Other than to confuse us all over minute details, is there a reason for this difference to exist? I realise right at the moment dev_dbg can be configured off by a variety of symbols and dev_printk(KERN_DEBUG) can't, conversely dev_printk doesn't pick up the location information and dev_dbg does, but there's no reason all that couldn't be harmonised so we didn't have to have arguments about which to use (and endless patches switching from one to the other)?

I suppose it's a good sub-topic: deciding which one of the 3.5 methods to print debug information we should be using across the kernel and remove the other 2.5.

Thanks,
Sasha

#6 lore raw
In reply to Guenter Roeck
Quoted 18 lines · Guenter Roeck
Quoted 8 lines · Greg KH
Quoted 7 lines · Guenter Roeck

Hi Bjorn,

Quoted 4 lines · Bjorn Helgaas
Quoted 2 lines · Sasha Levin

Mainly I'd like to talk about improving testing around the kernel, both by catching bugs and by improving the quality of debug output that comes out of the kernel.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

What sort of debug output improvements are you interested in? I spend a fair amount of time converting to dev_printk and %pR. They make the

I have been wondering about that, especially since dev_dbg() and 'dev_printk(KERN_DEBUG, ...)' are semantically different. Any reason for preferring dev_printk() over dev_dbg() ?

The opposite is true, please always use dev_dbg() as it properly ties

Quoted 7 lines

Hi Greg,

maybe my semantics wasn't perfect - my question was why Bjorn prefers dev_printk(KERN_DEBUG, ..) over dev_dbg(), which he answered.

I did not (want to) make the claim or even suggest that dev_printk() would be preferred over dev_dbg() in general.

Lest I give the wrong impression, I'm not opposed to dev_dbg(). I just think there are two separate changes that don't need to be made at the same time: (1) convert to dev_printk style, and (2) convert to the dynamic debug stuff. As a consumer of dmesg logs, I'm most interested in the former.

I prefer that a dmesg collected in the simplest possible way, with no special config or boot flags, be as useful as possible. So converting to dynamic debug requires much more thought about which messages should be always printed and which should become dynamic.

#7 lore raw
Quoted 4 lines · Bjorn Helgaas

I prefer that a dmesg collected in the simplest possible way, with no special config or boot flags, be as useful as possible. So converting to dynamic debug requires much more thought about which messages should be always printed and which should become dynamic.

Why would a debugging message ever not be dynamic? They are there for you to use, and turn off when you are done. If you want a user to report the output of them, then of course they should be dynamic so they can just write a line to a debugfs file and then start seeing them, they shouldn't be seeing them all the time, and they should never require a rebuild of the kernel to see them (which is why we deleted CONFIG_USB_DEBUG).

thanks,

greg k-h

#9 lore raw
In reply to Dave Jones
Quoted 7 lines · Dave Jones
Quoted 6 lines · Greg Kroah-Hartman
Quoted 4 lines · Bjorn Helgaas

I prefer that a dmesg collected in the simplest possible way, with no special config or boot flags, be as useful as possible. So converting to dynamic debug requires much more thought about which messages should be always printed and which should become dynamic.

Quoted 8 lines
Quoted 4 lines

Why would a debugging message ever not be dynamic? They are there for you to use, and turn off when you are done. If you want a user to report the output of them, then of course they should be dynamic so they can just write a line to a debugfs file and then start seeing them

This implies a user knows ahead of time what bugs they are going to hit, and which messages they need to enable. For hard-to-reproduce bugs, or bugs that exhibit non-obvious symptoms, this isn't workable.

Fair enough, but wouldn't those messages be "errors"?

Anyway, this is way off-topic from the original thread, it all comes down to specifics of the message that is being written and the surrounding issues of why it would be written.

thanks,

greg k-h

#8 lore raw
In reply to Greg KH
Quoted 10 lines · Greg Kroah-Hartman
Quoted 4 lines · Bjorn Helgaas

I prefer that a dmesg collected in the simplest possible way, with no special config or boot flags, be as useful as possible. So converting to dynamic debug requires much more thought about which messages should be always printed and which should become dynamic.

Why would a debugging message ever not be dynamic? They are there for you to use, and turn off when you are done. If you want a user to report the output of them, then of course they should be dynamic so they can just write a line to a debugfs file and then start seeing them

This implies a user knows ahead of time what bugs they are going to hit, and which messages they need to enable. For hard-to-reproduce bugs, or bugs that exhibit non-obvious symptoms, this isn't workable.

Dave

#10 lore raw
In reply to Greg KH
Quoted 17 lines · Greg KH
Quoted 7 lines · Dave Jones
Quoted 6 lines · Greg Kroah-Hartman
Quoted 4 lines · Bjorn Helgaas

I prefer that a dmesg collected in the simplest possible way, with no special config or boot flags, be as useful as possible. So converting to dynamic debug requires much more thought about which messages should be always printed and which should become dynamic.

Quoted 8 lines
Quoted 4 lines

Why would a debugging message ever not be dynamic? They are there for you to use, and turn off when you are done. If you want a user to report the output of them, then of course they should be dynamic so they can just write a line to a debugfs file and then start seeing them

This implies a user knows ahead of time what bugs they are going to hit, and which messages they need to enable. For hard-to-reproduce bugs, or bugs that exhibit non-obvious symptoms, this isn't workable.

Quoted 5 lines

Fair enough, but wouldn't those messages be "errors"?

Anyway, this is way off-topic from the original thread, it all comes down to specifics of the message that is being written and the surrounding issues of why it would be written.

One more thought on this tangent... I've had a task stuck at the bottom of my backlog to look at extending dynamic_debug call sites to optionally be tracepoints so that you could feasibly have debug on all the time with less run time impact.

#11 lore raw
Quoted 4 lines · Dan Williams

One more thought on this tangent... I've had a task stuck at the bottom of my backlog to look at extending dynamic_debug call sites to optionally be tracepoints so that you could feasibly have debug on all the time with less run time impact.

Making dynamic debug use jump labels should be enough to achieve this.

That used to actually be the case, but got reverted later, as it didn't properly handle gccs that didn't support asm goto (see 2d75af2f2a7).

Once that is fixed to be more general and handle older gccs as well, your task is done :)

Signature · 2 lines
Jiri Kosina
SUSE Labs
#17 lore raw
In reply to Sasha Levin
Quoted 2 lines · Sasha Levin

- Encouraging folks who add new sysctls (or features to existing sysctls) to contribute
testing code to the various testing projects around (trinity and such).

Which other fuzzers are people using besides trinity?

Also why don't we merge trinity under tools/testing/? I bet people would keep it in sync better if we did that.

regards,
dan carpenter

#18 lore raw
Quoted 5 lines · Dan Carpenter
Quoted 2 lines · Sasha Levin

- Encouraging folks who add new sysctls (or features to existing sysctls) to contribute
testing code to the various testing projects around (trinity and such).

Which other fuzzers are people using besides trinity?

I'm not sure, I think trinity sort of took over :)

My point was that people should be submitting a more significant amount of test code along with their new ABI - none of the recent syscalls added has any mention in LTP for example.

Quoted 2 lines

Also why don't we merge trinity under tools/testing/? I bet people would keep it in sync better if we did that.

No objections on my end. I believe that Dave previously objected because it's hard to make it work on sync with the kernel's release cycle.

I've actually presented a proposal in linuxcon back in 2013 for unifying trinity into the kernel, and using common headers for syscalls. This would force people who add/modify syscalls to update trinity, and it'll also make the ABI self documenting as one would need to be very specific about what parameters are expected for each syscall.

Thanks,
Sasha

#19 lore raw
Quoted 5 lines · Sasha Levin
Quoted 4 lines · Dan Carpenter
Quoted 2 lines · Sasha Levin

- Encouraging folks who add new sysctls (or features to existing sysctls) to contribute
testing code to the various testing projects around (trinity and such).

Quoted 7 lines
Quoted 1 line

Which other fuzzers are people using besides trinity?

I'm not sure, I think trinity sort of took over :)

My point was that people should be submitting a more significant amount of test code along with their new ABI - none of the recent syscalls added has any mention in LTP for example.

Because most of us hate LTP.

Seriously, I've lost days of my life many years ago tracing down "kernel bugs" that in the end were just bugs in LTP due to the horrid code in it.

Now yes, this was a long time ago, and hopefully it has gotten a lot better, but lots of people have this attitude which is why we are encouraging people to use the tests in the kernel tree itself, and is why new syscalls are all adding their tests there.

I've encouraged people to take the "good bits" of LTP and add them to the kernel test framework, but no one seems to want to do that...

thanks,

greg k-h

#21 lore raw
In reply to Sasha Levin
Quoted 13 lines · Sasha Levin
Quoted 5 lines · Dan Carpenter
Quoted 2 lines · Sasha Levin

- Encouraging folks who add new sysctls (or features to existing sysctls) to contribute
testing code to the various testing projects around (trinity and such).

Which other fuzzers are people using besides trinity?

I'm not sure, I think trinity sort of took over :)

My point was that people should be submitting a more significant amount of test code along with their new ABI - none of the recent syscalls added has any mention in LTP for example.

Quoted 5 lines
Quoted 2 lines

Also why don't we merge trinity under tools/testing/? I bet people would keep it in sync better if we did that.

No objections on my end. I believe that Dave previously objected because it's hard to make it work on sync with the kernel's release cycle.

A big factor is the same reason I'm not a huge fan of tools/testing/ in general. People want to do things like "run latest testing tools on old LTS/enterprise kernels".

Having to suck a subtree out of the latest kernel tarball, and then spend time trying to make it even compile/run against headers from an old kernel is something I've already lost way too many weeks of my life to, so I'm not really a fan of making that problem even worse by adding other tools to it.

Dave

#20 lore raw
In reply to Greg KH
Quoted 9 lines · Greg KH
Quoted 3 lines
Quoted 3 lines

My point was that people should be submitting a more significant amount of test code along with their new ABI - none of the recent syscalls added has any mention in LTP for example.

Because most of us hate LTP.

Seriously, I've lost days of my life many years ago tracing down "kernel bugs" that in the end were just bugs in LTP due to the horrid code in it.

Quoted 7 lines

Now yes, this was a long time ago, and hopefully it has gotten a lot better, but lots of people have this attitude which is why we are encouraging people to use the tests in the kernel tree itself, and is why new syscalls are all adding their tests there.

I've encouraged people to take the "good bits" of LTP and add them to the kernel test framework, but no one seems to want to do that...

I didn't really mean to focus on LTP here: pick any project that has a critical mass of users and send them ways to test your code now and for future changes.

Thanks,
Sasha

#22 lore raw
In reply to Dave Jones
Quoted 20 lines · Dave Jones
Quoted 13 lines · Sasha Levin
Quoted 5 lines · Dan Carpenter
Quoted 2 lines · Sasha Levin

- Encouraging folks who add new sysctls (or features to existing sysctls) to contribute
testing code to the various testing projects around (trinity and such).

Which other fuzzers are people using besides trinity?

I'm not sure, I think trinity sort of took over :)

My point was that people should be submitting a more significant amount of test code along with their new ABI - none of the recent syscalls added has any mention in LTP for example.

Quoted 5 lines
Quoted 2 lines

Also why don't we merge trinity under tools/testing/? I bet people would keep it in sync better if we did that.

No objections on my end. I believe that Dave previously objected because it's hard to make it work on sync with the kernel's release cycle.

Quoted 7 lines

A big factor is the same reason I'm not a huge fan of tools/testing/ in general. People want to do things like "run latest testing tools on old LTS/enterprise kernels".

Having to suck a subtree out of the latest kernel tarball, and then spend time trying to make it even compile/run against headers from an old kernel is something I've already lost way too many weeks of my life to, so I'm not really a fan of making that problem even worse by adding other tools to it.

An interesting model we can try here is to let development happen both in the kernel tree and outside of it, and patches will be exchanged between the two development trees (possibly automatically). Think of it as a "fork".

On one hand it'll provide the kernel with a proven testing framework, and on the other it'll give trinity a wider user/developer base.

Thanks,
Sasha