2.6.17-rc5-mm1
Another panic. This time on x440.
M.
http://test.kernel.org/abat/33803/debug/console.log
BUG: unable to handle kernel paging request at virtual address 22222232 printing eip: c012b6eb *pde = 15621001 *pte = 00000000 Oops: 0000 [#1] SMP last sysfs file: /class/vc/vcs1/dev CPU: 1 EIP: 0060:[<c012b6eb>] Not tainted VLI EFLAGS: 00010002 (2.6.17-rc5-mm1-autokern1 #1)
EIP is at check_deadlock+0x15/0xe0
eax: 22222222 ebx: 00000001 ecx: d4996000 edx: 00000001
esi: d686f550 edi: 22222222 ebp: 22222222 esp: d5bdfec8
ds: 007b es: 007b ss: 0068
Process mkdir09 (pid: 18867, threadinfo=d5bdf000 task=d5c0e000)
Stack: 00000000 d686f550 d3960568 22222222 c012b77b d3960568 d5bdf000 d5bdff00
d5c0e000 c012b922 d5bdff48 d3960568 00000246 c02d50de d5bdff00
d5bdff00
11111111 11111111 d5bdff00 ffffff9c d5bdff48 00000000 d5bdff48
ffffffef
Call Trace:
<c012b77b> check_deadlock+0xa5/0xe0 <c012b922>
debug_mutex_add_waiter+0x46/0x55
<c02d50de> __mutex_lock_slowpath+0x9e/0x1c0 <c0160061>
lookup_create+0x19/0x5b
<c016043a> sys_mkdirat+0x4c/0xc3 <c01604c0> sys_mkdir+0xf/0x13
<c02d6217> syscall_call+0x7/0xbQuoted 23 lines · Martin Bligh
Another panic. This time on x440.
M.
http://test.kernel.org/abat/33803/debug/console.log
BUG: unable to handle kernel paging request at virtual address 22222232 printing eip: c012b6eb *pde = 15621001 *pte = 00000000 Oops: 0000 [#1] SMP last sysfs file: /class/vc/vcs1/dev CPU: 1 EIP: 0060:[<c012b6eb>] Not tainted VLI EFLAGS: 00010002 (2.6.17-rc5-mm1-autokern1 #1)EIP is at
check_deadlock+0x15/0xe0
eax: 22222222 ebx: 00000001 ecx:d4996000edx: 00000001
esi:d686f550edi: 22222222 ebp: 22222222 esp:d5bdfec8
ds: 007b es: 007b ss: 0068
Process mkdir09 (pid: 18867, threadinfo=d5bdf000 task=d5c0e000)
Quoted 13 lines
Stack: 00000000 d686f550 d3960568 22222222 c012b77b d3960568 d5bdf000
d5bdff00
d5c0e000 c012b922 d5bdff48 d3960568 00000246 c02d50de d5bdff00
d5bdff00
11111111 11111111 d5bdff00 ffffff9c d5bdff48 00000000 d5bdff48
ffffffef
Call Trace:
<c012b77b> check_deadlock+0xa5/0xe0 <c012b922>
debug_mutex_add_waiter+0x46/0x55
<c02d50de> __mutex_lock_slowpath+0x9e/0x1c0 <c0160061>
lookup_create+0x19/0x5b
<c016043a> sys_mkdirat+0x4c/0xc3 <c01604c0> sys_mkdir+0xf/0x13
<c02d6217> syscall_call+0x7/0xbLooks like the lock validator came unstuck. But there's so much other crap happening in there it's hard to tell. Did you try it without all the lockdep stuff enabled?
Quoted 9 lines · Andrew Morton
Quoted 8 lines
EIP is at
check_deadlock+0x15/0xe0<
c012b77b>check_deadlock+0xa5/0xe0<c012b922>debug_mutex_add_waiter+0x46/0x55 <c02d50de> __mutex_lock_slowpath+0x9e/0x1c0 <c0160061> lookup_create+0x19/0x5b <c016043a> sys_mkdirat+0x4c/0xc3 <c01604c0> sys_mkdir+0xf/0x13 <c02d6217> syscall_call+0x7/0xb
Quoted 3 lines
Looks like the lock validator came unstuck. But there's so much other crap happening in there it's hard to tell. Did you try it without all the lockdep stuff enabled?
AFAICS this isnt the lock validator but the normal mutex debugging code (CONFIG_DEBUG_MUTEXES). The log does not indicate that lockdep was enabled.
Ingo
Quoted 13 lines · Ingo Molnar
Quoted 10 lines · Andrew Morton
Quoted 9 lines
EIP is at
check_deadlock+0x15/0xe0<
c012b77b>check_deadlock+0xa5/0xe0<c012b922>debug_mutex_add_waiter+0x46/0x55 <c02d50de> __mutex_lock_slowpath+0x9e/0x1c0 <c0160061> lookup_create+0x19/0x5b <c016043a> sys_mkdirat+0x4c/0xc3 <c01604c0> sys_mkdir+0xf/0x13 <c02d6217> syscall_call+0x7/0xb
Quoted 4 lines
Quoted 1 line
Looks like the lock validator came unstuck. But there's so much other crap happening in there it's hard to tell. Did you try it without all the lockdep stuff enabled?
AFAICS this isnt the lock validator but the normal mutex debugging code (
CONFIG_DEBUG_MUTEXES). The log does not indicate that lockdep was enabled.
Buggered if I know how that got turned on. I thought we turned it off by default now? That's what screwed up all the perf results before.
http://test.kernel.org/abat/33803/build/dotconfig
That's the build config it ran with.
CONFIG_DEBUG_MUTEXES=y
Grrr. Humpf. I can't see the option being turned on for lockdep ... what was the config option, and is it enabled by default?
M.
Quoted 11 lines · Martin J. Bligh
Quoted 3 lines
AFAICS this isnt the lock validator but the normal mutex debugging code (
CONFIG_DEBUG_MUTEXES). The log does not indicate that lockdep was enabled.Buggered if I know how that got turned on. I thought we turned it off by default now? That's what screwed up all the perf results before.
http://test.kernel.org/abat/33803/build/dotconfig
That's the build config it ran with.
CONFIG_DEBUG_MUTEXES=y
still ... it shouldnt have crashed on us. I did change it in -mm1 so i'll take a look tomorrow.
Quoted 2 lines
Grrr. Humpf. I can't see the option being turned on for lockdep ... what was the config option, and is it enabled by default?
these are the lock validator options in question:
# CONFIG_PROVE_SPIN_LOCKING is not set
# CONFIG_PROVE_RW_LOCKING is not set
# CONFIG_PROVE_MUTEX_LOCKING is not set
# CONFIG_PROVE_RWSEM_LOCKING is not set
and they are off by default.
Ingo
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Martin J. Bligh
Quoted 1 line
AFAICS this isnt the lock validator but the normal mutex debugging code (
CONFIG_DEBUG_MUTEXES). The log does not indicate that lockdep was enabled.Buggered if I know how that got turned on. I thought we turned it off by default now? That's what screwed up all the perf results before.
Quoted 11 lines
Quoted 4 lines
http://test.kernel.org/abat/33803/build/dotconfig
That's the build config it ran with.
CONFIG_DEBUG_MUTEXES=ystill ... it shouldnt have crashed on us. I did change it in -mm1 so i'll take a look tomorrow.
Quoted 2 lines
Grrr. Humpf. I can't see the option being turned on for lockdep ... what was the config option, and is it enabled by default?
In the -mm1 patch:
config DEBUG_MUTEXES
- bool "Mutex debugging, deadlock detection" - default n + bool "Mutex debugging, basic checks" + default y
Please don't do thatas a default.
It fucks up all the performance checking ;-(
M.
Quoted 11 lines · Martin J. Bligh
Quoted 2 lines
Quoted 2 lines
Grrr. Humpf. I can't see the option being turned on for lockdep ... what was the config option, and is it enabled by default?
In the -mm1 patch:
config
DEBUG_MUTEXES(quoted diff) +2
-2- bool "Mutex debugging, deadlock detection" - default n + bool "Mutex debugging, basic checks" + default y
Quoted 1 line
Please don't do thatas a default.
but ... i fixed the performance problem that caused the previous DEBUG_MUTEXES scalability problems. (there's no global mutex list anymore) We also default to e.g. DEBUG_SLAB which is alot more costly.
Quoted 1 line
It fucks up all the performance checking ;-(
i'm wondering, why doesnt your config have DEBUG_MUTEXES disabled? Then 'make oldconfig' would pick it up automatically.
Ingo
Quoted 2 lines
but ... i fixed the performance problem that caused the previous
DEBUG_MUTEXESscalability problems. (there's no global mutex list anymore) We also default to e.g.DEBUG_SLABwhich is alot more costly.
OK. So what's the perf impact of the new version on a 32 cpu machine? ;-) Maybe it's fine, maybe it's not.
Quoted 1 line
i'm wondering, why doesnt your config have
DEBUG_MUTEXESdisabled? Then 'make oldconfig' would pick it up automatically.
Because it builds off the same config file all the time. It was created before CONFIG_MUTEXES existed ... creating a situation where we have to explicitly disable new options all the time becomes a maintainance nightmare ;-(
If we don't want to do performance regression checking on -mm, that's fine, but I thought it was useful (has caught several things already). If we want debug options explicitly enabled, we can do a separate debug run, I'd think, but it makes it very difficult to do automated testing if we add random new debug options all the time on by default ...
If we really think the debug options we're turning on by default have zero perf impact, that's fine ... but it has not been my previous experience. People obviously haven't checked that carefully in the past, perhaps they are now and the world fixed itself, but I'm not that optimistic ...
M.
Quoted 6 lines · Martin J. Bligh
Quoted 3 lines
but ... i fixed the performance problem that caused the previous
DEBUG_MUTEXESscalability problems. (there's no global mutex list anymore) We also default to e.g.DEBUG_SLABwhich is alot more costly.OK. So what's the perf impact of the new version on a 32 cpu machine? ;-) Maybe it's fine, maybe it's not.
no idea, but it shouldnt be nearly as bad as say SLAB_DEBUG.
Quoted 7 lines
Quoted 2 lines
i'm wondering, why doesnt your config have
DEBUG_MUTEXESdisabled? Then 'make oldconfig' would pick it up automatically.Because it builds off the same config file all the time. It was created before
CONFIG_MUTEXESexisted ... creating a situation where we have to explicitly disable new options all the time becomes a maintainance nightmare ;-(
hm, why? Dont you disable DEBUG_SLAB? [that's a default y option too, and in your config it's disabled]
a oneliner script:
sed -i 's/CONFIG_MUTEX_DEBUGGING=y/# CONFIG_MUTEX_DEBUGGING is not set'
ought to do it, unless i'm missing something.
Really, there's an unfortunate friction of interests here:
on one side, the -mm kernel is about showcasing new code and finding bugs in them as fast as possible. Having new debugging options enabled by default is an important part of the testing effort. Users will care more about having no crashes than about having 0.5% more performance in select benchmarks.
on the other side, you obviously dont want a 0.5% overhead for select benchmarks, as that would mess up the history! A very fair and valid position too.
but one side has to give, we cant have both.
Quoted 2 lines
If we don't want to do performance regression checking on -mm, that's fine, but I thought it was useful (has caught several things already).
please dont misunderstand my position as being against your efforts - to the contrary, your performance regression testing has proven to be valuable numerous times! But you are a single intelligent person whom i can possibly talk into adding some scripting to ensure that certain options stay off in the .config - but i cannot cat-herd the many -mm testers on the other hand to all enable the debug options ;-) So i'm kind of forced trying to convince you - i cannot convince the basic human testing nature of keeping the defaults ;-)
Ingo
Quoted 4 lines
Quoted 1 line
OK. So what's the perf impact of the new version on a 32 cpu machine? ;-) Maybe it's fine, maybe it's not.
no idea, but it shouldnt be nearly as bad as say
SLAB_DEBUG.
The "no idea" is hardly reassuring ;-)
The latter point is definitely valid though, it's not an isolated issue.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
i'm wondering, why doesnt your config have
DEBUG_MUTEXESdisabled? Then 'make oldconfig' would pick it up automatically.Because it builds off the same config file all the time. It was created before
CONFIG_MUTEXESexisted ... creating a situation where we have to explicitly disable new options all the time becomes a maintainance nightmare ;-(
Quoted 7 lines
hm, why? Dont you disable
DEBUG_SLAB? [that's a default y option too, and in your config it's disabled]a oneliner script:
sed -i 's/CONFIG_MUTEX_DEBUGGING=y/#
CONFIG_MUTEX_DEBUGGINGis not set'ought to do it, unless i'm missing something.
because it's not maintainable in a fully automated system over time.
Quoted 7 lines
Really, there's an unfortunate friction of interests here:
on one side, the -mm kernel is about showcasing new code and finding bugs in them as fast as possible. Having new debugging options enabled by default is an important part of the testing effort. Users will care more about having no crashes than about having 0.5% more performance in select benchmarks.
on the other side, you obviously dont want a 0.5% overhead for select benchmarks, as that would mess up the history! A very fair and valid position too.
but one side has to give, we cant have both.
Above is a good description of the problem - I really want to get as much debugging stuff done automatically as possible. However, we can have both. We just need to do both runs, with an option to turn whatever the random debug options of the day on or off are. Something that won't change over time. CONFIG_DEBUG itself would seem ideal, except it kills lots of standard things like CONFIG_DEBUG_INFO, alt+sysrq support, and probably KALLSYMS and a few other things I forget. Need CONFIG_DEBUG_AFFFECTS_PERFORMANCE or something.
Quoted 3 lines
Quoted 1 line
If we don't want to do performance regression checking on -mm, that's fine, but I thought it was useful (has caught several things already).
please dont misunderstand my position as being against your efforts - to the contrary, your performance regression testing has proven to be valuable numerous times! But you are a single intelligent person whom i can possibly talk into adding some scripting to ensure that certain options stay off in the .config - but i cannot cat-herd the many -mm testers on the other hand to all enable the debug options ;-) So i'm kind of forced trying to convince you - i cannot convince the basic human testing nature of keeping the defaults ;-)
Is all good, is just frustrating on occcasion, sorry ;-) The problem is that especially in a remote, distributed system, it's impossible to maintain a continually changing set of config options for this over time. If we can get one option to flick off all the perf invasive stuff, that's a single change. If we have to add CONFIG_SLAB last week, CONFIG_DEBUG_MUTEX this week, and CONFIG_LOCK_DEBUG_STUFF next week, that gets much, much harder to maintain.
Hopefully that makes some sort of sense. Sorry, I realise the requirements are a little strange, but I really think we have to get testing automated or it just does not happen. The last round affected Intel, etc as well doing benchmarking.
Adding new runs is easy. Changing the harness is hard ;-)
M.
Quoted 10 lines · Martin Bligh
Quoted 5 lines
Quoted 2 lines
OK. So what's the perf impact of the new version on a 32 cpu machine? ;-) Maybe it's fine, maybe it's not.
no idea, but it shouldnt be nearly as bad as say
SLAB_DEBUG.The "no idea" is hardly reassuring ;-)
The latter point is definitely valid though, it's not an isolated issue.Adding new runs is easy. Changing the harness is hard ;-)
ok. How about a CONFIG_DEBUG_NO_OVERHEAD option, that would default to disabled but which you could set to y. Then we could make all the more expensive debug options:
default y if !CONFIG_DEBUG_NO_OVERHEAD
this would still mean you'd have to turn off CONFIG_DEBUG_NO_OVERHEAD, but it would be automatically maintainable for you after that initial effort, and we'd be careful to always flag new debugging options with this flag, if they are expensive. And initially i'd define "expensive" as "anything that adds runtime overhead".
would this be acceptable to you?
Ingo
Quoted 15 lines · Ingo Molnar
Quoted 10 lines · Martin Bligh
Quoted 4 lines
Quoted 1 line
OK. So what's the perf impact of the new version on a 32 cpu machine? ;-) Maybe it's fine, maybe it's not.
no idea, but it shouldnt be nearly as bad as say
SLAB_DEBUG.The "no idea" is hardly reassuring ;-)
The latter point is definitely valid though, it's not an isolated issue.Adding new runs is easy. Changing the harness is hard ;-)
Quoted 7 lines
ok. How about a
CONFIG_DEBUG_NO_OVERHEADoption, that would default to disabled but which you could set to y. Then we could make all the more expensive debug options:default y if !CONFIG_DEBUG_NO_OVERHEAD
this would still mean you'd have to turn off
CONFIG_DEBUG_NO_OVERHEAD, but it would be automatically maintainable for you after that initial effort, and we'd be careful to always flag new debugging options with this flag, if they are expensive. And initially i'd define "expensive" as "anything that adds runtime overhead".would this be acceptable to you?
Sure, makes sense. I don't care which way up it is, ie CONFIG_DEBUG_OVERHEAD vs CONFIG_DEBUG_NO_OVERHEAD, as long as it's easily separable.
There's probably other debug stuff we can turn on too, if we do that.
M.
Quoted 15 lines · Martin Bligh
Quoted 15 lines
Quoted 1 line
Adding new runs is easy. Changing the harness is hard ;-)
ok. How about a
CONFIG_DEBUG_NO_OVERHEADoption, that would default to disabled but which you could set to y. Then we could make all the more expensive debug options:default y if !CONFIG_DEBUG_NO_OVERHEAD
this would still mean you'd have to turn off
CONFIG_DEBUG_NO_OVERHEAD, but it would be automatically maintainable for you after that initial effort, and we'd be careful to always flag new debugging options with this flag, if they are expensive. And initially i'd define "expensive" as "anything that adds runtime overhead".
Quoted 7 lines
Quoted 1 line
would this be acceptable to you?
Sure, makes sense. I don't care which way up it is, ie
CONFIG_DEBUG_OVERHEADvsCONFIG_DEBUG_NO_OVERHEAD, as long as it's easily separable.There's probably other debug stuff we can turn on too, if we do that.
i've attached an updated patch that renames DEBUG_KERNEL_RUNTIME_OVERHEAD to DEBUG_KERNEL_OVERHEAD :-)
i like the DEBUG_KERNEL_OVERHEAD approach a lot better, because it relieves you of this constant (and apparently often losing) battle with Kconfig default values. It also gives us freedom to mark most of the transparent debugging options as default-enabled.
i think i'll also do a DEBUG_KERNEL_ALLCHECKS flag that if set will select and enable all the transparent runtime checks. Often people just want to enable everything that still leaves us with a usable kernel, and filtering out those debug options can be a challenge.
Ingo
lib/Kconfig.debug +27 -14
@@ -54,6 +54,15 @@ config DEBUG_KERNEL Say Y here if you are developing drivers or trying to debug and identify kernel problems. +config DEBUG_KERNEL_OVERHEAD + bool "Enable new debug options by default" + default y + help + Say Y here if you want to have new debugging options + enabled by default even if they cause runtime overhead. + (you can still disable/enable them manually, independently + of this switch) + config LOG_BUF_SHIFT int "Kernel log buffer size (16 => 64KB, 17 => 128KB)" if DEBUG_KERNEL range 12 21
@@ -113,7 +122,7 @@ config DEBUG_SLAB config DEBUG_SLAB_LEAK bool "Slab memory leak debugging" depends on DEBUG_SLAB - default y + default y if DEBUG_KERNEL_OVERHEAD help Enable /proc/slab_allocators - provides detailed information about which parts of the kernel are using slab objects. May be used for
@@ -122,7 +131,7 @@ config DEBUG_SLAB_LEAK config DEBUG_PREEMPT bool "Debug preemptible kernel" depends on DEBUG_KERNEL && PREEMPT - default y + default y if DEBUG_KERNEL_OVERHEAD depends on TRACE_IRQFLAGS_SUPPORT help If you say Y here then the kernel will use a debug variant of the
@@ -132,7 +141,7 @@ config DEBUG_PREEMPT config DEBUG_MUTEXES bool "Mutex debugging, basic checks" - default n + default y if DEBUG_KERNEL_OVERHEAD depends on DEBUG_KERNEL help This feature allows mutex semantics violations to be detected and
@@ -140,7 +149,7 @@ config DEBUG_MUTEXES config DEBUG_MUTEX_ALLOC bool "Detect incorrect freeing of live mutexes" - default n + default y if DEBUG_KERNEL_OVERHEAD depends on DEBUG_MUTEXES help This feature will check whether any held mutex is incorrectly
@@ -150,7 +159,7 @@ config DEBUG_MUTEX_ALLOC config DEBUG_MUTEX_DEADLOCKS bool "Detect mutex related deadlocks" - default n + default y if DEBUG_KERNEL_OVERHEAD depends on DEBUG_MUTEXES help This feature will automatically detect and report mutex related
@@ -158,7 +167,7 @@ config DEBUG_MUTEX_DEADLOCKS config DEBUG_RT_MUTEXES bool "RT Mutex debugging, deadlock detection" - default n + default y if DEBUG_KERNEL_OVERHEAD depends on DEBUG_KERNEL && RT_MUTEXES help This allows rt mutex semantics violations and rt mutex related
@@ -166,7 +175,7 @@ config DEBUG_RT_MUTEXES config DEBUG_PI_LIST bool - default n + default y if DEBUG_KERNEL_OVERHEAD depends on DEBUG_RT_MUTEXES config RT_MUTEX_TESTER
@@ -179,6 +188,7 @@ config RT_MUTEX_TESTER config DEBUG_SPINLOCK bool "Spinlock debugging" depends on DEBUG_KERNEL + default y if DEBUG_KERNEL_OVERHEAD help Say Y here and build SMP to catch missing spinlock initialization and certain other kinds of spinlock errors commonly made. This is
@@ -188,7 +198,7 @@ config DEBUG_SPINLOCK config PROVE_SPIN_LOCKING bool "Prove spin-locking correctness" depends on X86 - default n + default y if DEBUG_KERNEL_OVERHEAD help This feature enables the kernel to prove that all spinlock locking that occurs in the kernel runtime is mathematically
@@ -226,7 +236,7 @@ config PROVE_SPIN_LOCKING config PROVE_RW_LOCKING bool "Prove rw-locking correctness" depends on X86 - default n + default y if DEBUG_KERNEL_OVERHEAD help This feature enables the kernel to prove that all rwlock locking that occurs in the kernel runtime is mathematically
@@ -264,7 +274,7 @@ config PROVE_RW_LOCKING config PROVE_MUTEX_LOCKING bool "Prove mutex-locking correctness" depends on X86 - default n + default y if DEBUG_KERNEL_OVERHEAD help This feature enables the kernel to prove that all mutexlock locking that occurs in the kernel runtime is mathematically
@@ -302,7 +312,7 @@ config PROVE_MUTEX_LOCKING config PROVE_RWSEM_LOCKING bool "Prove rwsem-locking correctness" depends on X86 - default n + default y if DEBUG_KERNEL_OVERHEAD help This feature enables the kernel to prove that all rwsemlock locking that occurs in the kernel runtime is mathematically
@@ -348,7 +358,7 @@ config LOCKDEP config DEBUG_LOCKDEP bool "Lock dependency engine debugging" depends on LOCKDEP - default y + default y if DEBUG_KERNEL_OVERHEAD depends on TRACE_IRQFLAGS_SUPPORT help If you say Y here, the lock dependency engine will do
@@ -363,6 +373,7 @@ config TRACE_IRQFLAGS config DEBUG_SPINLOCK_SLEEP bool "Sleep-inside-spinlock checking" + default y if DEBUG_KERNEL_OVERHEAD depends on DEBUG_KERNEL help If you say Y here, various routines which may sleep will become very
@@ -390,6 +401,7 @@ config DEBUG_KOBJECT config DEBUG_HIGHMEM bool "Highmem debugging" depends on DEBUG_KERNEL && HIGHMEM + default y if DEBUG_KERNEL_OVERHEAD help This options enables addition error checking for high memory systems. Disable for production systems.
@@ -398,7 +410,7 @@ config DEBUG_BUGVERBOSE bool "Verbose BUG() reporting (adds 70K)" if DEBUG_KERNEL && EMBEDDED depends on BUG depends on ARM || ARM26 || M32R || M68K || SPARC32 || SPARC64 || X86_32 || FRV - default !EMBEDDED + default y if DEBUG_KERNEL_OVERHEAD && !EMBEDDED help Say Y here to make BUG() panics output the file name and line number of the BUG call as well as the EIP and oops trace. This aids
@@ -437,6 +449,7 @@ config DEBUG_FS config DEBUG_VM bool "Debug VM" depends on DEBUG_KERNEL + default y if DEBUG_KERNEL_OVERHEAD help Enable this to turn on extended checks in the virtual-memory system that may impact performance.
@@ -446,7 +459,7 @@ config DEBUG_VM config FRAME_POINTER bool "Compile the kernel with frame pointers" depends on DEBUG_KERNEL && (X86 || CRIS || M68K || M68KNOMMU || FRV || UML) - default y + default y if DEBUG_KERNEL_OVERHEAD help If you say Y here the resulting kernel image will be slightly larger and slower, but it might give very useful debugging information on
Hi,
On Thu, 1 Jun 2006, Ingo Molnar wrote:
Ingo, it would be nice to get any kind of feedback, this is already the third and you continue to produce more Kconfig mess, instead of fixing the stuff I already pointed out. :(
Quoted 21 lines
Index: linux/lib/Kconfig.debug
===================================================================lib/Kconfig.debug +9
-0@@ -54,6 +54,15 @@ config DEBUG_KERNEL Say Y here if you are developing drivers or trying to debug and identify kernel problems. +config DEBUG_KERNEL_OVERHEAD + bool "Enable new debug options by default" + default y + help + Say Y here if you want to have new debugging options + enabled by default even if they cause runtime overhead. + (you can still disable/enable them manually, independently + of this switch) + config LOG_BUF_SHIFT int "Kernel log buffer size (16 => 64KB, 17 => 128KB)" if DEBUG_KERNEL range 12 21
lib/Kconfig.debug config DEBUG_SLAB · line 128
Quoted 8 lines
lib/Kconfig.debug +1
-1@@ -113,7 +122,7 @@ config DEBUG_SLAB config DEBUG_SLAB_LEAK bool "Slab memory leak debugging" depends on DEBUG_SLAB - default y + default y if DEBUG_KERNEL_OVERHEAD help Enable /proc/slab_allocators - provides detailed information about which parts of the kernel are using slab objects. May be used for
This is nonsense.
If you do this do it at least correctly, e.g. something like:
config DEBUG_RUNTIME_CHECKS
bool "Enable runtime debug checks"
config DEBUG_RUNTIME_CHECKS_ALL
bool "Enable all runtime debug checks"
depends on DEBUG_RUNTIME_CHECKS
config DEBUG_KERNEL
bool "Kernel debugging"
...
config DEBUG_FOO
bool "foo" if DEBUG_KERNEL
default DEBUG_RUNTIME_CHECKS
Your current defaults are definitively NACKed for mainline, keep them separate!
bye, Roman
Quoted 11 lines
This is nonsense.
If you do this do it at least correctly, e.g. something like:config
DEBUG_RUNTIME_CHECKS
bool "Enable runtime debug checks"config
DEBUG_RUNTIME_CHECKS_ALL
bool "Enable all runtime debug checks"
depends onDEBUG_RUNTIME_CHECKS
Quoted 8 lines
config
DEBUG_KERNEL
bool "Kernel debugging"...
config
DEBUG_FOO
bool "foo" ifDEBUG_KERNEL
defaultDEBUG_RUNTIME_CHECKS
That doesn't seem to cover what we talked about clearly at all ? I suppose the _ALL stuff is meant to cover stuff with overhead, but frankly, what Ingo did seemed much clearer to me.
M.
Hi,
Quoted 3 lines · Martin J. Bligh
That doesn't seem to cover what we talked about clearly at all ? I suppose the
_ALLstuff is meant to cover stuff with overhead, but frankly, what Ingo did seemed much clearer to me.
It just didn't make much sense, a config option only to configure the default value of unseen values?
If we have too many debug options, I don't mind to hide them behind an advanced config option, but their default values should not differ between their visible and hidden state, so that the user sees the real values when he enables the advanced option.
A config option which only configures the default values is much less useful, in an already configured kernel it's completely useless to an user who only wants to enable some runtime checks and unless he reads the help text _carefully_, he might even think that he just enabled some runtime checks. AFAIC such stuff is NACKed.
bye, Roman
Quoted 5 lines · Roman Zippel
Hi,
On Thu, 1 Jun 2006, Martin J. Bligh wrote:
Quoted 8 lines
Quoted 3 lines
That doesn't seem to cover what we talked about clearly at all ? I suppose the
_ALLstuff is meant to cover stuff with overhead, but frankly, what Ingo did seemed much clearer to me.It just didn't make much sense, a config option only to configure the default value of unseen values?
If we have too many debug options, I don't mind to hide them behind an advanced config option, but their default values should not differ between their visible and hidden state, so that the user sees the real values when he enables the advanced option. A config option which only configures the default values is much less useful, in an already configured kernel it's completely useless to an user who only wants to enable some runtime checks and unless he reads the help text _carefully_, he might even think that he just enabled some runtime checks.
Did you read the discussion that lead up to it? I thought that quite clearly described why such a thing was needed.
Config options need to clearly distinguish what they're for, or people will screw them up ... given:
config DEBUG_RUNTIME_CHECKS
bool "Enable runtime debug checks"
config DEBUG_RUNTIME_CHECKS_ALL
bool "Enable all runtime debug checks"
depends on DEBUG_RUNTIME_CHECKS
... how is either the user meant to know under which of these to find things, or the coder introducing a new feature supposed to know where to put stuff? They're not very descriptive.
Please can we have a more constructive discussion that "it's wrong, nack" ?
M.
Hi,
Quoted 2 lines · Martin J. Bligh
Did you read the discussion that lead up to it? I thought that quite clearly described why such a thing was needed.
I did read it, what did I miss?
bye, Roman
Quoted 10 lines · Roman Zippel
Hi,
Quoted 2 lines · Martin J. Bligh
Did you read the discussion that lead up to it? I thought that quite clearly described why such a thing was needed.
I did read it, what did I miss?
So the point is to enable the performance-affecting debug options by default, but provide one clear hook to turn them all off, with a name that's consistent over time, so we can do it in an automated fashion.
To me that means having clear name so people know which option to hook stuff under, and preferably not hiding stuff behind extra config options to make them harder to find. Maybe I've missed the point of what you were trying to do, but it seemed to not meet that.
M.
Hi,
Quoted 3 lines · Martin Bligh
So the point is to enable the performance-affecting debug options by default, but provide one clear hook to turn them all off, with a name that's consistent over time, so we can do it in an automated fashion.
The latter is what Ingo's patch doesn't really do and the first should stay in -mm. If we talk about an -mm only patch we can relax the rules a little, my NACK was for mainline inclusion. The problem is that my request for a -mm only patch and other changes has been met with utter ignorance so far by Ingo.
Quoted 4 lines
To me that means having clear name so people know which option to hook stuff under, and preferably not hiding stuff behind extra config options to make them harder to find. Maybe I've missed the point of what you were trying to do, but it seemed to not meet that.
Well, if you don't want to enable a number of options, it's still better to hide them completely. There are number of options by reorganizing the debug menu a little, it only depends if we're talking here are about a -mm only crutch or something which might be useful to more than a handful of people. A few extra config options are not really a problem as long as they are logically grouped together (instead of having to enable random options all over the place).
bye, Roman
Quoted 1 line
Well, if you don't want to enable a number of options, it's still better to hide them completely. There are number of options by reorganizing the debug menu a little, it only depends if we're talking here are about a -mm only crutch or something which might be useful to more than a handful of people. A few extra config options are not really a problem as long as they are logically grouped together (instead of having to enable random options all over the place).
How is the user meant to know which of your config options a particular option is hidden under?
Same question for the developer as to where to put it?
Seems there are two isses - whether the naming is meaningful or not, and whether you want to hide options under other options,or just flip defaults ... does that sound correct?
M.
Hi,
Quoted 10 lines · Martin J. Bligh
Quoted 7 lines
Well, if you don't want to enable a number of options, it's still better to hide them completely. There are number of options by reorganizing the debug menu a little, it only depends if we're talking here are about a -mm only crutch or something which might be useful to more than a handful of people. A few extra config options are not really a problem as long as they are logically grouped together (instead of having to enable random options all over the place).
How is the user meant to know which of your config options a particular option is hidden under?
I don't quite understand where you want to go with this question. What do you consider an easy to find option? We have thousands of config options, they have to be organized somehow, so they are always hidden somewhere. Currently the debug menu is more a random collection of options, with a little organization it actually can make things easier to find, while still allowing a simple menu for the occasional user and an advanced menu.
Quoted 3 lines
Seems there are two isses - whether the naming is meaningful or not, and whether you want to hide options under other options,or just flip defaults ... does that sound correct?
With my suggestion I was more concentrating on the latter (the general organisation of the menu options) than the actual wording.
bye, Roman
Quoted 6 lines · Ingo Molnar
Quoted 1 line
Adding new runs is easy. Changing the harness is hard ;-)
ok. How about a
CONFIG_DEBUG_NO_OVERHEADoption, that would default to disabled but which you could set to y. Then we could make all the more expensive debug options:
Quoted 7 lines
default y if !CONFIG_DEBUG_NO_OVERHEAD
this would still mean you'd have to turn off
CONFIG_DEBUG_NO_OVERHEAD, but it would be automatically maintainable for you after that initial effort, and we'd be careful to always flag new debugging options with this flag, if they are expensive. And initially i'd define "expensive" as "anything that adds runtime overhead".
the patch below implements this and categorizes all debug options based on whether they have runtime overhead or not.
(i have left out debug options that are non-transparent - i.e. which print lots of stuff to the syslog like DEBUG_KOBJECT.)
Ingo
lib/Kconfig.debug +27 -14
@@ -54,6 +54,15 @@ config DEBUG_KERNEL Say Y here if you are developing drivers or trying to debug and identify kernel problems. +config DEBUG_KERNEL_RUNTIME_OVERHEAD + bool "Enable new debug options by default" + default y + help + Say Y here if you want to have new debugging options + enabled by default even if they cause runtime overhead. + (you can still disable/enable them manually, independently + of this switch) + config LOG_BUF_SHIFT int "Kernel log buffer size (16 => 64KB, 17 => 128KB)" if DEBUG_KERNEL range 12 21
@@ -113,7 +122,7 @@ config DEBUG_SLAB config DEBUG_SLAB_LEAK bool "Slab memory leak debugging" depends on DEBUG_SLAB - default y + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD help Enable /proc/slab_allocators - provides detailed information about which parts of the kernel are using slab objects. May be used for
@@ -122,7 +131,7 @@ config DEBUG_SLAB_LEAK config DEBUG_PREEMPT bool "Debug preemptible kernel" depends on DEBUG_KERNEL && PREEMPT - default y + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD depends on TRACE_IRQFLAGS_SUPPORT help If you say Y here then the kernel will use a debug variant of the
@@ -132,7 +141,7 @@ config DEBUG_PREEMPT config DEBUG_MUTEXES bool "Mutex debugging, basic checks" - default n + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD depends on DEBUG_KERNEL help This feature allows mutex semantics violations to be detected and
@@ -140,7 +149,7 @@ config DEBUG_MUTEXES config DEBUG_MUTEX_ALLOC bool "Detect incorrect freeing of live mutexes" - default n + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD depends on DEBUG_MUTEXES help This feature will check whether any held mutex is incorrectly
@@ -150,7 +159,7 @@ config DEBUG_MUTEX_ALLOC config DEBUG_MUTEX_DEADLOCKS bool "Detect mutex related deadlocks" - default n + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD depends on DEBUG_MUTEXES help This feature will automatically detect and report mutex related
@@ -158,7 +167,7 @@ config DEBUG_MUTEX_DEADLOCKS config DEBUG_RT_MUTEXES bool "RT Mutex debugging, deadlock detection" - default n + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD depends on DEBUG_KERNEL && RT_MUTEXES help This allows rt mutex semantics violations and rt mutex related
@@ -166,7 +175,7 @@ config DEBUG_RT_MUTEXES config DEBUG_PI_LIST bool - default n + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD depends on DEBUG_RT_MUTEXES config RT_MUTEX_TESTER
@@ -179,6 +188,7 @@ config RT_MUTEX_TESTER config DEBUG_SPINLOCK bool "Spinlock debugging" depends on DEBUG_KERNEL + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD help Say Y here and build SMP to catch missing spinlock initialization and certain other kinds of spinlock errors commonly made. This is
@@ -188,7 +198,7 @@ config DEBUG_SPINLOCK config PROVE_SPIN_LOCKING bool "Prove spin-locking correctness" depends on X86 - default n + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD help This feature enables the kernel to prove that all spinlock locking that occurs in the kernel runtime is mathematically
@@ -226,7 +236,7 @@ config PROVE_SPIN_LOCKING config PROVE_RW_LOCKING bool "Prove rw-locking correctness" depends on X86 - default n + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD help This feature enables the kernel to prove that all rwlock locking that occurs in the kernel runtime is mathematically
@@ -264,7 +274,7 @@ config PROVE_RW_LOCKING config PROVE_MUTEX_LOCKING bool "Prove mutex-locking correctness" depends on X86 - default n + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD help This feature enables the kernel to prove that all mutexlock locking that occurs in the kernel runtime is mathematically
@@ -302,7 +312,7 @@ config PROVE_MUTEX_LOCKING config PROVE_RWSEM_LOCKING bool "Prove rwsem-locking correctness" depends on X86 - default n + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD help This feature enables the kernel to prove that all rwsemlock locking that occurs in the kernel runtime is mathematically
@@ -348,7 +358,7 @@ config LOCKDEP config DEBUG_LOCKDEP bool "Lock dependency engine debugging" depends on LOCKDEP - default y + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD depends on TRACE_IRQFLAGS_SUPPORT help If you say Y here, the lock dependency engine will do
@@ -363,6 +373,7 @@ config TRACE_IRQFLAGS config DEBUG_SPINLOCK_SLEEP bool "Sleep-inside-spinlock checking" + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD depends on DEBUG_KERNEL help If you say Y here, various routines which may sleep will become very
@@ -390,6 +401,7 @@ config DEBUG_KOBJECT config DEBUG_HIGHMEM bool "Highmem debugging" depends on DEBUG_KERNEL && HIGHMEM + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD help This options enables addition error checking for high memory systems. Disable for production systems.
@@ -398,7 +410,7 @@ config DEBUG_BUGVERBOSE bool "Verbose BUG() reporting (adds 70K)" if DEBUG_KERNEL && EMBEDDED depends on BUG depends on ARM || ARM26 || M32R || M68K || SPARC32 || SPARC64 || X86_32 || FRV - default !EMBEDDED + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD && !EMBEDDED help Say Y here to make BUG() panics output the file name and line number of the BUG call as well as the EIP and oops trace. This aids
@@ -437,6 +449,7 @@ config DEBUG_FS config DEBUG_VM bool "Debug VM" depends on DEBUG_KERNEL + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD help Enable this to turn on extended checks in the virtual-memory system that may impact performance.
@@ -446,7 +459,7 @@ config DEBUG_VM config FRAME_POINTER bool "Compile the kernel with frame pointers" depends on DEBUG_KERNEL && (X86 || CRIS || M68K || M68KNOMMU || FRV || UML) - default y + default y if DEBUG_KERNEL_RUNTIME_OVERHEAD help If you say Y here the resulting kernel image will be slightly larger and slower, but it might give very useful debugging information on
Hi,
Quoted 11 lines · Ingo Molnar
on one side, the -mm kernel is about showcasing new code and finding bugs in them as fast as possible. Having new debugging options enabled by default is an important part of the testing effort. Users will care more about having no crashes than about having 0.5% more performance in select benchmarks.
on the other side, you obviously dont want a 0.5% overhead for select benchmarks, as that would mess up the history! A very fair and valid position too.
but one side has to give, we cant have both.
As I mentioned before, please keep these defaults as a -mm-only patch, Giving them testing in -mm is fine, but defaults are already way too much abused as is. The default rule should be to enable an option explicitly, if it's needed, it should not be auto-enabled, because its author likes it so much. Using a "default y" should be close to hiding the option via CONFIG_EMBEDDED or some other option and the default should not differ between hidden and visible state, e.g.:
config FOO
bool "foo" if BAR
default y
bye, Roman
Quoted 1 line · Martin J. Bligh
Quoted 1 line
Quoted 1 line
Quoted 1 line
EIP is at
check_deadlock+0x15/0xe0
i took your config and ran a few tests. Your kernel crashed here:
BUG: unable to handle kernel paging request at virtual address 22222232 CPU: 1 EIP: 0060:[<c012b6eb>] Not tainted VLI EFLAGS: 00010002 (2.6.17-rc5-mm1-autokern1 #1)
EIP is at check_deadlock+0x15/0xe0
eax: 22222222 ebx: 00000001 ecx: d4996000 edx: 00000001
esi: d686f550 edi: 22222222 ebp: 22222222 esp: d5bdfec8
ds: 007b es: 007b ss: 0068
Process mkdir09 (pid: 18867, threadinfo=d5bdf000 task=d5c0e000)
Stack: 00000000 d686f550 d3960568 22222222 c012b77b d3960568 d5bdf000 d5bdff00
d5c0e000 c012b922 d5bdff48 d3960568 00000246 c02d50de d5bdff00 d5bdff00
11111111 11111111 d5bdff00 ffffff9c d5bdff48 00000000 d5bdff48 ffffffefwhich is:
(gdb) list *0xc012a26e0xc012a26e is in check_deadlock (kernel/mutex-debug.c:98).
93 struct task_struct *task;
94
95 if (!debug_locks)
96 return 0;
97
98 ti = lock->owner;
99 if (!ti)
100 return 0;
101
102 task = ti->task;
(gdb)
i.e. "lock" was 0x22222222. I have not changed this particular deadlock detection code for some time and it's been stable for many weeks. (the changes in -mm1 were in another area of mutex-debug.c, and even those mostly removed code and didnt add new logic)
the stack looks weird:
Stack: 00000000 d686f550 d3960568 22222222 c012b77b d3960568 d5bdf000 d5bdff00
d5c0e000 c012b922 d5bdff48 d3960568 00000246 c02d50de d5bdff00 d5bdff00
11111111 11111111 d5bdff00 ffffff9c d5bdff48 00000000 d5bdff48 ffffffefwhere does the 0x11111111 and the 0x22222222 come from? It doesnt ring any bells in terms of kernel-internal magic or poison values. My impression is that there's some other badness going on (some sort of memory corruption) and the mutex debugging code just got caught up in the crossfire.
Ingo