Skip to contentSkip to conversation

Clarify CONFIG_DEBUG_INFO's bloaty nature

PATCH In review Borislav Petkov 9 participants 17 messages Feb 3, 2014 – Feb 11, 2014

Subsystems (2)
lore
Share & tools
lore

Borislav Petkov

author maintainer
#0 lore raw

PATCH 1/1 Clarify CONFIG_DEBUG_INFO's bloaty nature

From: Borislav Petkov <[email protected]>

Hold down when it makes sense to enable CONFIG_DEBUG_INFO and when CONFIG_KALLSYMS suffices.

---
 lib/Kconfig.debug | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
lib/Kconfig.debug +5 -1
index dbf94a7d25a8..9095c2078095 100644
@@ -126,7 +126,11 @@ config DEBUG_INFO
 	  This adds debug symbols to the kernel and modules (gcc -g), and
 	  is needed if you intend to use kernel crashdump or binary object
 	  tools like crash, kgdb, LKCD, gdb, etc on the kernel.
-	  Say Y here only if you plan to debug the kernel.
+
+	  If you only want to have resolved symbols in kernel traces and
+	  are not going to need support for those tools above, you don't need
+	  to enable this as it is a huge bloat and build slowdown;
+	  enable CONFIG_KALLSYMS instead.
 
 	  If unsure, say N.
 
Signature · 1 line
1.8.5.2.192.g7794a68
#1 lore raw

lib/Kconfig.debug config DEBUG_INFO · line 132

Quoted 13 lines · Borislav Petkov
lib/Kconfig.debug +4 -1
index dbf94a7d25a8..9095c2078095 100644
@@ -126,7 +126,11 @@ config DEBUG_INFO
 	  This adds debug symbols to the kernel and modules (gcc -g), and
 	  is needed if you intend to use kernel crashdump or binary object
 	  tools like crash, kgdb, LKCD, gdb, etc on the kernel.
-	  Say Y here only if you plan to debug the kernel.
+
+	  If you only want to have resolved symbols in kernel traces and
+	  are not going to need support for those tools above, you don't need
+	  to enable this as it is a huge bloat and build slowdown;

How do you define "huge bloat" if the size of vmlinux doesn't increase? Would some people consider that to be acceptable but now mysteriously confused because we don't know what "huge bloat" you're referring to?

It also begs the question about the meaning of DEBUG_INFO_REDUCED and this comment doesn't even allow us to see that it's an option.

So this doesn't look good.

Quoted 4 lines
lib/Kconfig.debug +1 -0
+	  enable CONFIG_KALLSYMS instead.
 
 	  If unsure, say N.
 
#2 lore raw
Quoted 2 lines · David Rientjes

How do you define "huge bloat" if the size of vmlinux doesn't increase?

Don't be silly. The size of all the object files increase *hugely*.

This is my fs/builtin.o in my normal config:

-rw-rw-r--. 1 torvalds torvalds 2838613 Feb 3 14:32 fs/built-in.o

and with the *reduced* debug info, ie .config diff as follows:

+CONFIG_DEBUG_INFO=y
+CONFIG_DEBUG_INFO_REDUCED=y

I get this:

-rw-rw-r--. 1 torvalds torvalds 11793317 Feb 3 14:33 fs/built-in.o

iow, that "reduced" debug info is still more than 4x bigger than a non-debug-info kernel.

David, if you don't think that debug-info bloats the build, you're in huge *huge* denial. CONFIG_DEBUG_INFO (even with the "reduced" thing) is ABSOLUTELY HORRIBLE CRAP.

Quoted 2 lines

Would some people consider that to be acceptable but now mysteriously confused because we don't know what "huge bloat" you're referring to?

That >4x size increase doesn't match your idea of "huge bloat"?

The object files being much bigger really does screw you especially on laptops that often have less memory and less capable IO subsystems. The final links in particular tend to be much bigger and use much more memory.

I suspect a lot of people are in denial about just how *horrible* the overhead of debug builds are. And yeah, if you have oodles of memory to cache things, it's not too bad. But you really want to have *lots* of memory, because otherwise you won't be caching all those object files in RAM, and your build easily becomes IO bound at link time. A factor of four size difference will not be helping your poor slow laptop disk..

Linus

maintainer
#3 lore raw
Quoted 5 lines · On Mon, 3 Feb 2014 14:47:15 -0800 Linus Torvalds…
Quoted 2 lines · David Rientjes

How do you define "huge bloat" if the size of vmlinux doesn't increase?

Don't be silly. The size of all the object files increase *hugely*.

yup, I disable this in my allmodconfig testing, to great effect.

That being said, I do think the text should make clear that the bloat is a compile-time impact and not a runtime one. Something like

lib/Kconfig.debug renamed from lib/Kconfig.debug~lib-kconfigdebug-clarify-config_debug_infos-bloaty-nature-fix +3 -3
@@ -128,9 +128,9 @@ config DEBUG_INFO
 	  tools like crash, kgdb, LKCD, gdb, etc on the kernel.
 
 	  If you only want to have resolved symbols in kernel traces and
-	  are not going to need support for those tools above, you don't need
-	  to enable this as it is a huge bloat and build slowdown;
-	  enable CONFIG_KALLSYMS instead.
+	  are not going to need support for the above tools, you don't need
+	  to enable this.  It hugely bloat object files' on-disk sizes and slows
+	  the build.  Enable CONFIG_KALLSYMS instead.
 
 	  If unsure, say N.
 

_

#4 lore raw
Quoted 14 lines · Andrew Morton
Quoted 3 lines
Quoted 1 line

How do you define "huge bloat" if the size of vmlinux doesn't increase?

Don't be silly. The size of all the object files increase *hugely*.

yup, I disable this in my allmodconfig testing, to great effect.

That being said, I do think the text should make clear that the bloat is a compile-time impact and not a runtime one. Something like

lib/Kconfig.debug renamed from lib/Kconfig.debug~lib-kconfigdebug-clarify-config_debug_infos-bloaty-nature-fix +0 -0
@@ -128,9 +128,9 @@ config DEBUG_INFO
 	  tools like crash, kgdb, LKCD, gdb, etc on the kernel.
 

lib/Kconfig.debug config DEBUG_INFO · line 136

Quoted 10 lines
lib/Kconfig.debug renamed from lib/Kconfig.debug~lib-kconfigdebug-clarify-config_debug_infos-bloaty-nature-fix +3 -3
 	  If you only want to have resolved symbols in kernel traces and
-	  are not going to need support for those tools above, you don't need
-	  to enable this as it is a huge bloat and build slowdown;
-	  enable CONFIG_KALLSYMS instead.
+	  are not going to need support for the above tools, you don't need
+	  to enable this.  It hugely bloat object files' on-disk sizes and slows
+	  the build.  Enable CONFIG_KALLSYMS instead.
 
 	  If unsure, say N.
 

Um, I still don't think this is right since CONFIG_DEBUG_INFO_REDUCED requires you to say yes here and that config option actually specifies the tools for which it is effective. Are you suggesting that we decouple CONFIG_DEBUG_INFO from CONFIG_DEBUG_INFO_REDUCED and make CONFIG_DEBUG_INFO select the reduced variant? Or... what?

#5 lore raw
Quoted 24 lines · David Rientjes
Quoted 22 lines · Andrew Morton
Quoted 3 lines
Quoted 1 line

How do you define "huge bloat" if the size of vmlinux doesn't increase?

Don't be silly. The size of all the object files increase *hugely*.

yup, I disable this in my allmodconfig testing, to great effect.

That being said, I do think the text should make clear that the bloat is a compile-time impact and not a runtime one. Something like

lib/Kconfig.debug renamed from lib/Kconfig.debug~lib-kconfigdebug-clarify-config_debug_infos-bloaty-nature-fix +3 -3
@@ -128,9 +128,9 @@ config DEBUG_INFO
 	  tools like crash, kgdb, LKCD, gdb, etc on the kernel.
 
 	  If you only want to have resolved symbols in kernel traces and
-	  are not going to need support for those tools above, you don't need
-	  to enable this as it is a huge bloat and build slowdown;
-	  enable CONFIG_KALLSYMS instead.
+	  are not going to need support for the above tools, you don't need
+	  to enable this.  It hugely bloat object files' on-disk sizes and slows
+	  the build.  Enable CONFIG_KALLSYMS instead.
 
Quoted 9 lines

lib/Kconfig.debug config DEBUG_INFO · line 136

Quoted 2 lines
lib/Kconfig.debug renamed from lib/Kconfig.debug~lib-kconfigdebug-clarify-config_debug_infos-bloaty-nature-fix +0 -0
 	  If unsure, say N.
 

Um, I still don't think this is right since CONFIG_DEBUG_INFO_REDUCED requires you to say yes here and that config option actually specifies the tools for which it is effective. Are you suggesting that we decouple CONFIG_DEBUG_INFO from CONFIG_DEBUG_INFO_REDUCED and make CONFIG_DEBUG_INFO select the reduced variant? Or... what?

I think the right solution is what I suggested in another recent discussion:

"So to reign in debuginfo in allyesconfig/allmodconfig builds we could
do something like:

config SAVE_TIME_AND_DISK_SPACE
bool "Faster and leaner build object files: compile without debug info" default y

choice
prompt "Choose DEBUGINFO bloat level" depends on !SAVE_TIME_AND_DISK_SPACE default DEBUG_INFO_REDUCED
help
This option allows to select the debuginfo model.

config DEBUG_INFO_REDUCED
bool "Reduced amount of debugging information, 4x regular .o size"

config DEBUG_INFO

        bool "Full debuginfo, 10x regular .o size, 'planet killer' version"

endchoice

That would default to Y and would disable debuginfo by default."

Thanks,

Ingo

#6 lore raw
Quoted 4 lines · Ingo Molnar

config SAVE_TIME_AND_DISK_SPACE
bool "Faster and leaner build object files: compile without debug info" default y

Thinking about it, we already have this. It's called "CONFIG_COMPILE_TEST".

It very much gets set by all{yes,mod}config, and it also says "we're interested in testing the compile, not so much running the result".

So let's just make the DEBUG_INFO depend on !COMPILE_TEST.

Trivial oneliner patch (independent of the discussion about improving the documentation), so I'll just apply it.

I can't imagine that this would be controversial:

config DEBUG_INFO
bool "Compile the kernel with debug info"
- depends on DEBUG_KERNEL
+ depends on DEBUG_KERNEL && !COMPILE_TEST
help

Hm?

Linus

#7 lore raw
Quoted 6 lines · Linus Torvalds

Trivial oneliner patch (independent of the discussion about improving the documentation), so I'll just apply it.

I can't imagine that this would be controversial:

Quoted 7 lines

config DEBUG_INFO
bool "Compile the kernel with debug info"
- depends on DEBUG_KERNEL
+ depends on DEBUG_KERNEL && !COMPILE_TEST
help

Hm?

I have used debug info on allyesconfig in the past to run statistics on the resulting binary. It's still useful, even though quite slow. Make it depend on !EXPERT or somesuch? I agree for normal usages it's fine.

BTW you probably also want to do the same for the x86 instruction decoder test. It seems to be the biggest compile time waster in allyesconfig and I usually disable it too.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
#8 lore raw
Quoted 4 lines · Andi Kleen

I have used debug info on allyesconfig in the past to run statistics on the resulting binary. It's still useful, even though quite slow. Make it depend on !EXPERT or somesuch? I agree for normal usages it's fine.

EXPERT is useless, people set it randomly.

I'd suggest that for any statistics gathering, you just disable COMPILE_TEST. You can do that by just editing the .config file after doing the allyesconfig, or by pre-seeding the allyesconfig with COMPILE_TEST disabled.

Your use-case is rather special, after all. I do *not* believe that we should make one of the goals of "make all{yes,mod}config" be that kind of odd specialized thing.

Quoted 3 lines

BTW you probably also want to do the same for the x86 instruction decoder test. It seems to be the biggest compile time waster in allyesconfig and I usually disable it too.

Agreed, that should be disabled too by default. Although in that case I think the COMPILE_TEST combination makes more sense, since it's a coverage issue.

Suggestions for symbols?

Linus

#9 lore raw

PATCH 1/1 x86: Disable CONFIG_X86_DECODER_SELFTEST in allmod/allyesconfigs

Quoted 7 lines · Linus Torvalds
Quoted 3 lines

BTW you probably also want to do the same for the x86 instruction decoder test. It seems to be the biggest compile time waster in allyesconfig and I usually disable it too.

Agreed, that should be disabled too by default. Although in that case I think the COMPILE_TEST combination makes more sense, since it's a coverage issue.

I'd say randconfig will cover it often enough, and the failure is also borderline build coverage related: you cannot really make the decoder test fail via source level changes, only with changes in the build environment, so I agree with Andi that we can disable this one too - via the patch below.

Thanks,

Ingo

arch/x86/Kconfig.debug +5 -0
index 0f3621e..f0fca39 100644
@@ -184,6 +184,11 @@ config HAVE_MMIOTRACE_SUPPORT
 config X86_DECODER_SELFTEST
 	bool "x86 instruction decoder selftest"
 	depends on DEBUG_KERNEL && KPROBES
+	#
+	# It can take some time to validate the image,
+	# make sure {allyes|allmod}config doesn't enable it:
+	#
+	depends on !COMPILE_TEST
 	---help---
 	 Perform x86 instruction decoder selftests at build time.
 	 This option is useful for checking the sanity of x86 instruction
#10 lore raw

Re: [PATCH] x86: Disable CONFIG_X86_DECODER_SELFTEST in allmod/allyesconfigs

Quoted 11 lines · Ingo Molnar
Quoted 7 lines · Linus Torvalds
Quoted 3 lines

BTW you probably also want to do the same for the x86 instruction decoder test. It seems to be the biggest compile time waster in allyesconfig and I usually disable it too.

Agreed, that should be disabled too by default. Although in that case I think the COMPILE_TEST combination makes more sense, since it's a coverage issue.

Quoted 5 lines

I'd say randconfig will cover it often enough, and the failure is also borderline build coverage related: you cannot really make the decoder test fail via source level changes, only with changes in the build environment, so I agree with Andi that we can disable this one too - via the patch below.

For the record, even with a (currently) modern CPU such as a Xeon E5-2630 @ 2.30GHz, the script takes 4m30s on 3.14-rc1 i386 allyesconfig. And since it runs single threaded at the end of the build, it really adds that amount to the wall time spent on the build. So enthusiastically Acked, FWIW.

Paul.
--

Quoted 28 lines

Thanks,

Ingo

arch/x86/Kconfig.debug +5 -1
index 0f3621e..f0fca39 100644
@@ -184,6 +184,11 @@ config HAVE_MMIOTRACE_SUPPORT
 config X86_DECODER_SELFTEST
        bool "x86 instruction decoder selftest"
        depends on DEBUG_KERNEL && KPROBES
+       #
+       # It can take some time to validate the image,
+       # make sure {allyes|allmod}config doesn't enable it:
+       #
+       depends on !COMPILE_TEST
        ---help---
         Perform x86 instruction decoder selftests at build time.
         This option is useful for checking the sanity of x86 instruction
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#11 lore raw

Re: [PATCH] x86: Disable CONFIG_X86_DECODER_SELFTEST in allmod/allyesconfigs

Quoted 5 lines

For the record, even with a (currently) modern CPU such as a Xeon E5-2630 @ 2.30GHz, the script takes 4m30s on 3.14-rc1 i386 allyesconfig. And since it runs single threaded at the end of the build, it really adds that amount to the wall time spent on the build. So enthusiastically Acked, FWIW.

I suppose we could just multi thread it. It should "embarassingly" scale without problems. But turning it off by default is a first step.

-Andi

#12 lore raw

[tip:x86/urgent] x86/debug: Disable CONFIG_X86_DECODER_SELFTEST in allmod/allyesconfigs

In reply to Ingo Molnar
x86/debug: Disable CONFIG_X86_DECODER_SELFTEST in allmod/allyesconfigs
Commit-ID:  8c778d541bec8a41cbd8fa73b7cba90b2d6606c1
Gitweb:     http://git.kernel.org/tip/8c778d541bec8a41cbd8fa73b7cba90b2d6606c1
Author:     Ingo Molnar <[email protected]>
AuthorDate: Wed, 5 Feb 2014 06:51:37 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sun, 9 Feb 2014 15:18:42 +0100

x86/debug: Disable CONFIG_X86_DECODER_SELFTEST in allmod/allyesconfigs

* Linus Torvalds <[email protected]> wrote:

Andi Kleen suggested to turn off CONFIG_X86_DECODER_SELFTEST=y in allyesconfig, because it costs a lot of CPU time.

(In practice randconfig builds will cover this test frequently
enough to provide build coverage.)

---
 arch/x86/Kconfig.debug | 5 +++++
 1 file changed, 5 insertions(+)
arch/x86/Kconfig.debug +5 -0
index 321a52c..b542ec1 100644
@@ -185,6 +185,11 @@ config X86_DECODER_SELFTEST
 	bool "x86 instruction decoder selftest"
 	depends on DEBUG_KERNEL && KPROBES
 	depends on !COMPILE_TEST
+	#
+	# It can take some time to validate the image,
+	# make sure {allyes|allmod}config doesn't enable it:
+	#
+	depends on !COMPILE_TEST
 	---help---
 	 Perform x86 instruction decoder selftests at build time.
 	 This option is useful for checking the sanity of x86 instruction
#13 lore raw

Re: [PATCH] x86: Disable CONFIG_X86_DECODER_SELFTEST in allmod/allyesconfigs

In reply to Ingo Molnar

Hi Ingo,

Quoted 30 lines · On Wed, 5 Feb 2014 06:51:37 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 7 lines · Linus Torvalds
Quoted 3 lines

BTW you probably also want to do the same for the x86 instruction decoder test. It seems to be the biggest compile time waster in allyesconfig and I usually disable it too.

Agreed, that should be disabled too by default. Although in that case I think the COMPILE_TEST combination makes more sense, since it's a coverage issue.

I'd say randconfig will cover it often enough, and the failure is also borderline build coverage related: you cannot really make the decoder test fail via source level changes, only with changes in the build environment, so I agree with Andi that we can disable this one too - via the patch below.

Thanks,

Ingo

arch/x86/Kconfig.debug +0 -0
index 0f3621e..f0fca39 100644
@@ -184,6 +184,11 @@ config HAVE_MMIOTRACE_SUPPORT
 config X86_DECODER_SELFTEST

arch/x86/Kconfig.debug config HAVE_MMIOTRACE_SUPPORT · line 194

Quoted 10 lines
arch/x86/Kconfig.debug +5 -0
 	bool "x86 instruction decoder selftest"
 	depends on DEBUG_KERNEL && KPROBES
+	#
+	# It can take some time to validate the image,
+	# make sure {allyes|allmod}config doesn't enable it:
+	#
+	depends on !COMPILE_TEST
 	---help---
 	 Perform x86 instruction decoder selftests at build time.
 	 This option is useful for checking the sanity of x86 instruction

When this got applied to the tip tree, Linus had already applied an earlier version as commit f8f202348208, and this one (commit 8c778d541bec) now duplicates the actual dependency in !COMPILE_TEST (but does add a useful comment) ...

Signature · 2 lines
Cheers,
Stephen Rothwell <[email protected]>
#14 lore raw

Re: [PATCH] x86: Disable CONFIG_X86_DECODER_SELFTEST in allmod/allyesconfigs

Quoted 40 lines · Stephen Rothwell

Hi Ingo,

Quoted 37 lines · On Wed, 5 Feb 2014 06:51:37 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 7 lines · Linus Torvalds
Quoted 3 lines

BTW you probably also want to do the same for the x86 instruction decoder test. It seems to be the biggest compile time waster in allyesconfig and I usually disable it too.

Agreed, that should be disabled too by default. Although in that case I think the COMPILE_TEST combination makes more sense, since it's a coverage issue.

I'd say randconfig will cover it often enough, and the failure is also borderline build coverage related: you cannot really make the decoder test fail via source level changes, only with changes in the build environment, so I agree with Andi that we can disable this one too - via the patch below.

Thanks,

Ingo

arch/x86/Kconfig.debug +5 -0
index 0f3621e..f0fca39 100644
@@ -184,6 +184,11 @@ config HAVE_MMIOTRACE_SUPPORT
 config X86_DECODER_SELFTEST
 	bool "x86 instruction decoder selftest"
 	depends on DEBUG_KERNEL && KPROBES
+	#
+	# It can take some time to validate the image,
+	# make sure {allyes|allmod}config doesn't enable it:
+	#
+	depends on !COMPILE_TEST
Quoted 8 lines

arch/x86/Kconfig.debug config HAVE_MMIOTRACE_SUPPORT · line 194

Quoted 3 lines
arch/x86/Kconfig.debug +0 -0
 	---help---
 	 Perform x86 instruction decoder selftests at build time.
 	 This option is useful for checking the sanity of x86 instruction

When this got applied to the tip tree, Linus had already applied an earlier version as commit f8f202348208, and this one (commit 8c778d541bec) now duplicates the actual dependency in !COMPILE_TEST (but does add a useful comment) ...

Ok, I've zapped it from x86/urgent, thanks for letting me know!

Ingo

#15 lore raw
In reply to Linus Torvalds
Quoted 18 lines · Linus Torvalds
Quoted 4 lines · Ingo Molnar

config SAVE_TIME_AND_DISK_SPACE
bool "Faster and leaner build object files: compile without debug info" default y

Thinking about it, we already have this. It's called "CONFIG_COMPILE_TEST".

It very much gets set by all{yes,mod}config, and it also says "we're interested in testing the compile, not so much running the result".

So let's just make the DEBUG_INFO depend on !COMPILE_TEST.

Trivial oneliner patch (independent of the discussion about improving the documentation), so I'll just apply it.

I can't imagine that this would be controversial:

Quoted 7 lines

config DEBUG_INFO
bool "Compile the kernel with debug info"
- depends on DEBUG_KERNEL
+ depends on DEBUG_KERNEL && !COMPILE_TEST
help

Hm?

Fully agreed.

Thanks,

Ingo

author maintainer
#16 lore raw
In reply to Andrew Morton
Quoted 16 lines · Andrew Morton
Quoted 5 lines · On Mon, 3 Feb 2014 14:47:15 -0800 Linus Torvalds…
Quoted 2 lines · David Rientjes

How do you define "huge bloat" if the size of vmlinux doesn't increase?

Don't be silly. The size of all the object files increase *hugely*.

yup, I disable this in my allmodconfig testing, to great effect.

That being said, I do think the text should make clear that the bloat is a compile-time impact and not a runtime one. Something like

lib/Kconfig.debug config DEBUG_INFO · line 133

Quoted 9 lines
lib/Kconfig.debug renamed from lib/Kconfig.debug~lib-kconfigdebug-clarify-config_debug_infos-bloaty-nature-fix +3 -3
@@ -128,9 +128,9 @@ config DEBUG_INFO
 	  tools like crash, kgdb, LKCD, gdb, etc on the kernel.
 
 	  If you only want to have resolved symbols in kernel traces and
-	  are not going to need support for those tools above, you don't need
-	  to enable this as it is a huge bloat and build slowdown;
-	  enable CONFIG_KALLSYMS instead.
+	  are not going to need support for the above tools, you don't need
+	  to enable this.  It hugely bloat object files' on-disk sizes and slows
+	  the build.  Enable CONFIG_KALLSYMS instead.

Yes, this is better.

Andrew, can you add that or you want me to send a new version?

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--