kernel hacking: GCC optimization for better debug experience (-Og)

PATCH Superseded Du Changbin 2 participants 14 messages Oct 19, 2018 – Oct 29, 2018

Subsystems (5)

Arm (32-bit) x86 Kbuild Library code +1 more subsystems

lore

Share & tools

lore

Du Changbin

author

2018-10-19 12:49 UTC #0 lore raw

v2 cover kernel hacking: GCC optimization for better debug experience (-Og)

Hi all,
I have posted this series several months ago but interrupted by personal affairs. Now I get time to complete this task. Thanks for all of the reviewers.

I know some kernel developers was searching for a method to dissable GCC optimizations, probably they want to apply GCC '-O0' option. But since Linux kernel relies on GCC optimization to remove some dead code, so '-O0' just breaks the build. They do need this because they want to debug kernel with qemu, simics, kgtp or kgdb.

Thanks for the GCC '-Og' optimization level introduced in GCC 4.8, which offers a reasonable level of optimization while maintaining fast compilation and a good debugging experience. It is similar to '-O1' while perferring to keep debug ability over runtime speed. With '-Og', we can build a kernel with better debug ability and little performance drop after some simple change.

In this series, firstly introduce a new config CONFIG_NO_AUTO_INLINE after two fixes for this new option. With this option, only functions explicitly marked with "inline" will be inlined. This will allow the function tracer to trace more functions because it only traces functions that the compiler has not inlined.

Then introduce new config CC_OPTIMIZE_FOR_DEBUGGING which apply '-Og' optimization level for whole kernel, with a simple fix in fix_to_virt(). Currently I have only tested this option on x86 and ARM platform. Other platforms should also work but probably need some compiling fixes as what having done in this series. I leave that to who want to try this debug option.

Comparison of vmlinux size: a bit smaller.

w/o CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
$ size vmlinux
text data bss dec hex filename
22665554 9709674 2920908 35296136 21a9388 vmlinux

w/ CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
$ size vmlinux
text data bss dec hex filename
21499032 10102758 2920908 34522698 20ec64a vmlinux

Comparison of system performance: a bit drop (~6%).

    This benchmark of kernel compilation is suggested by Ingo Molnar.
    https://lkml.org/lkml/2018/5/2/74

    Preparation: Set cpufreq to 'performance'.
    for ((cpu=0; cpu<120; cpu++)); do
      G=/sys/devices/system/cpu/cpu$cpu/cpufreq/scaling_governor
      [ -f $G ] && echo performance > $G
    done

    w/o CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
    $ perf stat --repeat 5 --null --pre                 '\
        cp -a kernel ../kernel.copy.$(date +%s);         \
        rm -rf *;                                        \
        git checkout .;                                  \
        echo 1 > /proc/sys/vm/drop_caches;               \
        find ../kernel* -type f | xargs cat >/dev/null;  \
        make -j kernel >/dev/null;                       \
        make clean >/dev/null 2>&1;                      \
        sync                                            '\
                                                         \
        make -j8 >/dev/null

     Performance counter stats for 'make -j8' (5 runs):

        219.764246652 seconds time elapsed                   ( +-  0.78% )

    w/ CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
    $ perf stat --repeat 5 --null --pre                 '\
        cp -a kernel ../kernel.copy.$(date +%s);         \
        rm -rf *;                                        \
        git checkout .;                                  \
        echo 1 > /proc/sys/vm/drop_caches;               \
        find ../kernel* -type f | xargs cat >/dev/null;  \
        make -j kernel >/dev/null;                       \
        make clean >/dev/null 2>&1;                      \
        sync                                            '\
                                                         \
        make -j8 >/dev/null

    Performance counter stats for 'make -j8' (5 runs):

         233.574187771 seconds time elapsed                  ( +-  0.19% )

Du Changbin (4):
x86/mm: surround level4_kernel_pgt with #ifdef

    CONFIG_X86_5LEVEL...#endif

kernel hacking: new config NO_AUTO_INLINE to disable compiler
auto-inline optimizations
ARM: mm: fix build error in fix_to_virt with
CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
kernel hacking: new config CC_OPTIMIZE_FOR_DEBUGGING to apply GCC -Og
optimization

 Makefile                          | 11 +++++++++++
 arch/arm/mm/mmu.c                 |  2 +-
 arch/x86/include/asm/pgtable_64.h |  2 ++
 arch/x86/kernel/head64.c          | 13 ++++++-------
 include/linux/compiler-gcc.h      |  2 +-
 include/linux/compiler.h          |  2 +-
 init/Kconfig                      | 19 +++++++++++++++++++
 lib/Kconfig.debug                 | 17 +++++++++++++++++
 8 files changed, 58 insertions(+), 10 deletions(-)

Signature · 1 line

2.17.1

Du Changbin

author

2018-10-19 12:49 UTC #1 lore raw

PATCH v2 1/4 x86/mm: surround level4_kernel_pgt with #ifdef CONFIG_X86_5LEVEL...#endif

The level4_kernel_pgt is only defined when X86_5LEVEL is enabled. So surround level4_kernel_pgt with #ifdef CONFIG_X86_5LEVEL...#endif to make code correct.

Signed-off-by Du Changbin <[email protected]>

Acked-by Steven Rostedt (VMware) <[email protected]>

---
 arch/x86/include/asm/pgtable_64.h |  2 ++
 arch/x86/kernel/head64.c          | 13 ++++++-------
 2 files changed, 8 insertions(+), 7 deletions(-)

arch/x86/include/asm/pgtable_64.h +2 -0

index 9c85b54bf03c..9333f7fa5bdb 100644
@@ -16,7 +16,9 @@
 #include <linux/threads.h>
 #include <asm/fixmap.h>
 
+#ifdef CONFIG_X86_5LEVEL
 extern p4d_t level4_kernel_pgt[512];
+#endif
 extern p4d_t level4_ident_pgt[512];
 extern pud_t level3_kernel_pgt[512];
 extern pud_t level3_ident_pgt[512];

arch/x86/kernel/head64.c +6 -7

index ddee1f0870c4..4a59ef93c258 100644
@@ -151,16 +151,15 @@ unsigned long __head __startup_64(unsigned long physaddr,
 
 	pgd = fixup_pointer(&early_top_pgt, physaddr);
 	p = pgd + pgd_index(__START_KERNEL_map);
-	if (la57)
-		*p = (unsigned long)level4_kernel_pgt;
-	else
-		*p = (unsigned long)level3_kernel_pgt;
-	*p += _PAGE_TABLE_NOENC - __START_KERNEL_map + load_delta;
-
+#ifdef CONFIG_X86_5LEVEL
 	if (la57) {
+		*p = (unsigned long)level4_kernel_pgt;
 		p4d = fixup_pointer(&level4_kernel_pgt, physaddr);
 		p4d[511] += load_delta;
-	}
+	} else
+#endif
+		*p = (unsigned long)level3_kernel_pgt;
+	*p += _PAGE_TABLE_NOENC - __START_KERNEL_map + load_delta;
 
 	pud = fixup_pointer(&level3_kernel_pgt, physaddr);
 	pud[510] += load_delta;

Signature · 1 line

2.17.1

Masahiro Yamada

2018-10-28 01:44 UTC #2 lore raw

Re: [PATCH v2 1/4] x86/mm: surround level4_kernel_pgt with #ifdef CONFIG_X86_5LEVEL...#endif

Quoted 4 lines · Du Changbin

The level4_kernel_pgt is only defined when X86_5LEVEL is enabled. So surround level4_kernel_pgt with #ifdef CONFIG_X86_5LEVEL...#endif to make code correct.

For clarification, is it better to mention
that this is a preparation for CONFIG_CC_OPTIMIZE_FOR_DEBUGGING ?

Quoted 12 lines

Signed-off-by Du Changbin <[email protected]>
Acked-by Steven Rostedt (VMware) <[email protected]>
---
 arch/x86/include/asm/pgtable_64.h |  2 ++
 arch/x86/kernel/head64.c          | 13 ++++++-------
 2 files changed, 8 insertions(+), 7 deletions(-)
arch/x86/include/asm/pgtable_64.h +0 -0
index 9c85b54bf03c..9333f7fa5bdb 100644

arch/x86/include/asm/pgtable_64.h line 21

Quoted 6 lines

arch/x86/include/asm/pgtable_64.h +2 -0

@@ -16,7 +16,9 @@
 #include <linux/threads.h>
 #include <asm/fixmap.h>
 
+#ifdef CONFIG_X86_5LEVEL
 extern p4d_t level4_kernel_pgt[512];
+#endif

Is this #ifdef necessary?

It is harmless to declaring unused stuff.

Quoted 31 lines

arch/x86/include/asm/pgtable_64.h +0 -0

 extern p4d_t level4_ident_pgt[512];
 extern pud_t level3_kernel_pgt[512];
 extern pud_t level3_ident_pgt[512];

arch/x86/kernel/head64.c +6 -8

index ddee1f0870c4..4a59ef93c258 100644
@@ -151,16 +151,15 @@ unsigned long __head __startup_64(unsigned long physaddr,
 
        pgd = fixup_pointer(&early_top_pgt, physaddr);
        p = pgd + pgd_index(__START_KERNEL_map);
-       if (la57)
-               *p = (unsigned long)level4_kernel_pgt;
-       else
-               *p = (unsigned long)level3_kernel_pgt;
-       *p += _PAGE_TABLE_NOENC - __START_KERNEL_map + load_delta;
-
+#ifdef CONFIG_X86_5LEVEL
        if (la57) {
+               *p = (unsigned long)level4_kernel_pgt;
                p4d = fixup_pointer(&level4_kernel_pgt, physaddr);
                p4d[511] += load_delta;
-       }
+       } else
+#endif
+               *p = (unsigned long)level3_kernel_pgt;
+       *p += _PAGE_TABLE_NOENC - __START_KERNEL_map + load_delta;
 
        pud = fixup_pointer(&level3_kernel_pgt, physaddr);
        pud[510] += load_delta;
--

Quoted 2 lines

2.17.1

Hmm, this code looks a bit ugly...

Does the following one liner work with CONFIG_CC_OPTIMIZE_FOR_DEBUGGING ?

arch/x86/kernel/head64.c +1 -1

index 8047379..579847f 100644
@@ -97,7 +97,7 @@ static bool __head check_la57_support(unsigned long physaddr)
        return true;
 }
 #else
-static bool __head check_la57_support(unsigned long physaddr)
+static __always_inline bool __head check_la57_support(unsigned long physaddr)
 {
        return false;
 }

--
Best Regards
Masahiro Yamada

Changbin Du

author

2018-10-28 12:46 UTC #3 lore raw

Re: [PATCH v2 1/4] x86/mm: surround level4_kernel_pgt with #ifdef CONFIG_X86_5LEVEL...#endif

Quoted 9 lines · Masahiro Yamada

Quoted 4 lines · Du Changbin
The level4_kernel_pgt is only defined when X86_5LEVEL is enabled. So surround level4_kernel_pgt with #ifdef CONFIG_X86_5LEVEL...#endif to make code correct.
For clarification, is it better to mention
that this is a preparation for CONFIG_CC_OPTIMIZE_FOR_DEBUGGING ?

yes, I will update the commit msg.

Quoted 73 lines

Quoted 13 lines

Signed-off-by Du Changbin <[email protected]>
Acked-by Steven Rostedt (VMware) <[email protected]>
---
 arch/x86/include/asm/pgtable_64.h |  2 ++
 arch/x86/kernel/head64.c          | 13 ++++++-------
 2 files changed, 8 insertions(+), 7 deletions(-)
arch/x86/include/asm/pgtable_64.h +0 -0
index 9c85b54bf03c..9333f7fa5bdb 100644

arch/x86/include/asm/pgtable_64.h line 21

Quoted 6 lines

arch/x86/include/asm/pgtable_64.h +2 -0

@@ -16,7 +16,9 @@
 #include <linux/threads.h>
 #include <asm/fixmap.h>
 
+#ifdef CONFIG_X86_5LEVEL
 extern p4d_t level4_kernel_pgt[512];
+#endif

Is this #ifdef necessary?

It is harmless to declaring unused stuff.

Quoted 31 lines

arch/x86/include/asm/pgtable_64.h +0 -0

 extern p4d_t level4_ident_pgt[512];
 extern pud_t level3_kernel_pgt[512];
 extern pud_t level3_ident_pgt[512];

arch/x86/kernel/head64.c +6 -8

index ddee1f0870c4..4a59ef93c258 100644
@@ -151,16 +151,15 @@ unsigned long __head __startup_64(unsigned long physaddr,
 
        pgd = fixup_pointer(&early_top_pgt, physaddr);
        p = pgd + pgd_index(__START_KERNEL_map);
-       if (la57)
-               *p = (unsigned long)level4_kernel_pgt;
-       else
-               *p = (unsigned long)level3_kernel_pgt;
-       *p += _PAGE_TABLE_NOENC - __START_KERNEL_map + load_delta;
-
+#ifdef CONFIG_X86_5LEVEL
        if (la57) {
+               *p = (unsigned long)level4_kernel_pgt;
                p4d = fixup_pointer(&level4_kernel_pgt, physaddr);
                p4d[511] += load_delta;
-       }
+       } else
+#endif
+               *p = (unsigned long)level3_kernel_pgt;
+       *p += _PAGE_TABLE_NOENC - __START_KERNEL_map + load_delta;
 
        pud = fixup_pointer(&level3_kernel_pgt, physaddr);
        pud[510] += load_delta;
--

Quoted 2 lines

2.17.1

Hmm, this code looks a bit ugly...

Does the following one liner work with CONFIG_CC_OPTIMIZE_FOR_DEBUGGING ?

arch/x86/kernel/head64.c +0 -0

index 8047379..579847f 100644

arch/x86/kernel/head64.c static bool __head check_la57_support(unsigned long physaddr) · line 103

Quoted 8 lines

arch/x86/kernel/head64.c +1 -1

@@ -97,7 +97,7 @@ static bool __head check_la57_support(unsigned long physaddr)
        return true;
 }
 #else
-static bool __head check_la57_support(unsigned long physaddr)
+static __always_inline bool __head check_la57_support(unsigned long physaddr)
 {
        return false;
 }

This is much better. I just declared it as 'inline'. Thanks for your suggestion.

Quoted 6 lines

--
Best Regards
Masahiro Yamada

Signature · 2 lines

Thanks,
Changbin Du

Du Changbin

author

2018-10-19 12:49 UTC #4 lore raw

PATCH v2 2/4 kernel hacking: new config NO_AUTO_INLINE to disable compiler auto-inline optimizations

In reply to Du Changbin

This patch add a new kernel hacking option NO_AUTO_INLINE. Selecting this option will prevent the compiler from optimizing the kernel by auto-inlining functions not marked with the inline keyword.

With this option, only functions explicitly marked with "inline" will be inlined. This will allow the function tracer to trace more functions because it only traces functions that the compiler has not inlined.

Signed-off-by Du Changbin <[email protected]>

Acked-by Steven Rostedt (VMware) <[email protected]>

---
 Makefile          |  6 ++++++
 lib/Kconfig.debug | 17 +++++++++++++++++
 2 files changed, 23 insertions(+)

Makefile +6 -0

index e8b599b4dcde..757d6507cb5c 100644
@@ -749,6 +749,12 @@ KBUILD_CFLAGS 	+= $(call cc-option, -femit-struct-debug-baseonly) \
 		   $(call cc-option,-fno-var-tracking)
 endif
 
+ifdef CONFIG_NO_AUTO_INLINE
+KBUILD_CFLAGS   += $(call cc-option, -fno-inline-functions) \
+		   $(call cc-option, -fno-inline-small-functions) \
+		   $(call cc-option, -fno-inline-functions-called-once)
+endif
+
 ifdef CONFIG_FUNCTION_TRACER
 ifdef CONFIG_FTRACE_MCOUNT_RECORD
   # gcc 5 supports generating the mcount tables directly

lib/Kconfig.debug +17 -0

index 4966c4fbe7f7..c7c28ee01dfc 100644
@@ -211,6 +211,23 @@ config GDB_SCRIPTS
 	  instance. See Documentation/dev-tools/gdb-kernel-debugging.rst
 	  for further details.
 
+config NO_AUTO_INLINE
+	bool "Disable compiler auto-inline optimizations"
+	help
+	  This will prevent the compiler from optimizing the kernel by
+	  auto-inlining functions not marked with the inline keyword.
+	  With this option, only functions explicitly marked with
+	  "inline" will be inlined. This will allow the function tracer
+	  to trace more functions because it only traces functions that
+	  the compiler has not inlined.
+
+	  Enabling this function can help debugging a kernel if using
+	  the function tracer. But it can also change how the kernel
+	  works, because inlining functions may change the timing,
+	  which could make it difficult while debugging race conditions.
+
+	  If unsure, select N.
+
 config ENABLE_MUST_CHECK
 	bool "Enable __must_check logic"
 	default y

Signature · 1 line

2.17.1

Masahiro Yamada

2018-10-28 13:11 UTC #5 lore raw

Re: [PATCH v2 2/4] kernel hacking: new config NO_AUTO_INLINE to disable compiler auto-inline optimizations

Quoted 11 lines · Du Changbin

This patch add a new kernel hacking option NO_AUTO_INLINE. Selecting this option will prevent the compiler from optimizing the kernel by auto-inlining functions not marked with the inline keyword.
With this option, only functions explicitly marked with "inline" will be inlined. This will allow the function tracer to trace more functions because it only traces functions that the compiler has not inlined.
Signed-off-by Du Changbin <[email protected]>
Acked-by Steven Rostedt (VMware) <[email protected]>

This subject
"kernel hacking: new config NO_AUTO_INLINE to disable compiler auto-inline optimizations"
is also too long.

Could you please make it a bit shorter?

Signature · 2 lines

Best Regards
Masahiro Yamada

Du Changbin

author

2018-10-19 12:49 UTC #6 lore raw

PATCH v2 3/4 ARM: mm: fix build error in fix_to_virt with CONFIG_CC_OPTIMIZE_FOR_DEBUGGING

In reply to Du Changbin

With '-Og' optimization level, GCC would not optimize a count for a loop as a constant value. But BUILD_BUG_ON() only accept compile-time constant values. Let's use __fix_to_virt() to avoid the error.

arch/arm/mm/mmu.o: In function `fix_to_virt':
/home/changbin/work/linux/./include/asm-generic/fixmap.h:31: undefined reference to `__compiletime_assert_31' Makefile:1051: recipe for target 'vmlinux' failed
make: *** [vmlinux] Error 1

Signed-off-by Du Changbin <[email protected]>

Acked-by Steven Rostedt (VMware) <[email protected]>

---
 arch/arm/mm/mmu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

arch/arm/mm/mmu.c +1 -1

index e46a6a446cdd..c08d74e76714 100644
@@ -1599,7 +1599,7 @@ static void __init early_fixmap_shutdown(void)
 		pte_t *pte;
 		struct map_desc map;
 
-		map.virtual = fix_to_virt(i);
+		map.virtual = __fix_to_virt(i);
 		pte = pte_offset_early_fixmap(pmd_off_k(map.virtual), map.virtual);
 
 		/* Only i/o device mappings are supported ATM */

Signature · 1 line

2.17.1

Du Changbin

author

2018-10-19 12:49 UTC #7 lore raw

PATCH v2 4/4 kernel hacking: new config CC_OPTIMIZE_FOR_DEBUGGING to apply GCC -Og optimization

In reply to Du Changbin

This will apply GCC '-Og' optimization level which is supported since GCC 4.8. This optimization level offers a reasonable level of optimization while maintaining fast compilation and a good debugging experience. It is similar to '-O1' while perferring to keep debug ability over runtime speed.

If enabling this option breaks your kernel, you should either disable this or find a fix (mostly in the arch code). Currently this option has only been tested on x86_64 and arm platform.

This option can satisfy people who was searching for a method to disable compiler optimizations so to achieve better kernel debugging experience with kgdb or qemu.

The main problem of '-Og' is we must not use __attribute__((error(msg))). The compiler will report error though the call to error function still can be optimize out. So we must fallback to array tricky.

Comparison of vmlinux size: a bit smaller.

w/o CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
$ size vmlinux
text data bss dec hex filename
22665554 9709674 2920908 35296136 21a9388 vmlinux

w/ CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
$ size vmlinux
text data bss dec hex filename
21499032 10102758 2920908 34522698 20ec64a vmlinux

Comparison of system performance: a bit drop (~6%).

    This benchmark of kernel compilation is suggested by Ingo Molnar.
    https://lkml.org/lkml/2018/5/2/74

    Preparation: Set cpufreq to 'performance'.
    for ((cpu=0; cpu<120; cpu++)); do
      G=/sys/devices/system/cpu/cpu$cpu/cpufreq/scaling_governor
      [ -f $G ] && echo performance > $G
    done

    w/o CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
    $ perf stat --repeat 5 --null --pre                 '\
        cp -a kernel ../kernel.copy.$(date +%s);         \
        rm -rf *;                                        \
        git checkout .;                                  \
        echo 1 > /proc/sys/vm/drop_caches;               \
        find ../kernel* -type f | xargs cat >/dev/null;  \
        make -j kernel >/dev/null;                       \
        make clean >/dev/null 2>&1;                      \
        sync                                            '\
                                                         \
        make -j8 >/dev/null

    Performance counter stats for 'make -j8' (5 runs):

        219.764246652 seconds time elapsed                   ( +-  0.78% )

    w/ CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
    $ perf stat --repeat 5 --null --pre                 '\
        cp -a kernel ../kernel.copy.$(date +%s);         \
        rm -rf *;                                        \
        git checkout .;                                  \
        echo 1 > /proc/sys/vm/drop_caches;               \
        find ../kernel* -type f | xargs cat >/dev/null;  \
        make -j kernel >/dev/null;                       \
        make clean >/dev/null 2>&1;                      \
        sync                                            '\
                                                         \
        make -j8 >/dev/null

    Performance counter stats for 'make -j8' (5 runs):

         233.574187771 seconds time elapsed                  ( +-  0.19% )

Signed-off-by Du Changbin <[email protected]>

Acked-by Steven Rostedt (VMware) <[email protected]>

---
 Makefile                     |  5 +++++
 include/linux/compiler-gcc.h |  2 +-
 include/linux/compiler.h     |  2 +-
 init/Kconfig                 | 19 +++++++++++++++++++
 4 files changed, 26 insertions(+), 2 deletions(-)

Makefile +5 -0

index 757d6507cb5c..ea908cfe8594 100644
@@ -657,6 +657,10 @@ KBUILD_CFLAGS	+= $(call cc-disable-warning, format-truncation)
 KBUILD_CFLAGS	+= $(call cc-disable-warning, format-overflow)
 KBUILD_CFLAGS	+= $(call cc-disable-warning, int-in-bool-context)
 
+ifdef CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
+KBUILD_CFLAGS	+= $(call cc-option, -Og)
+KBUILD_CFLAGS	+= $(call cc-disable-warning,maybe-uninitialized,)
+else
 ifdef CONFIG_CC_OPTIMIZE_FOR_SIZE
 KBUILD_CFLAGS	+= $(call cc-option,-Oz,-Os)
 KBUILD_CFLAGS	+= $(call cc-disable-warning,maybe-uninitialized,)

@@ -667,6 +671,7 @@ else
 KBUILD_CFLAGS   += -O2
 endif
 endif
+endif
 
 KBUILD_CFLAGS += $(call cc-ifversion, -lt, 0409, \
 			$(call cc-disable-warning,maybe-uninitialized,))

include/linux/compiler-gcc.h +1 -1

index 4d36b27214fd..2a76f7c64b54 100644
@@ -85,7 +85,7 @@
 
 #define __compiletime_object_size(obj) __builtin_object_size(obj, 0)
 
-#ifndef __CHECKER__
+#if !defined(__CHECKER__) && !defined(CONFIG_CC_OPTIMIZE_FOR_DEBUGGING)
 #define __compiletime_warning(message) __attribute__((warning(message)))
 #define __compiletime_error(message) __attribute__((error(message)))

include/linux/compiler.h +1 -1

index 681d866efb1e..9385c62e9f00 100644
@@ -320,7 +320,7 @@ static inline void *offset_to_ptr(const int *off)
  * sparse see a constant array size without breaking compiletime_assert on old
  * versions of GCC (e.g. 4.2.4), so hide the array from sparse altogether.
  */
-# ifndef __CHECKER__
+# if !defined(__CHECKER__) && !defined(CONFIG_CC_OPTIMIZE_FOR_DEBUGGING)
 #  define __compiletime_error_fallback(condition) \
 	do { ((void)sizeof(char[1 - 2 * condition])); } while (0)
 # endif

init/Kconfig +19 -0

index 1e234e2f1cba..b01c7c654570 100644
@@ -1081,6 +1081,25 @@ config CC_OPTIMIZE_FOR_SIZE
 
 	  If unsure, say N.
 
+config CC_OPTIMIZE_FOR_DEBUGGING
+	bool "Optimize for better debugging experience (-Og)"
+	select NO_AUTO_INLINE
+	help
+	  This will apply GCC '-Og' optimization level which is supported
+	  since GCC 4.8. This optimization level offers a reasonable level
+	  of optimization while maintaining fast compilation and a good
+	  debugging experience. It is similar to '-O1' while preferring to
+	  keep debug ability over runtime speed. The overall performance
+	  will drop a bit (~6%).
+
+	  Use only if you want to debug the kernel, especially if you want
+	  to have better kernel debugging experience with gdb facilities
+	  like kgdb or qemu. If enabling this option breaks your kernel,
+	  you should either disable this or find a fix (mostly in the arch
+	  code).
+
+	  If unsure, select N.
+
 endchoice
 
 config HAVE_LD_DEAD_CODE_DATA_ELIMINATION

Signature · 1 line

2.17.1

Masahiro Yamada

2018-10-28 02:28 UTC #8 lore raw

Re: [PATCH v2 4/4] kernel hacking: new config CC_OPTIMIZE_FOR_DEBUGGING to apply GCC -Og optimization

Quoted 89 lines · Du Changbin

This will apply GCC '-Og' optimization level which is supported since GCC 4.8. This optimization level offers a reasonable level of optimization while maintaining fast compilation and a good debugging experience. It is similar to '-O1' while perferring to keep debug ability over runtime speed.
If enabling this option breaks your kernel, you should either disable this or find a fix (mostly in the arch code). Currently this option has only been tested on x86_64 and arm platform.
This option can satisfy people who was searching for a method to disable compiler optimizations so to achieve better kernel debugging experience with kgdb or qemu.
The main problem of '-Og' is we must not use __attribute__((error(msg))). The compiler will report error though the call to error function still can be optimize out. So we must fallback to array tricky.
Comparison of vmlinux size: a bit smaller.
w/o CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
$ size vmlinux
text data bss dec hex filename
22665554 9709674 2920908 35296136 21a9388 vmlinux
w/ CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
$ size vmlinux
text data bss dec hex filename
21499032 10102758 2920908 34522698 20ec64a vmlinux
Comparison of system performance: a bit drop (~6%).
    This benchmark of kernel compilation is suggested by Ingo Molnar.
    https://lkml.org/lkml/2018/5/2/74

    Preparation: Set cpufreq to 'performance'.
    for ((cpu=0; cpu<120; cpu++)); do
      G=/sys/devices/system/cpu/cpu$cpu/cpufreq/scaling_governor
      [ -f $G ] && echo performance > $G
    done

    w/o CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
    $ perf stat --repeat 5 --null --pre                 '\
        cp -a kernel ../kernel.copy.$(date +%s);         \
        rm -rf *;                                        \
        git checkout .;                                  \
        echo 1 > /proc/sys/vm/drop_caches;               \
        find ../kernel* -type f | xargs cat >/dev/null;  \
        make -j kernel >/dev/null;                       \
        make clean >/dev/null 2>&1;                      \
        sync                                            '\
                                                         \
        make -j8 >/dev/null

    Performance counter stats for 'make -j8' (5 runs):

        219.764246652 seconds time elapsed                   ( +-  0.78% )

    w/ CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
    $ perf stat --repeat 5 --null --pre                 '\
        cp -a kernel ../kernel.copy.$(date +%s);         \
        rm -rf *;                                        \
        git checkout .;                                  \
        echo 1 > /proc/sys/vm/drop_caches;               \
        find ../kernel* -type f | xargs cat >/dev/null;  \
        make -j kernel >/dev/null;                       \
        make clean >/dev/null 2>&1;                      \
        sync                                            '\
                                                         \
        make -j8 >/dev/null

    Performance counter stats for 'make -j8' (5 runs):

         233.574187771 seconds time elapsed                  ( +-  0.19% )
Signed-off-by Du Changbin <[email protected]>
Acked-by Steven Rostedt (VMware) <[email protected]>
---
 Makefile                     |  5 +++++
 include/linux/compiler-gcc.h |  2 +-
 include/linux/compiler.h     |  2 +-
 init/Kconfig                 | 19 +++++++++++++++++++
 4 files changed, 26 insertions(+), 2 deletions(-)
Makefile +0 -0
index 757d6507cb5c..ea908cfe8594 100644

Makefile KBUILD_CFLAGS += $(call cc-disable-warning, format-truncation) · line 661

Quoted 5 lines

Makefile +2 -0

@@ -657,6 +657,10 @@ KBUILD_CFLAGS      += $(call cc-disable-warning, format-truncation)
 KBUILD_CFLAGS  += $(call cc-disable-warning, format-overflow)
 KBUILD_CFLAGS  += $(call cc-disable-warning, int-in-bool-context)
 
+ifdef CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
+KBUILD_CFLAGS  += $(call cc-option, -Og)

This line is a problem.

As you stated in your patch description, -Og is supported on GCC 4.8+ but the minimum GCC version supported for Linux kernel is GCC 4.6

If you use either GCC 4.6 or 4.7,
$(call cc-option, -Og) is evaluated to empty.

It means no optimization flag is given to the compiler, then the build fails.

I'd like to suggest to change as follows:

Makefile +0 -0

index 122ab74..9af3dc4 100644
@@ -663,7 +663,7 @@ KBUILD_CFLAGS       += $(call cc-disable-warning,

format-overflow)
KBUILD_CFLAGS += $(call cc-disable-warning, int-in-bool-context)

ifdef CONFIG_CC_OPTIMIZE_FOR_DEBUGGING

-KBUILD_CFLAGS  += $(call cc-option, -Og)
+KBUILD_CFLAGS  += -Og
 KBUILD_CFLAGS  += $(call cc-disable-warning,maybe-uninitialized,)

else
ifdef CONFIG_CC_OPTIMIZE_FOR_SIZE

init/Kconfig +1 -0

index b01c7c6..a0424dc 100644
@@ -1083,6 +1083,7 @@ config CC_OPTIMIZE_FOR_SIZE
 
 config CC_OPTIMIZE_FOR_DEBUGGING
        bool "Optimize for better debugging experience (-Og)"
+       depends on $(cc-option,-Og)
        select NO_AUTO_INLINE
        help
          This will apply GCC '-Og' optimization level which is supported

If -Og is unsupported by the compiler, this is hidden from the Kconfig menu.

Also, in order to fix a warning from 'make tiny.config' could you add the following change?

kernel/configs/tiny.config +1 -0

index 7fa0c4a..599ea86 100644
@@ -1,5 +1,6 @@
 # CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE is not set
 CONFIG_CC_OPTIMIZE_FOR_SIZE=y
+# CONFIG_CC_OPTIMIZE_FOR_DEBUGGING is not set
 # CONFIG_KERNEL_GZIP is not set
 # CONFIG_KERNEL_BZIP2 is not set
 # CONFIG_KERNEL_LZMA is not set

Quoted 72 lines

Makefile +3 -0

+KBUILD_CFLAGS  += $(call cc-disable-warning,maybe-uninitialized,)
+else
 ifdef CONFIG_CC_OPTIMIZE_FOR_SIZE
 KBUILD_CFLAGS  += $(call cc-option,-Oz,-Os)
 KBUILD_CFLAGS  += $(call cc-disable-warning,maybe-uninitialized,)

@@ -667,6 +671,7 @@ else
 KBUILD_CFLAGS   += -O2
 endif
 endif
+endif
 
 KBUILD_CFLAGS += $(call cc-ifversion, -lt, 0409, \
                        $(call cc-disable-warning,maybe-uninitialized,))

include/linux/compiler-gcc.h +1 -1

index 4d36b27214fd..2a76f7c64b54 100644
@@ -85,7 +85,7 @@
 
 #define __compiletime_object_size(obj) __builtin_object_size(obj, 0)
 
-#ifndef __CHECKER__
+#if !defined(__CHECKER__) && !defined(CONFIG_CC_OPTIMIZE_FOR_DEBUGGING)
 #define __compiletime_warning(message) __attribute__((warning(message)))
 #define __compiletime_error(message) __attribute__((error(message)))

include/linux/compiler.h +1 -1

index 681d866efb1e..9385c62e9f00 100644
@@ -320,7 +320,7 @@ static inline void *offset_to_ptr(const int *off)
  * sparse see a constant array size without breaking compiletime_assert on old
  * versions of GCC (e.g. 4.2.4), so hide the array from sparse altogether.
  */
-# ifndef __CHECKER__
+# if !defined(__CHECKER__) && !defined(CONFIG_CC_OPTIMIZE_FOR_DEBUGGING)
 #  define __compiletime_error_fallback(condition) \
        do { ((void)sizeof(char[1 - 2 * condition])); } while (0)
 # endif

init/Kconfig +19 -1

index 1e234e2f1cba..b01c7c654570 100644
@@ -1081,6 +1081,25 @@ config CC_OPTIMIZE_FOR_SIZE
 
          If unsure, say N.
 
+config CC_OPTIMIZE_FOR_DEBUGGING
+       bool "Optimize for better debugging experience (-Og)"
+       select NO_AUTO_INLINE
+       help
+         This will apply GCC '-Og' optimization level which is supported
+         since GCC 4.8. This optimization level offers a reasonable level
+         of optimization while maintaining fast compilation and a good
+         debugging experience. It is similar to '-O1' while preferring to
+         keep debug ability over runtime speed. The overall performance
+         will drop a bit (~6%).
+
+         Use only if you want to debug the kernel, especially if you want
+         to have better kernel debugging experience with gdb facilities
+         like kgdb or qemu. If enabling this option breaks your kernel,
+         you should either disable this or find a fix (mostly in the arch
+         code).
+
+         If unsure, select N.
+
 endchoice
 
 config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
--

2.17.1

Signature · 2 lines

Best Regards
Masahiro Yamada

Changbin Du

author

2018-10-28 12:51 UTC #9 lore raw

Re: [PATCH v2 4/4] kernel hacking: new config CC_OPTIMIZE_FOR_DEBUGGING to apply GCC -Og optimization

Quoted 103 lines · Masahiro Yamada

Quoted 89 lines · Du Changbin
This will apply GCC '-Og' optimization level which is supported since GCC 4.8. This optimization level offers a reasonable level of optimization while maintaining fast compilation and a good debugging experience. It is similar to '-O1' while perferring to keep debug ability over runtime speed.
If enabling this option breaks your kernel, you should either disable this or find a fix (mostly in the arch code). Currently this option has only been tested on x86_64 and arm platform.
This option can satisfy people who was searching for a method to disable compiler optimizations so to achieve better kernel debugging experience with kgdb or qemu.
The main problem of '-Og' is we must not use __attribute__((error(msg))). The compiler will report error though the call to error function still can be optimize out. So we must fallback to array tricky.
Comparison of vmlinux size: a bit smaller.
w/o CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
$ size vmlinux
text data bss dec hex filename
22665554 9709674 2920908 35296136 21a9388 vmlinux
w/ CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
$ size vmlinux
text data bss dec hex filename
21499032 10102758 2920908 34522698 20ec64a vmlinux
Comparison of system performance: a bit drop (~6%).
    This benchmark of kernel compilation is suggested by Ingo Molnar.
    https://lkml.org/lkml/2018/5/2/74

    Preparation: Set cpufreq to 'performance'.
    for ((cpu=0; cpu<120; cpu++)); do
      G=/sys/devices/system/cpu/cpu$cpu/cpufreq/scaling_governor
      [ -f $G ] && echo performance > $G
    done

    w/o CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
    $ perf stat --repeat 5 --null --pre                 '\
        cp -a kernel ../kernel.copy.$(date +%s);         \
        rm -rf *;                                        \
        git checkout .;                                  \
        echo 1 > /proc/sys/vm/drop_caches;               \
        find ../kernel* -type f | xargs cat >/dev/null;  \
        make -j kernel >/dev/null;                       \
        make clean >/dev/null 2>&1;                      \
        sync                                            '\
                                                         \
        make -j8 >/dev/null

    Performance counter stats for 'make -j8' (5 runs):

        219.764246652 seconds time elapsed                   ( +-  0.78% )

    w/ CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
    $ perf stat --repeat 5 --null --pre                 '\
        cp -a kernel ../kernel.copy.$(date +%s);         \
        rm -rf *;                                        \
        git checkout .;                                  \
        echo 1 > /proc/sys/vm/drop_caches;               \
        find ../kernel* -type f | xargs cat >/dev/null;  \
        make -j kernel >/dev/null;                       \
        make clean >/dev/null 2>&1;                      \
        sync                                            '\
                                                         \
        make -j8 >/dev/null

    Performance counter stats for 'make -j8' (5 runs):

         233.574187771 seconds time elapsed                  ( +-  0.19% )
Signed-off-by Du Changbin <[email protected]>
Acked-by Steven Rostedt (VMware) <[email protected]>
---
 Makefile                     |  5 +++++
 include/linux/compiler-gcc.h |  2 +-
 include/linux/compiler.h     |  2 +-
 init/Kconfig                 | 19 +++++++++++++++++++
 4 files changed, 26 insertions(+), 2 deletions(-)
Makefile +0 -0
index 757d6507cb5c..ea908cfe8594 100644
Makefile KBUILD_CFLAGS += $(call cc-disable-warning, format-truncation) · line 661
Quoted 5 lines
Makefile +2 -0
@@ -657,6 +657,10 @@ KBUILD_CFLAGS      += $(call cc-disable-warning, format-truncation)
 KBUILD_CFLAGS  += $(call cc-disable-warning, format-overflow)
 KBUILD_CFLAGS  += $(call cc-disable-warning, int-in-bool-context)
 
+ifdef CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
+KBUILD_CFLAGS  += $(call cc-option, -Og)
This line is a problem.
As you stated in your patch description, -Og is supported on GCC 4.8+ but the minimum GCC version supported for Linux kernel is GCC 4.6

Quoted 9 lines

If you use either GCC 4.6 or 4.7,
$(call cc-option, -Og) is evaluated to empty.
It means no optimization flag is given to the compiler, then the build fails.
I'd like to suggest to change as follows:

Thanks for pointing this out and kind suggestions. I have tested with your modification. Will update patch serias soon.

Quoted 137 lines

Makefile +1 -1

index 122ab74..9af3dc4 100644
@@ -663,7 +663,7 @@ KBUILD_CFLAGS       += $(call cc-disable-warning,
 format-overflow)
 KBUILD_CFLAGS  += $(call cc-disable-warning, int-in-bool-context)
 
 ifdef CONFIG_CC_OPTIMIZE_FOR_DEBUGGING
-KBUILD_CFLAGS  += $(call cc-option, -Og)
+KBUILD_CFLAGS  += -Og
 KBUILD_CFLAGS  += $(call cc-disable-warning,maybe-uninitialized,)
 else

ifdef CONFIG_CC_OPTIMIZE_FOR_SIZE

init/Kconfig +1 -0

index b01c7c6..a0424dc 100644
@@ -1083,6 +1083,7 @@ config CC_OPTIMIZE_FOR_SIZE
 
 config CC_OPTIMIZE_FOR_DEBUGGING
        bool "Optimize for better debugging experience (-Og)"
+       depends on $(cc-option,-Og)
        select NO_AUTO_INLINE
        help
          This will apply GCC '-Og' optimization level which is supported

If -Og is unsupported by the compiler, this is hidden from the Kconfig menu.

Also, in order to fix a warning from 'make tiny.config' could you add the following change?

kernel/configs/tiny.config +1 -0

index 7fa0c4a..599ea86 100644
@@ -1,5 +1,6 @@
 # CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE is not set
 CONFIG_CC_OPTIMIZE_FOR_SIZE=y
+# CONFIG_CC_OPTIMIZE_FOR_DEBUGGING is not set
 # CONFIG_KERNEL_GZIP is not set
 # CONFIG_KERNEL_BZIP2 is not set
 # CONFIG_KERNEL_LZMA is not set

Quoted 79 lines

Makefile +3 -0

+KBUILD_CFLAGS  += $(call cc-disable-warning,maybe-uninitialized,)
+else
 ifdef CONFIG_CC_OPTIMIZE_FOR_SIZE
 KBUILD_CFLAGS  += $(call cc-option,-Oz,-Os)
 KBUILD_CFLAGS  += $(call cc-disable-warning,maybe-uninitialized,)

@@ -667,6 +671,7 @@ else
 KBUILD_CFLAGS   += -O2
 endif
 endif
+endif
 
 KBUILD_CFLAGS += $(call cc-ifversion, -lt, 0409, \
                        $(call cc-disable-warning,maybe-uninitialized,))

include/linux/compiler-gcc.h +1 -1

index 4d36b27214fd..2a76f7c64b54 100644
@@ -85,7 +85,7 @@
 
 #define __compiletime_object_size(obj) __builtin_object_size(obj, 0)
 
-#ifndef __CHECKER__
+#if !defined(__CHECKER__) && !defined(CONFIG_CC_OPTIMIZE_FOR_DEBUGGING)
 #define __compiletime_warning(message) __attribute__((warning(message)))
 #define __compiletime_error(message) __attribute__((error(message)))

include/linux/compiler.h +1 -1

index 681d866efb1e..9385c62e9f00 100644
@@ -320,7 +320,7 @@ static inline void *offset_to_ptr(const int *off)
  * sparse see a constant array size without breaking compiletime_assert on old
  * versions of GCC (e.g. 4.2.4), so hide the array from sparse altogether.
  */
-# ifndef __CHECKER__
+# if !defined(__CHECKER__) && !defined(CONFIG_CC_OPTIMIZE_FOR_DEBUGGING)
 #  define __compiletime_error_fallback(condition) \
        do { ((void)sizeof(char[1 - 2 * condition])); } while (0)
 # endif

init/Kconfig +19 -1

index 1e234e2f1cba..b01c7c654570 100644
@@ -1081,6 +1081,25 @@ config CC_OPTIMIZE_FOR_SIZE
 
          If unsure, say N.
 
+config CC_OPTIMIZE_FOR_DEBUGGING
+       bool "Optimize for better debugging experience (-Og)"
+       select NO_AUTO_INLINE
+       help
+         This will apply GCC '-Og' optimization level which is supported
+         since GCC 4.8. This optimization level offers a reasonable level
+         of optimization while maintaining fast compilation and a good
+         debugging experience. It is similar to '-O1' while preferring to
+         keep debug ability over runtime speed. The overall performance
+         will drop a bit (~6%).
+
+         Use only if you want to debug the kernel, especially if you want
+         to have better kernel debugging experience with gdb facilities
+         like kgdb or qemu. If enabling this option breaks your kernel,
+         you should either disable this or find a fix (mostly in the arch
+         code).
+
+         If unsure, select N.
+
 endchoice
 
 config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
--

2.17.1

Signature · 2 lines

Best Regards
Masahiro Yamada

Signature · 2 lines

Thanks,
Changbin Du

Masahiro Yamada

2018-10-28 13:09 UTC #10 lore raw

Re: [PATCH v2 4/4] kernel hacking: new config CC_OPTIMIZE_FOR_DEBUGGING to apply GCC -Og optimization

Hi Changbin,

Quoted 3 lines · Changbin Du

Quoted 1 line
Thanks for pointing this out and kind suggestions. I have tested with your modification. Will update patch serias soon.

I have one more request.

The patch subject
"kernel hacking: new config CC_OPTIMIZE_FOR_DEBUGGING to apply GCC -Og optimization"
is too long (83 characters)

I'd like the subject to be within 70 columns or so.

Thanks.

Signature · 2 lines

Best Regards
Masahiro Yamada

Changbin Du

author

2018-10-28 13:14 UTC #11 lore raw

Re: [PATCH v2 4/4] kernel hacking: new config CC_OPTIMIZE_FOR_DEBUGGING to apply GCC -Og optimization

Quoted 11 lines · Masahiro Yamada

Hi Changbin,
Quoted 3 lines · Changbin Du
Quoted 1 line
Thanks for pointing this out and kind suggestions. I have tested with your modification. Will update patch serias soon.
I have one more request.

Quoted 8 lines

The patch subject
"kernel hacking: new config CC_OPTIMIZE_FOR_DEBUGGING to apply GCC -Og optimization"
is too long (83 characters)
I'd like the subject to be within 70 columns or so.

oh, I just sent new version. Could me update the title if there are more suggestions then?

Quoted 8 lines

Thanks.
Signature · 2 lines
Best Regards
Masahiro Yamada

Signature · 2 lines

Thanks,
Changbin Du

Masahiro Yamada

2018-10-29 13:01 UTC #12 lore raw

Re: [PATCH v2 4/4] kernel hacking: new config CC_OPTIMIZE_FOR_DEBUGGING to apply GCC -Og optimization

Hi.

Quoted 19 lines · Changbin Du

Quoted 17 lines · Masahiro Yamada
Hi Changbin,
Quoted 3 lines · Changbin Du
Quoted 1 line
Thanks for pointing this out and kind suggestions. I have tested with your modification. Will update patch serias soon.
I have one more request.
The patch subject
"kernel hacking: new config CC_OPTIMIZE_FOR_DEBUGGING to apply GCC -Og optimization"
is too long (83 characters)

Quoted 4 lines

Quoted 2 lines
I'd like the subject to be within 70 columns or so.
oh, I just sent new version. Could me update the title if there are more suggestions then?

How about this?

2/4: kernel hacking: add a config option to disable compiler auto-inlining 3/4: ARM: mm: fix build error in fix_to_virt with -Og optimization level 4/4: kernel hacking: support building kernel with -Og optimization level

(The CONFIG name is so long that the subject
would easily exceeds 80-columns with it.)

I applied this series to linux-kbuild,
and will send another pull request in this MW.

If you come up with better patch titles,
please let me know.
I will fix them up locally.

Signature · 2 lines

Best Regards
Masahiro Yamada

Changbin Du

author

2018-10-29 14:56 UTC #13 lore raw

Re: [PATCH v2 4/4] kernel hacking: new config CC_OPTIMIZE_FOR_DEBUGGING to apply GCC -Og optimization

Quoted 40 lines · Masahiro Yamada

Hi.
Quoted 19 lines · Changbin Du
Quoted 17 lines · Masahiro Yamada
Hi Changbin,
Quoted 3 lines · Changbin Du
Quoted 1 line
Thanks for pointing this out and kind suggestions. I have tested with your modification. Will update patch serias soon.
I have one more request.
The patch subject
"kernel hacking: new config CC_OPTIMIZE_FOR_DEBUGGING to apply GCC -Og optimization"
is too long (83 characters)
Quoted 4 lines
Quoted 2 lines
I'd like the subject to be within 70 columns or so.
oh, I just sent new version. Could me update the title if there are more suggestions then?
How about this?
2/4: kernel hacking: add a config option to disable compiler auto-inlining 3/4: ARM: mm: fix build error in fix_to_virt with -Og optimization level 4/4: kernel hacking: support building kernel with -Og optimization level
(The CONFIG name is so long that the subject
would easily exceeds 80-columns with it.)

Quoted 8 lines

I applied this series to linux-kbuild,
and will send another pull request in this MW.
If you come up with better patch titles,
please let me know.
I will fix them up locally.

I am just fine with these new titles. Thanks.

Quoted 5 lines

Signature · 3 lines
-- 
Best Regards
Masahiro Yamada

Signature · 2 lines

Thanks,
Changbin Du