perf fixes

Quoted 22 lines · Arnaldo Carvalho de Melo

Em Thu, Sep 12, 2013 at 11:10:37AM -0700, Linus Torvalds escreveu:
Quoted 12 lines · Linus Torvalds
Quoted 2 lines
When I compiled "perf" at the same time as doing a big kernel compile, the kernel compile failed
Oops. That may actually have been me being a bit *too* eager with a "make allmodconfig" build. I can't reproduce it, and I'm starting to suspect that I instead had two kernel compiles going, not one kernel compile and a tools/perf/ compile.
"Yo Dawg, I heard you like kernel compiles, so I put a kernel
compile in your kernel compile so that you can compile the kernel while you compile the kernel".
:-)
Quoted 1 line
But at least the "make install" problem is repeatable, though.
Well, I just tried it, and the only thing that gets rebuilt are the CHK environment tests that try to figure out what can be built into perf, i.e. perl, python, libaudit, etc.

Quoted 5 lines

Its something that annoys me as well, but not so much as to make me figure out how to make those be done only if some source file changed.
But then, if you remove, say, libelf from your system so that you get a perf tool that uses just /proc/kallsyms, it wouldn't detect it...

I don't think package removal is a particularly common usecase.

Quoted 2 lines

Perhaps in that case we should say: want a new build with a different environment? Do a 'make clean' first.

Exactly. The most common pattern is:

make

   # see warnings about missing dependencies
   install missing packages

make

   # no warnings, happy camper

Downgrades, package removals almost never happen in real life, let alone in typical build flows.

So in the simplest approximation, if we detected just the best-case: 'all libraries are present, we can do a full build' case and cached that fact across builds (and cleared the cached flag on 'make clean'), that would help speeding up the main usecase already.

But a cached flag per _successful_ config/feature-tests.mak testcase would work well too. I.e. only repeat checks that failed in the past. Once it succeeds there's no need to re-check.

Independent of all this is the the bug of repeat checks Linus noticed, that's indeed annoying and should be fixed separately of any feature test cache.

Thanks,

Ingo

David Ahern

2013-09-12 19:58 UTC #7 lore raw

Quoted 2 lines · Arnaldo Carvalho de Melo

Its something that annoys me as well, but not so much as to make me figure out how to make those be done only if some source file changed.

Jiri and I have both taken stabs at a config-based build rather than probing. Just need to finish it.

David

Arnaldo Carvalho de Melo

2013-09-12 20:02 UTC #8 lore raw

Em Thu, Sep 12, 2013 at 12:58:28PM -0700, David Ahern escreveu:

Quoted 6 lines · Arnaldo Carvalho de Melo

Quoted 2 lines
Its something that annoys me as well, but not so much as to make me figure out how to make those be done only if some source file changed.
Jiri and I have both taken stabs at a config-based build rather than probing. Just need to finish it.

But even then it would be nice to have a:

make autoconfig

That is what we have now, but only then we would try to do all that probing. Caching it somehow, as Ingo suggested, would get the best of both worlds.

- Arnaldo

Ingo Molnar

author

2013-09-12 20:31 UTC #9 lore raw

Quoted 10 lines · Arnaldo Carvalho de Melo

Em Thu, Sep 12, 2013 at 12:58:28PM -0700, David Ahern escreveu:
Quoted 8 lines · Arnaldo Carvalho de Melo
Quoted 4 lines
Its something that annoys me as well, but not so much as to make me figure out how to make those be done only if some source file changed.
Jiri and I have both taken stabs at a config-based build rather than probing. Just need to finish it.

Quoted 7 lines

But even then it would be nice to have a:
make autoconfig
That is what we have now, but only then we would try to do all that probing. Caching it somehow, as Ingo suggested, would get the best of both worlds.

Caching should be relatively straightforward: I'd suggest splitting up all the feature tests current present in tools/perf/config/feature-tests.mak into separate .c files which build into working binaries.

Instead of having 20 testcases crammed into a single Makefile:

comet:~/tip/tools/perf> git grep -w main config/feature-tests.mak config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(int argc, char *argv[]) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void)

Why not put those into 20x standalone .c files and try to build them as an explicit make command attempt from within the main Makefile? If the build fails then the feature flag is set to off, if it succeeds it's defined.

That makes feature tests generally easier to debug, easier to understand and generally easier to extend as well.

'Feature test binaries' that are already built will be 'cached' automatically, and the result of their build attempt will be a quick success.

This is relatively simple, well-known, powerful and I see no reason why it wouldn't work. No separate config language or special files are needed - just .c files, binaries and a Makefile.

'make clean' would get rid of the binaries.

[ It could even be made to work against package removal if we really
wanted to: if there's a common, known include file related to the feature (such as /usr/include/libaudit.h), then that include file could be added to the feature-test binary's list of dependencies. If the package is removed and the include file goes away, then the dependency breaks automatically and the binary is 'rebuilt', auto-detecting the failed dependency. ]

Thanks,

Ingo

Ingo Molnar

author

2013-09-12 20:43 UTC #10 lore raw

Quoted 29 lines · Ingo Molnar

Instead of having 20 testcases crammed into a single Makefile:
comet:~/tip/tools/perf> git grep -w main config/feature-tests.mak config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(int argc, char *argv[]) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void) config/feature-tests.mak:int main(void)
Why not put those into 20x standalone .c files and try to build them as an explicit make command attempt from within the main Makefile? If the build fails then the feature flag is set to off, if it succeeds it's defined.

Quoted 6 lines

That makes feature tests generally easier to debug, easier to understand and generally easier to extend as well.
'Feature test binaries' that are already built will be 'cached' automatically, and the result of their build attempt will be a quick success.

A successfull 'make' attempt of a target file within a separate, simple Makefile is very, very fast:

comet:~/tip/tools/power/x86/turbostat> perf stat --null --sync --repeat 10 make turbostat
make: `turbostat' is up to date.
make: `turbostat' is up to date.
make: `turbostat' is up to date.
make: `turbostat' is up to date.
make: `turbostat' is up to date.
make: `turbostat' is up to date.
make: `turbostat' is up to date.
make: `turbostat' is up to date.
make: `turbostat' is up to date.
make: `turbostat' is up to date.

Performance counter stats for 'make turbostat' (10 runs):

0.002146182 seconds time elapsed ( +- 0.35% )

That's 2 msecs only. 20 such tests will take less than 50 msecs in the 'cached' case.

Thanks,

Ingo

Ingo Molnar

author

2013-09-15 09:10 UTC #11 lore raw

PATCH 1/1 perf test-hack: Split out feature tests to cache them and to build them in parallel

So, the discussion died down a bit, so just to demonstrate how feature tests should be done IMO, here's a quick hack to perf that adds a new 'make feature-check' test-target and factors out 4 standalone feature tests:

 $ make feature-check
 ...
 Testing features:
 dwarf  support:  disabled
 ELF    support:  enabled
 glibc  support:  enabled
 Bionic support:  disabled

Repeat invocations only rebuild the testcases that failed. If no testcase fails then nothing is rebuilt and the features check runs very fast.

Even in the worst-case when most testcases are rebuilt there's a big improvement in runtime due to building the testcases in parallel.

NOTE: the new testcases are not fed back into perf's NO_DWARF/NO_LIBELF/etc. flag hierarchy yet, the patch only demonstrates that this method is a viable and fast method that solves most of our current problems in this area.

Other advantages over the current config/feature-tests.mak method:

- The individual testcases can be built/tested in the feature-checks/ directory as well, individually

- The testcases are simple, plain, standalone C files, unlike config/feature-tests.mak which suffers from escaping complications.

'make clean' in feature-checks/ gets rid of all testcase binaries.

Thanks,

Ingo

---------------------->

tools/perf/Makefile +14 -0

index 3a0ff7f..e6398e7 100644
@@ -816,3 +816,17 @@ clean: $(LIBTRACEEVENT)-clean $(LIBLK)-clean
 .PHONY: all install clean strip $(LIBTRACEEVENT) $(LIBLK)
 .PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell
 .PHONY: .FORCE-PERF-VERSION-FILE TAGS tags cscope .FORCE-PERF-CFLAGS
+
+#
+# Build the feature check binaries in parallel, ignore errors, ignore return value and suppress output:
+#
+feature-check-build:
+	@echo "Testing features:"
+	@-make -i -j -C feature-checks >/dev/null 2>&1
+
+feature-check: feature-check-build
+	@echo -n "dwarf  support:  "; [ -f feature-checks/test-dwarf  ] && echo enabled || echo disabled
+	@echo -n "ELF    support:  "; [ -f feature-checks/test-libelf ] && echo enabled || echo disabled
+	@echo -n "glibc  support:  "; [ -f feature-checks/test-glibc  ] && echo enabled || echo disabled
+	@echo -n "Bionic support:  "; [ -f feature-checks/test-bionic ] && echo enabled || echo disabled
+

tools/perf/feature-checks/Makefile new file +26 -0

new file mode 100644
index 0000000..dea10c8
@@ -0,0 +1,26 @@
+
+FILES=test-hello test-dwarf test-libelf test-glibc test-bionic
+
+all: $(FILES)
+
+###############################
+
+test-hello: test-hello.c
+	$(CC) -o $@ [email protected]
+
+test-dwarf: test-dwarf.c
+	$(CC) -o $@ [email protected] -ldw
+
+test-libelf: test-libelf.c
+	$(CC) -o $@ [email protected] -lelf
+
+test-glibc: test-glibc.c
+	$(CC) -o $@ [email protected]
+
+test-bionic: test-bionic.c
+	$(CC) -o $@ [email protected]
+
+###############################
+
+clean:
+	rm -f $(FILES)

tools/perf/feature-checks/test-bionic.c new file +6 -0

new file mode 100644
index 0000000..eac24e9
@@ -0,0 +1,6 @@
+#include <android/api-level.h>
+
+int main(void)
+{
+	return __ANDROID_API__;
+}

tools/perf/feature-checks/test-dwarf.c new file +9 -0

new file mode 100644
index 0000000..783dfcd
@@ -0,0 +1,9 @@
+#include <dwarf.h>
+#include <elfutils/libdw.h>
+#include <elfutils/version.h>
+
+int main(void)
+{
+	Dwarf *dbg = dwarf_begin(0, DWARF_C_READ);
+	return (long)dbg;
+}

tools/perf/feature-checks/test-glibc.c new file +8 -0

new file mode 100644
index 0000000..13c66a5
@@ -0,0 +1,8 @@
+#include <gnu/libc-version.h>
+
+int main(void)
+{
+	const char *version = gnu_get_libc_version();
+	return (long)version;
+}
+

tools/perf/feature-checks/test-hello new file binary +0 -0

new file mode 100755
index 0000000..6e9a668
Binary files /dev/null and b/tools/perf/feature-checks/test-hello differ

tools/perf/feature-checks/test-hello.c new file +6 -0

new file mode 100644
index 0000000..c9f398d
@@ -0,0 +1,6 @@
+#include <stdio.h>
+
+int main(void)
+{
+	return puts("hi");
+}

tools/perf/feature-checks/test-libelf new file binary +0 -0

new file mode 100755
index 0000000..bafde82
Binary files /dev/null and b/tools/perf/feature-checks/test-libelf differ

tools/perf/feature-checks/test-libelf.c new file +7 -0

new file mode 100644
index 0000000..1a08f97
@@ -0,0 +1,7 @@
+#include <libelf.h>
+
+int main(void)
+{
+	Elf *elf = elf_begin(0, ELF_C_READ, 0);
+	return (long)elf;
+}

Ingo Molnar

author

2013-09-30 16:42 UTC #12 lore raw

PATCH 1/1 perf auto-dep: Speed up feature tests by building them in parallel

This series (with combo patch attached) implements (much) faster perf-tools feature-auto-detection.

I used 3 tricks to implement feature auto-dependencies and to speed up feature detection:

- standalone Makefile in config/feature-checks/ built in parallel

- split-out standalone .c files in config/feature-checks/*.c

- used GCC's auto-dependency generation feature (-MD) to track the effects of system library addition/removal.

Before the changes a fully cached re-build of an already built tree took 2.6 second:

    $ perf stat --null --sync --repeat 3 -a make Makefile

2.669467209 seconds time elapsed ( +- 0.07% )

After the changes it takes only 0.6 seconds:

0.599161560 seconds time elapsed ( +- 0.25% )

So the empty build got 4.4x faster.

The 'cache cold' full rebuild got faster as well:

    $ perf stat --null --sync --pre 'make clean; make -C config/feature-checks/ clean' --repeat 3 -a make Makefile

   before:  2.671373213 seconds time elapsed                                          ( +-  0.12% )
    after:  0.959772417 seconds time elapsed                                          ( +-  0.22% )

That's 2.7x faster.

The series also implements new feature detection output, which looks like this:

Auto-detecting system features:

 ...            stackprotector-all: [ on  ]
 ...         volatile-register-var: [ on  ]
 ...                fortify-source: [ on  ]
 ...                        libelf: [ on  ]
 ...                   libelf-mmap: [ on  ]
 ...                         glibc: [ on  ]
 ...                         dwarf: [ on  ]
 ...             libelf-getphdrnum: [ on  ]
 ...                     libunwind: [ on  ]
 ...                      libaudit: [ on  ]
 ...                      libslang: [ on  ]
 ...                          gtk2: [ on  ]
 ...                  gtk2-infobar: [ on  ]
 ...                       libperl: [ OFF ]
 ...                     libpython: [ on  ]
 ...             libpython-version: [ on  ]
 ...                        libbfd: [ on  ]
 ...                       strlcpy: [ OFF ]
 ...                       on-exit: [ on  ]
 ...                     backtrace: [ on  ]
 ...                       libnuma: [ on  ]

This can be used to get an overview about feature detection status. Users will want to maximize the number of 'on' entries.

An 'on' feature is 'cached' in that its feature-check binary is present during future builds:

 $ ls -l config/feature-checks/test-libaudit
 -rwxrwxr-x 1 mingo mingo 7264 Sep 30 18:31 config/feature-checks/test-libaudit

and it's only remade if its underlying system libraries are changed:
removed or updated. Note that unlike the hack-version I sent before, cached features immediately turn into 'OFF' state once a dependent package is removed.

'OFF' features failed to pass due to missing dependencies. Those are re-made on every build. The fewer of them, the faster the build gets.

The series also streamlines feature names as exposed to the rest of perf:
it standardizes on the HAVE_{FEATURE}_SUPPORT pattern. We used to have a mixture of HAVE_{FEATURE} and {FEATURE}_SUPPORT constants.

There's still the annoyance left that Linus reported originally: that the config pass is repeated twice on 'make install' - this results in the feature matrix to be output twice.

Comments, suggestions, bugreports are welcome.

Thanks,

Ingo

----------------------->

Subject: perf autodep: Implement faster auto-detection of features
From: Ingo Molnar <[email protected]>
Date: Mon Sep 30 18:14:50 CEST 2013

---
 tools/perf/Makefile                                           |    1 
 tools/perf/arch/x86/include/perf_regs.h                       |    6 
 tools/perf/arch/x86/util/unwind.c                             |    4 
 tools/perf/bench/mem-memcpy-arch.h                            |    2 
 tools/perf/bench/mem-memcpy.c                                 |    2 
 tools/perf/bench/mem-memset-arch.h                            |    2 
 tools/perf/bench/mem-memset.c                                 |    2 
 tools/perf/builtin-bench.c                                    |    4 
 tools/perf/builtin-inject.c                                   |    2 
 tools/perf/builtin-probe.c                                    |   14 
 tools/perf/builtin-record.c                                   |   12 
 tools/perf/config/Makefile                                    |  237 +++++-----
 tools/perf/config/feature-checks/Makefile                     |  110 ++++
 tools/perf/config/feature-checks/test-backtrace.c             |   10 
 tools/perf/config/feature-checks/test-bionic.c                |    6 
 tools/perf/config/feature-checks/test-dwarf.c                 |    9 
 tools/perf/config/feature-checks/test-fortify-source.c        |    6 
 tools/perf/config/feature-checks/test-glibc.c                 |    8 
 tools/perf/config/feature-checks/test-gtk2-infobar.c          |   10 
 tools/perf/config/feature-checks/test-gtk2.c                  |   10 
 tools/perf/config/feature-checks/test-hello.c                 |    6 
 tools/perf/config/feature-checks/test-libaudit.c              |    7 
 tools/perf/config/feature-checks/test-libbfd.c                |    7 
 tools/perf/config/feature-checks/test-libelf-getphdrnum.c     |    7 
 tools/perf/config/feature-checks/test-libelf-mmap.c           |    7 
 tools/perf/config/feature-checks/test-libelf.c                |    7 
 tools/perf/config/feature-checks/test-libnuma.c               |    8 
 tools/perf/config/feature-checks/test-libperl.c               |    9 
 tools/perf/config/feature-checks/test-libpython-version.c     |   10 
 tools/perf/config/feature-checks/test-libpython.c             |    7 
 tools/perf/config/feature-checks/test-libslang.c              |    6 
 tools/perf/config/feature-checks/test-libunwind.c             |   20 
 tools/perf/config/feature-checks/test-on-exit.c               |    6 
 tools/perf/config/feature-checks/test-stackprotector-all.c    |    6 
 tools/perf/config/feature-checks/test-stackprotector.c        |    6 
 tools/perf/config/feature-checks/test-strlcpy.c               |    8 
 tools/perf/config/feature-checks/test-volatile-register-var.c |    6 
 tools/perf/perf.c                                             |    4 
 tools/perf/ui/gtk/browser.c                                   |    2 
 tools/perf/ui/gtk/gtk.h                                       |    4 
 tools/perf/ui/gtk/util.c                                      |    4 
 tools/perf/ui/ui.h                                            |    4 
 tools/perf/util/annotate.h                                    |    4 
 tools/perf/util/cache.h                                       |    2 
 tools/perf/util/generate-cmdlist.sh                           |    4 
 tools/perf/util/hist.h                                        |    4 
 tools/perf/util/include/dwarf-regs.h                          |    2 
 tools/perf/util/map.c                                         |    2 
 tools/perf/util/path.c                                        |    2 
 tools/perf/util/perf_regs.h                                   |    4 
 tools/perf/util/probe-event.c                                 |    4 
 tools/perf/util/probe-finder.h                                |    4 
 tools/perf/util/symbol-elf.c                                  |    2 
 tools/perf/util/symbol.h                                      |    8 
 tools/perf/util/unwind.h                                      |    4 
 tools/perf/util/util.c                                        |    4 
 56 files changed, 491 insertions(+), 167 deletions(-)

tools/perf/Makefile +1 -0

Index: tip/tools/perf/Makefile
===================================================================
@@ -816,3 +816,4 @@ clean: $(LIBTRACEEVENT)-clean $(LIBLK)-c
 .PHONY: all install clean strip $(LIBTRACEEVENT) $(LIBLK)
 .PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell
 .PHONY: .FORCE-PERF-VERSION-FILE TAGS tags cscope .FORCE-PERF-CFLAGS
+

tools/perf/arch/x86/include/perf_regs.h +3 -3

Index: tip/tools/perf/arch/x86/include/perf_regs.h
===================================================================
@@ -5,7 +5,7 @@
 #include "../../util/types.h"
 #include <asm/perf_regs.h>
 
-#ifndef ARCH_X86_64
+#ifndef HAVE_ARCH_X86_64_SUPPORT
 #define PERF_REGS_MASK ((1ULL << PERF_REG_X86_32_MAX) - 1)
 #else
 #define REG_NOSUPPORT ((1ULL << PERF_REG_X86_DS) | \

@@ -52,7 +52,7 @@ static inline const char *perf_reg_name(
 		return "FS";
 	case PERF_REG_X86_GS:
 		return "GS";
-#ifdef ARCH_X86_64
+#ifdef HAVE_ARCH_X86_64_SUPPORT
 	case PERF_REG_X86_R8:
 		return "R8";
 	case PERF_REG_X86_R9:

@@ -69,7 +69,7 @@ static inline const char *perf_reg_name(
 		return "R14";
 	case PERF_REG_X86_R15:
 		return "R15";
-#endif /* ARCH_X86_64 */
+#endif /* HAVE_ARCH_X86_64_SUPPORT */
 	default:
 		return NULL;
 	}

tools/perf/arch/x86/util/unwind.c +2 -2

Index: tip/tools/perf/arch/x86/util/unwind.c
===================================================================
@@ -4,7 +4,7 @@
 #include "perf_regs.h"
 #include "../../util/unwind.h"
 
-#ifdef ARCH_X86_64
+#ifdef HAVE_ARCH_X86_64_SUPPORT
 int unwind__arch_reg_id(int regnum)
 {
 	int id;

@@ -108,4 +108,4 @@ int unwind__arch_reg_id(int regnum)
 
 	return id;
 }
-#endif /* ARCH_X86_64 */
+#endif /* HAVE_ARCH_X86_64_SUPPORT */

tools/perf/bench/mem-memcpy-arch.h +1 -1

Index: tip/tools/perf/bench/mem-memcpy-arch.h
===================================================================
@@ -1,5 +1,5 @@
 
-#ifdef ARCH_X86_64
+#ifdef HAVE_ARCH_X86_64_SUPPORT
 
 #define MEMCPY_FN(fn, name, desc)		\
 	extern void *fn(void *, const void *, size_t);

tools/perf/bench/mem-memcpy.c +1 -1

Index: tip/tools/perf/bench/mem-memcpy.c
===================================================================
@@ -58,7 +58,7 @@ struct routine routines[] = {
 	{ "default",
 	  "Default memcpy() provided by glibc",
 	  memcpy },
-#ifdef ARCH_X86_64
+#ifdef HAVE_ARCH_X86_64_SUPPORT
 
 #define MEMCPY_FN(fn, name, desc) { name, desc, fn },
 #include "mem-memcpy-x86-64-asm-def.h"

tools/perf/bench/mem-memset-arch.h +1 -1

Index: tip/tools/perf/bench/mem-memset-arch.h
===================================================================
@@ -1,5 +1,5 @@
 
-#ifdef ARCH_X86_64
+#ifdef HAVE_ARCH_X86_64_SUPPORT
 
 #define MEMSET_FN(fn, name, desc)		\
 	extern void *fn(void *, int, size_t);

tools/perf/bench/mem-memset.c +1 -1

Index: tip/tools/perf/bench/mem-memset.c
===================================================================
@@ -58,7 +58,7 @@ static const struct routine routines[] =
 	{ "default",
 	  "Default memset() provided by glibc",
 	  memset },
-#ifdef ARCH_X86_64
+#ifdef HAVE_ARCH_X86_64_SUPPORT
 
 #define MEMSET_FN(fn, name, desc) { name, desc, fn },
 #include "mem-memset-x86-64-asm-def.h"

tools/perf/builtin-bench.c +2 -2

Index: tip/tools/perf/builtin-bench.c
===================================================================
@@ -35,7 +35,7 @@ struct bench_suite {
 /* sentinel: easy for help */
 #define suite_all { "all", "Test all benchmark suites", NULL }
 
-#ifdef LIBNUMA_SUPPORT
+#ifdef HAVE_LIBNUMA_SUPPORT
 static struct bench_suite numa_suites[] = {
 	{ "mem",
 	  "Benchmark for NUMA workloads",

@@ -80,7 +80,7 @@ struct bench_subsys {
 };
 
 static struct bench_subsys subsystems[] = {
-#ifdef LIBNUMA_SUPPORT
+#ifdef HAVE_LIBNUMA_SUPPORT
 	{ "numa",
 	  "NUMA scheduling and MM behavior",
 	  numa_suites },

tools/perf/builtin-inject.c +1 -1

Index: tip/tools/perf/builtin-inject.c
===================================================================
@@ -231,7 +231,7 @@ static int perf_event__inject_buildid(st
 				 * account this as unresolved.
 				 */
 			} else {
-#ifdef LIBELF_SUPPORT
+#ifdef HAVE_LIBELF_SUPPORT
 				pr_warning("no symbols found in %s, maybe "
 					   "install a debug package?\n",
 					   al.map->dso->long_name);

tools/perf/builtin-probe.c +7 -7

Index: tip/tools/perf/builtin-probe.c
===================================================================
@@ -173,7 +173,7 @@ static int opt_set_target(const struct o
 	if  (str && !params.target) {
 		if (!strcmp(opt->long_name, "exec"))
 			params.uprobes = true;
-#ifdef DWARF_SUPPORT
+#ifdef HAVE_DWARF_SUPPORT
 		else if (!strcmp(opt->long_name, "module"))
 			params.uprobes = false;
 #endif

@@ -187,7 +187,7 @@ static int opt_set_target(const struct o
 	return ret;
 }
 
-#ifdef DWARF_SUPPORT
+#ifdef HAVE_DWARF_SUPPORT
 static int opt_show_lines(const struct option *opt __maybe_unused,
 			  const char *str, int unset __maybe_unused)
 {

@@ -257,7 +257,7 @@ int cmd_probe(int argc, const char **arg
 		"perf probe [<options>] --add 'PROBEDEF' [--add 'PROBEDEF' ...]",
 		"perf probe [<options>] --del '[GROUP:]EVENT' ...",
 		"perf probe --list",
-#ifdef DWARF_SUPPORT
+#ifdef HAVE_DWARF_SUPPORT
 		"perf probe [<options>] --line 'LINEDESC'",
 		"perf probe [<options>] --vars 'PROBEPOINT'",
 #endif

@@ -271,7 +271,7 @@ int cmd_probe(int argc, const char **arg
 	OPT_CALLBACK('d', "del", NULL, "[GROUP:]EVENT", "delete a probe event.",
 		opt_del_probe_event),
 	OPT_CALLBACK('a', "add", NULL,
-#ifdef DWARF_SUPPORT
+#ifdef HAVE_DWARF_SUPPORT
 		"[EVENT=]FUNC[@SRC][+OFF|%return|:RL|;PT]|SRC:AL|SRC;PT"
 		" [[NAME=]ARG ...]",
 #else

@@ -283,7 +283,7 @@ int cmd_probe(int argc, const char **arg
 		"\t\tFUNC:\tFunction name\n"
 		"\t\tOFF:\tOffset from function entry (in byte)\n"
 		"\t\t%return:\tPut the probe at function return\n"
-#ifdef DWARF_SUPPORT
+#ifdef HAVE_DWARF_SUPPORT
 		"\t\tSRC:\tSource code path\n"
 		"\t\tRL:\tRelative line number from function entry.\n"
 		"\t\tAL:\tAbsolute line number in file.\n"

@@ -296,7 +296,7 @@ int cmd_probe(int argc, const char **arg
 		opt_add_probe_event),
 	OPT_BOOLEAN('f', "force", &params.force_add, "forcibly add events"
 		    " with existing name"),
-#ifdef DWARF_SUPPORT
+#ifdef HAVE_DWARF_SUPPORT
 	OPT_CALLBACK('L', "line", NULL,
 		     "FUNC[:RLN[+NUM|-RLN2]]|SRC:ALN[+NUM|-ALN2]",
 		     "Show source code lines.", opt_show_lines),

@@ -408,7 +408,7 @@ int cmd_probe(int argc, const char **arg
 		return ret;
 	}
 
-#ifdef DWARF_SUPPORT
+#ifdef HAVE_DWARF_SUPPORT
 	if (params.show_lines && !params.uprobes) {
 		if (params.mod_events) {
 			pr_err("  Error: Don't use --line with"

tools/perf/builtin-record.c +6 -6

Index: tip/tools/perf/builtin-record.c
===================================================================
@@ -29,7 +29,7 @@
 #include <sched.h>
 #include <sys/mman.h>
 
-#ifndef HAVE_ON_EXIT
+#ifndef HAVE_ON_EXIT_SUPPORT
 #ifndef ATEXIT_MAX
 #define ATEXIT_MAX 32
 #endif

@@ -684,7 +684,7 @@ error:
 	return ret;
 }
 
-#ifdef LIBUNWIND_SUPPORT
+#ifdef HAVE_LIBUNWIND_SUPPORT
 static int get_stack_size(char *str, unsigned long *_size)
 {
 	char *endptr;

@@ -710,7 +710,7 @@ static int get_stack_size(char *str, uns
 	       max_size, str);
 	return -1;
 }
-#endif /* LIBUNWIND_SUPPORT */
+#endif /* HAVE_LIBUNWIND_SUPPORT */
 
 int record_parse_callchain_opt(const struct option *opt,
 			       const char *arg, int unset)

@@ -748,7 +748,7 @@ int record_parse_callchain_opt(const str
 				       "needed for -g fp\n");
 			break;
 
-#ifdef LIBUNWIND_SUPPORT
+#ifdef HAVE_LIBUNWIND_SUPPORT
 		/* Dwarf style */
 		} else if (!strncmp(name, "dwarf", sizeof("dwarf"))) {
 			const unsigned long default_stack_dump_size = 8192;

@@ -768,7 +768,7 @@ int record_parse_callchain_opt(const str
 			if (!ret)
 				pr_debug("callchain: stack dump size %d\n",
 					 opts->stack_dump_size);
-#endif /* LIBUNWIND_SUPPORT */
+#endif /* HAVE_LIBUNWIND_SUPPORT */
 		} else {
 			pr_err("callchain: Unknown -g option "
 			       "value: %s\n", arg);

@@ -815,7 +815,7 @@ static struct perf_record record = {
 
 #define CALLCHAIN_HELP "do call-graph (stack chain/backtrace) recording: "
 
-#ifdef LIBUNWIND_SUPPORT
+#ifdef HAVE_LIBUNWIND_SUPPORT
 const char record_callchain_help[] = CALLCHAIN_HELP "[fp] dwarf";
 #else
 const char record_callchain_help[] = CALLCHAIN_HELP "[fp]";

tools/perf/config/Makefile +129 ~~-108~~

377 lines not on this page · raw message

tools/perf/config/feature-checks/Makefile new file +110 -0

110 lines not on this page · raw message

tools/perf/config/feature-checks/test-backtrace.c new file +10 -0

Index: tip/tools/perf/config/feature-checks/test-backtrace.c
===================================================================
@@ -0,0 +1,10 @@
+#include <execinfo.h>
+#include <stdio.h>
+
+int main(void)
+{
+	backtrace(NULL, 0);
+	backtrace_symbols(NULL, 0);
+
+	return 0;
+}

tools/perf/config/feature-checks/test-bionic.c new file +6 -0

Index: tip/tools/perf/config/feature-checks/test-bionic.c
===================================================================
@@ -0,0 +1,6 @@
+#include <android/api-level.h>
+
+int main(void)
+{
+	return __ANDROID_API__;
+}

tools/perf/config/feature-checks/test-dwarf.c new file +9 -0

Index: tip/tools/perf/config/feature-checks/test-dwarf.c
===================================================================
@@ -0,0 +1,9 @@
+#include <dwarf.h>
+#include <elfutils/libdw.h>
+#include <elfutils/version.h>
+
+int main(void)
+{
+	Dwarf *dbg = dwarf_begin(0, DWARF_C_READ);
+	return (long)dbg;
+}

tools/perf/config/feature-checks/test-fortify-source.c new file +6 -0

Index: tip/tools/perf/config/feature-checks/test-fortify-source.c
===================================================================
@@ -0,0 +1,6 @@
+#include <stdio.h>
+
+int main(void)
+{
+	return puts("hi");
+}

tools/perf/config/feature-checks/test-glibc.c new file +8 -0

Index: tip/tools/perf/config/feature-checks/test-glibc.c
===================================================================
@@ -0,0 +1,8 @@
+#include <gnu/libc-version.h>
+
+int main(void)
+{
+	const char *version = gnu_get_libc_version();
+	return (long)version;
+}
+

tools/perf/config/feature-checks/test-gtk2-infobar.c new file +10 -0

Index: tip/tools/perf/config/feature-checks/test-gtk2-infobar.c
===================================================================
@@ -0,0 +1,10 @@
+#pragma GCC diagnostic ignored "-Wstrict-prototypes"
+#include <gtk/gtk.h>
+#pragma GCC diagnostic error "-Wstrict-prototypes"
+
+int main(void)
+{
+	gtk_info_bar_new();
+
+	return 0;
+}

tools/perf/config/feature-checks/test-gtk2.c new file +10 -0

Index: tip/tools/perf/config/feature-checks/test-gtk2.c
===================================================================
@@ -0,0 +1,10 @@
+#pragma GCC diagnostic ignored "-Wstrict-prototypes"
+#include <gtk/gtk.h>
+#pragma GCC diagnostic error "-Wstrict-prototypes"
+
+int main(int argc, char *argv[])
+{
+        gtk_init(&argc, &argv);
+
+        return 0;
+}

tools/perf/config/feature-checks/test-hello.c new file +6 -0

Index: tip/tools/perf/config/feature-checks/test-hello.c
===================================================================
@@ -0,0 +1,6 @@
+#include <stdio.h>
+
+int main(void)
+{
+	return puts("hi");
+}

tools/perf/config/feature-checks/test-libaudit.c new file +7 -0

Index: tip/tools/perf/config/feature-checks/test-libaudit.c
===================================================================
@@ -0,0 +1,7 @@
+#include <libaudit.h>
+
+int main(void)
+{
+	printf("error message: %s\n", audit_errno_to_name(0));
+	return audit_open();
+}

tools/perf/config/feature-checks/test-libbfd.c new file +7 -0

Index: tip/tools/perf/config/feature-checks/test-libbfd.c
===================================================================
@@ -0,0 +1,7 @@
+#include <bfd.h>
+
+int main(void)
+{
+	bfd_demangle(0, 0, 0);
+	return 0;
+}

tools/perf/config/feature-checks/test-libelf-getphdrnum.c new file +7 -0

Index: tip/tools/perf/config/feature-checks/test-libelf-getphdrnum.c
===================================================================
@@ -0,0 +1,7 @@
+#include <libelf.h>
+#
+int main(void)
+{
+	size_t dst;
+	return elf_getphdrnum(0, &dst);
+}

tools/perf/config/feature-checks/test-libelf-mmap.c new file +7 -0

Index: tip/tools/perf/config/feature-checks/test-libelf-mmap.c
===================================================================
@@ -0,0 +1,7 @@
+#include <libelf.h>
+#
+int main(void)
+{
+	Elf *elf = elf_begin(0, ELF_C_READ_MMAP, 0);
+	return (long)elf;
+}

tools/perf/config/feature-checks/test-libelf.c new file +7 -0

Index: tip/tools/perf/config/feature-checks/test-libelf.c
===================================================================
@@ -0,0 +1,7 @@
+#include <libelf.h>
+
+int main(void)
+{
+	Elf *elf = elf_begin(0, ELF_C_READ, 0);
+	return (long)elf;
+}

tools/perf/config/feature-checks/test-libnuma.c new file +8 -0

Index: tip/tools/perf/config/feature-checks/test-libnuma.c
===================================================================
@@ -0,0 +1,8 @@
+#include <numa.h>
+#include <numaif.h>
+
+int main(void)
+{
+	numa_available();
+	return 0;
+}

tools/perf/config/feature-checks/test-libperl.c new file +9 -0

Index: tip/tools/perf/config/feature-checks/test-libperl.c
===================================================================
@@ -0,0 +1,9 @@
+#include <EXTERN.h>
+#include <perl.h>
+
+int main(void)
+{
+	perl_alloc();
+
+	return 0;
+}

tools/perf/config/feature-checks/test-libpython-version.c new file +10 -0

Index: tip/tools/perf/config/feature-checks/test-libpython-version.c
===================================================================
@@ -0,0 +1,10 @@
+#include <Python.h>
+
+#if PY_VERSION_HEX >= 0x03000000
+	#error
+#endif
+
+int main(void)
+{
+	return 0;
+}

tools/perf/config/feature-checks/test-libpython.c new file +7 -0

Index: tip/tools/perf/config/feature-checks/test-libpython.c
===================================================================
@@ -0,0 +1,7 @@
+#include <Python.h>
+#
+int main(void)
+{
+	Py_Initialize();
+	return 0;
+}

tools/perf/config/feature-checks/test-libslang.c new file +6 -0

Index: tip/tools/perf/config/feature-checks/test-libslang.c
===================================================================
@@ -0,0 +1,6 @@
+#include <slang.h>
+
+int main(void)
+{
+	return SLsmg_init_smg();
+}

tools/perf/config/feature-checks/test-libunwind.c new file +20 -0

Index: tip/tools/perf/config/feature-checks/test-libunwind.c
===================================================================
@@ -0,0 +1,20 @@
+#include <libunwind.h>
+#include <stdlib.h>
+
+extern int UNW_OBJ(dwarf_search_unwind_table) (unw_addr_space_t as,
+                                      unw_word_t ip,
+                                      unw_dyn_info_t *di,
+                                      unw_proc_info_t *pi,
+                                      int need_unwind_info, void *arg);
+
+
+#define dwarf_search_unwind_table UNW_OBJ(dwarf_search_unwind_table)
+
+int main(void)
+{
+	unw_addr_space_t addr_space;
+	addr_space = unw_create_addr_space(NULL, 0);
+	unw_init_remote(NULL, addr_space, NULL);
+	dwarf_search_unwind_table(addr_space, 0, NULL, NULL, 0, NULL);
+	return 0;
+}

tools/perf/config/feature-checks/test-on-exit.c new file +6 -0

Index: tip/tools/perf/config/feature-checks/test-on-exit.c
===================================================================
@@ -0,0 +1,6 @@
+#include <stdio.h>
+
+int main(void)
+{
+	return on_exit(NULL, NULL);
+}

tools/perf/config/feature-checks/test-stackprotector-all.c new file +6 -0

Index: tip/tools/perf/config/feature-checks/test-stackprotector-all.c
===================================================================
@@ -0,0 +1,6 @@
+#include <stdio.h>
+
+int main(void)
+{
+	return puts("hi");
+}

tools/perf/config/feature-checks/test-stackprotector.c new file +6 -0

Index: tip/tools/perf/config/feature-checks/test-stackprotector.c
===================================================================
@@ -0,0 +1,6 @@
+#include <stdio.h>
+
+int main(void)
+{
+	return puts("hi");
+}

tools/perf/config/feature-checks/test-strlcpy.c new file +8 -0

Index: tip/tools/perf/config/feature-checks/test-strlcpy.c
===================================================================
@@ -0,0 +1,8 @@
+#include <stdlib.h>
+extern size_t strlcpy(char *dest, const char *src, size_t size);
+
+int main(void)
+{
+	strlcpy(NULL, NULL, 0);
+	return 0;
+}

tools/perf/config/feature-checks/test-volatile-register-var.c new file +6 -0

Index: tip/tools/perf/config/feature-checks/test-volatile-register-var.c
===================================================================
@@ -0,0 +1,6 @@
+#include <stdio.h>
+
+int main(void)
+{
+	return puts("hi");
+}

tools/perf/perf.c +2 -2

Index: tip/tools/perf/perf.c
===================================================================
@@ -49,14 +49,14 @@ static struct cmd_struct commands[] = {
 	{ "version",	cmd_version,	0 },
 	{ "script",	cmd_script,	0 },
 	{ "sched",	cmd_sched,	0 },
-#ifdef LIBELF_SUPPORT
+#ifdef HAVE_LIBELF_SUPPORT
 	{ "probe",	cmd_probe,	0 },
 #endif
 	{ "kmem",	cmd_kmem,	0 },
 	{ "lock",	cmd_lock,	0 },
 	{ "kvm",	cmd_kvm,	0 },
 	{ "test",	cmd_test,	0 },
-#ifdef LIBAUDIT_SUPPORT
+#ifdef HAVE_LIBAUDIT_SUPPORT
 	{ "trace",	cmd_trace,	0 },
 #endif
 	{ "inject",	cmd_inject,	0 },

tools/perf/ui/gtk/browser.c +1 -1

Index: tip/tools/perf/ui/gtk/browser.c
===================================================================
@@ -43,7 +43,7 @@ const char *perf_gtk__get_percent_color(
 	return NULL;
 }
 
-#ifdef HAVE_GTK_INFO_BAR
+#ifdef HAVE_GTK_INFO_BAR_SUPPORT
 GtkWidget *perf_gtk__setup_info_bar(void)
 {
 	GtkWidget *info_bar;

tools/perf/ui/gtk/gtk.h +2 -2

Index: tip/tools/perf/ui/gtk/gtk.h
===================================================================
@@ -12,7 +12,7 @@ struct perf_gtk_context {
 	GtkWidget *main_window;
 	GtkWidget *notebook;
 
-#ifdef HAVE_GTK_INFO_BAR
+#ifdef HAVE_GTK_INFO_BAR_SUPPORT
 	GtkWidget *info_bar;
 	GtkWidget *message_label;
 #endif

@@ -39,7 +39,7 @@ void perf_gtk__resize_window(GtkWidget *
 const char *perf_gtk__get_percent_color(double percent);
 GtkWidget *perf_gtk__setup_statusbar(void);
 
-#ifdef HAVE_GTK_INFO_BAR
+#ifdef HAVE_GTK_INFO_BAR_SUPPORT
 GtkWidget *perf_gtk__setup_info_bar(void);
 #else
 static inline GtkWidget *perf_gtk__setup_info_bar(void)

tools/perf/ui/gtk/util.c +2 -2

Index: tip/tools/perf/ui/gtk/util.c
===================================================================
@@ -53,7 +53,7 @@ static int perf_gtk__error(const char *f
 	return 0;
 }
 
-#ifdef HAVE_GTK_INFO_BAR
+#ifdef HAVE_GTK_INFO_BAR_SUPPORT
 static int perf_gtk__warning_info_bar(const char *format, va_list args)
 {
 	char *msg;

@@ -105,7 +105,7 @@ static int perf_gtk__warning_statusbar(c
 
 struct perf_error_ops perf_gtk_eops = {
 	.error		= perf_gtk__error,
-#ifdef HAVE_GTK_INFO_BAR
+#ifdef HAVE_GTK_INFO_BAR_SUPPORT
 	.warning	= perf_gtk__warning_info_bar,
 #else
 	.warning	= perf_gtk__warning_statusbar,

tools/perf/ui/ui.h +2 -2

Index: tip/tools/perf/ui/ui.h
===================================================================
@@ -12,7 +12,7 @@ extern int use_browser;
 void setup_browser(bool fallback_to_pager);
 void exit_browser(bool wait_for_ok);
 
-#ifdef SLANG_SUPPORT
+#ifdef HAVE_SLANG_SUPPORT
 int ui__init(void);
 void ui__exit(bool wait_for_ok);
 #else

@@ -23,7 +23,7 @@ static inline int ui__init(void)
 static inline void ui__exit(bool wait_for_ok __maybe_unused) {}
 #endif
 
-#ifdef GTK2_SUPPORT
+#ifdef HAVE_GTK2_SUPPORT
 int perf_gtk__init(void);
 void perf_gtk__exit(bool wait_for_ok);
 #else

tools/perf/util/annotate.h +2 -2

Index: tip/tools/perf/util/annotate.h
===================================================================
@@ -150,7 +150,7 @@ int symbol__tty_annotate(struct symbol *
 			 struct perf_evsel *evsel, bool print_lines,
 			 bool full_paths, int min_pcnt, int max_lines);
 
-#ifdef SLANG_SUPPORT
+#ifdef HAVE_SLANG_SUPPORT
 int symbol__tui_annotate(struct symbol *sym, struct map *map,
 			 struct perf_evsel *evsel,
 			 struct hist_browser_timer *hbt);

@@ -165,7 +165,7 @@ static inline int symbol__tui_annotate(s
 }
 #endif
 
-#ifdef GTK2_SUPPORT
+#ifdef HAVE_GTK2_SUPPORT
 int symbol__gtk_annotate(struct symbol *sym, struct map *map,
 			 struct perf_evsel *evsel,
 			 struct hist_browser_timer *hbt);

tools/perf/util/cache.h +1 -1

Index: tip/tools/perf/util/cache.h
===================================================================
@@ -70,7 +70,7 @@ extern char *perf_path(const char *fmt,
 extern char *perf_pathdup(const char *fmt, ...)
 	__attribute__((format (printf, 1, 2)));
 
-#ifndef HAVE_STRLCPY
+#ifndef HAVE_STRLCPY_SUPPORT
 extern size_t strlcpy(char *dest, const char *src, size_t size);
 #endif

tools/perf/util/generate-cmdlist.sh +2 -2

Index: tip/tools/perf/util/generate-cmdlist.sh
===================================================================
@@ -22,7 +22,7 @@ do
      }' "Documentation/perf-$cmd.txt"
 done
 
-echo "#ifdef LIBELF_SUPPORT"
+echo "#ifdef HAVE_LIBELF_SUPPORT"
 sed -n -e 's/^perf-\([^ 	]*\)[ 	].* full.*/\1/p' command-list.txt |
 sort |
 while read cmd

@@ -35,5 +35,5 @@ do
 	    p
      }' "Documentation/perf-$cmd.txt"
 done
-echo "#endif /* LIBELF_SUPPORT */"
+echo "#endif /* HAVE_LIBELF_SUPPORT */"
 echo "};"

tools/perf/util/hist.h +2 -2

Index: tip/tools/perf/util/hist.h
===================================================================
@@ -183,7 +183,7 @@ struct hist_browser_timer {
 	int refresh;
 };
 
-#ifdef SLANG_SUPPORT
+#ifdef HAVE_SLANG_SUPPORT
 #include "../ui/keysyms.h"
 int hist_entry__tui_annotate(struct hist_entry *he, struct perf_evsel *evsel,
 			     struct hist_browser_timer *hbt);

@@ -224,7 +224,7 @@ static inline int script_browse(const ch
 #define K_SWITCH_INPUT_DATA -3000
 #endif
 
-#ifdef GTK2_SUPPORT
+#ifdef HAVE_GTK2_SUPPORT
 int perf_evlist__gtk_browse_hists(struct perf_evlist *evlist, const char *help,
 				  struct hist_browser_timer *hbt __maybe_unused,
 				  float min_pcnt);

tools/perf/util/include/dwarf-regs.h +1 -1

Index: tip/tools/perf/util/include/dwarf-regs.h
===================================================================
@@ -1,7 +1,7 @@
 #ifndef _PERF_DWARF_REGS_H_
 #define _PERF_DWARF_REGS_H_
 
-#ifdef DWARF_SUPPORT
+#ifdef HAVE_DWARF_SUPPORT
 const char *get_arch_regstr(unsigned int n);
 #endif

tools/perf/util/map.c +1 -1

Index: tip/tools/perf/util/map.c
===================================================================
@@ -172,7 +172,7 @@ int map__load(struct map *map, symbol_fi
 		pr_warning(", continuing without symbols\n");
 		return -1;
 	} else if (nr == 0) {
-#ifdef LIBELF_SUPPORT
+#ifdef HAVE_LIBELF_SUPPORT
 		const size_t len = strlen(name);
 		const size_t real_len = len - sizeof(DSO__DELETED);

tools/perf/util/path.c +1 -1

Index: tip/tools/perf/util/path.c
===================================================================
@@ -22,7 +22,7 @@ static const char *get_perf_dir(void)
 	return ".";
 }
 
-#ifndef HAVE_STRLCPY
+#ifndef HAVE_STRLCPY_SUPPORT
 size_t strlcpy(char *dest, const char *src, size_t size)
 {
 	size_t ret = strlen(src);

tools/perf/util/perf_regs.h +2 -2

Index: tip/tools/perf/util/perf_regs.h
===================================================================
@@ -1,7 +1,7 @@
 #ifndef __PERF_REGS_H
 #define __PERF_REGS_H
 
-#ifdef HAVE_PERF_REGS
+#ifdef HAVE_PERF_REGS_SUPPORT
 #include <perf_regs.h>
 #else
 #define PERF_REGS_MASK	0

@@ -10,5 +10,5 @@ static inline const char *perf_reg_name(
 {
 	return NULL;
 }
-#endif /* HAVE_PERF_REGS */
+#endif /* HAVE_PERF_REGS_SUPPORT */
 #endif /* __PERF_REGS_H */

tools/perf/util/probe-event.c +2 -2

Index: tip/tools/perf/util/probe-event.c
===================================================================
@@ -201,7 +201,7 @@ static int convert_to_perf_probe_point(s
 	return 0;
 }
 
-#ifdef DWARF_SUPPORT
+#ifdef HAVE_DWARF_SUPPORT
 /* Open new debuginfo of given module */
 static struct debuginfo *open_debuginfo(const char *module)
 {

@@ -630,7 +630,7 @@ int show_available_vars(struct perf_prob
 	return ret;
 }
 
-#else	/* !DWARF_SUPPORT */
+#else	/* !HAVE_DWARF_SUPPORT */
 
 static int kprobe_convert_to_perf_probe(struct probe_trace_point *tp,
 					struct perf_probe_point *pp)

tools/perf/util/probe-finder.h +2 -2

Index: tip/tools/perf/util/probe-finder.h
===================================================================
@@ -14,7 +14,7 @@ static inline int is_c_varname(const cha
 	return isalpha(name[0]) || name[0] == '_';
 }
 
-#ifdef DWARF_SUPPORT
+#ifdef HAVE_DWARF_SUPPORT
 
 #include "dwarf-aux.h"

@@ -105,6 +105,6 @@ struct line_finder {
 	int			found;
 };
 
-#endif /* DWARF_SUPPORT */
+#endif /* HAVE_DWARF_SUPPORT */
 
 #endif /*_PROBE_FINDER_H */

tools/perf/util/symbol-elf.c +1 -1

Index: tip/tools/perf/util/symbol-elf.c
===================================================================
@@ -8,7 +8,7 @@
 #include "symbol.h"
 #include "debug.h"
 
-#ifndef HAVE_ELF_GETPHDRNUM
+#ifndef HAVE_ELF_GETPHDRNUM_SUPPORT
 static int elf_getphdrnum(Elf *elf, size_t *dst)
 {
 	GElf_Ehdr gehdr;

tools/perf/util/symbol.h +4 -4

Index: tip/tools/perf/util/symbol.h
===================================================================
@@ -13,7 +13,7 @@
 #include <libgen.h>
 #include "build-id.h"
 
-#ifdef LIBELF_SUPPORT
+#ifdef HAVE_LIBELF_SUPPORT
 #include <libelf.h>
 #include <gelf.h>
 #endif

@@ -21,7 +21,7 @@
 
 #include "dso.h"
 
-#ifdef HAVE_CPLUS_DEMANGLE
+#ifdef HAVE_CPLUS_DEMANGLE_SUPPORT
 extern char *cplus_demangle(const char *, int);
 
 static inline char *bfd_demangle(void __maybe_unused *v, const char *c, int i)

@@ -46,7 +46,7 @@ static inline char *bfd_demangle(void __
  * libelf 0.8.x and earlier do not support ELF_C_READ_MMAP;
  * for newer versions we can use mmap to reduce memory usage:
  */
-#ifdef LIBELF_MMAP
+#ifdef HAVE_LIBELF_MMAP_SUPPORT
 # define PERF_ELF_C_READ_MMAP ELF_C_READ_MMAP
 #else
 # define PERF_ELF_C_READ_MMAP ELF_C_READ

@@ -178,7 +178,7 @@ struct symsrc {
 	int fd;
 	enum dso_binary_type type;
 
-#ifdef LIBELF_SUPPORT
+#ifdef HAVE_LIBELF_SUPPORT
 	Elf *elf;
 	GElf_Ehdr ehdr;

tools/perf/util/unwind.h +2 -2

Index: tip/tools/perf/util/unwind.h
===================================================================
@@ -13,7 +13,7 @@ struct unwind_entry {
 
 typedef int (*unwind_entry_cb_t)(struct unwind_entry *entry, void *arg);
 
-#ifdef LIBUNWIND_SUPPORT
+#ifdef HAVE_LIBUNWIND_SUPPORT
 int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
 			struct machine *machine,
 			struct thread *thread,

@@ -31,5 +31,5 @@ unwind__get_entries(unwind_entry_cb_t cb
 {
 	return 0;
 }
-#endif /* LIBUNWIND_SUPPORT */
+#endif /* HAVE_LIBUNWIND_SUPPORT */
 #endif /* __UNWIND_H */

tools/perf/util/util.c +2 -2

Index: tip/tools/perf/util/util.c
===================================================================
@@ -1,7 +1,7 @@
 #include "../perf.h"
 #include "util.h"
 #include <sys/mman.h>
-#ifdef BACKTRACE_SUPPORT
+#ifdef HAVE_BACKTRACE_SUPPORT
 #include <execinfo.h>
 #endif
 #include <stdio.h>

@@ -204,7 +204,7 @@ int hex2u64(const char *ptr, u64 *long_v
 }
 
 /* Obtain a backtrace and print it to stdout. */
-#ifdef BACKTRACE_SUPPORT
+#ifdef HAVE_BACKTRACE_SUPPORT
 void dump_stack(void)
 {
 	void *array[16];

Arnaldo Carvalho de Melo

2013-09-30 17:12 UTC #13 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Em Mon, Sep 30, 2013 at 06:42:10PM +0200, Ingo Molnar escreveu:

Quoted 3 lines

This series (with combo patch attached) implements (much) faster perf-tools feature-auto-detection.

Cool stuff!

Testing:

[acme@zoo linux]$ cat /etc/fedora-release
Fedora release 18 (Spherical Cow)
[acme@zoo linux]$ time make -j4 -C tools/perf O=/tmp/build/perf install-bin

Auto-detecting system features:

...            stackprotector-all: [ on  ]
...         volatile-register-var: [ on  ]
...                fortify-source: [ on  ]
...                        libelf: [ on  ]
...                   libelf-mmap: [ on  ]
...                         glibc: [ on  ]
...                         dwarf: [ on  ]
...             libelf-getphdrnum: [ on  ]
...                     libunwind: [ on  ]
...                      libaudit: [ on  ]
...                      libslang: [ on  ]
...                          gtk2: [ on  ]
...                  gtk2-infobar: [ on  ]
...                       libperl: [ on  ]
...                     libpython: [ on  ]
...             libpython-version: [ on  ]
...                        libbfd: [ on  ]
...                       strlcpy: [ OFF ]
...                       on-exit: [ on  ]
...                     backtrace: [ on  ]
...                       libnuma: [ on  ]

make: Entering directory `/home/git/linux/tools/perf'
MKDIR /tmp/build/perf/arch/

Checking why that strlcpy failed...

- Arnaldo

Arnaldo Carvalho de Melo

2013-09-30 17:28 UTC #14 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Em Mon, Sep 30, 2013 at 02:12:20PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 11 lines

Em Mon, Sep 30, 2013 at 06:42:10PM +0200, Ingo Molnar escreveu:
Quoted 3 lines
This series (with combo patch attached) implements (much) faster perf-tools feature-auto-detection.
Cool stuff!
Testing:
[acme@zoo linux]$ cat /etc/fedora-release
Fedora release 18 (Spherical Cow)

Before:

Performance counter stats for 'make -j4 -C tools/perf O=/tmp/build/perf' (3 runs):

       7.336208823 seconds time elapsed               ( +-  5.75% )

After:

       3.831295406 seconds time elapsed               ( +-  1.20% )

- Arnaldo

Arnaldo Carvalho de Melo

2013-09-30 17:31 UTC #15 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Em Mon, Sep 30, 2013 at 02:27:41PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 8 lines

Em Mon, Sep 30, 2013 at 02:12:20PM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 7 lines
Em Mon, Sep 30, 2013 at 06:42:10PM +0200, Ingo Molnar escreveu:
Quoted 2 lines
This series (with combo patch attached) implements (much) faster perf-tools feature-auto-detection.
Cool stuff!
Testing:

Removed python-devel and it correctly turned those off:

... libpython: [ OFF ]
... libpython-version: [ OFF ]

One possible way to speed this up a bit more would be to somehow state that if the test A fails, then don't even bother testing test B, like in the case above.

- Arnaldo

Arnaldo Carvalho de Melo

2013-09-30 17:36 UTC #16 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Em Mon, Sep 30, 2013 at 02:30:52PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 6 lines

Em Mon, Sep 30, 2013 at 02:27:41PM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 4 lines
Em Mon, Sep 30, 2013 at 02:12:20PM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 3 lines
Em Mon, Sep 30, 2013 at 06:42:10PM +0200, Ingo Molnar escreveu:
Quoted 2 lines
This series (with combo patch attached) implements (much) faster perf-tools feature-auto-detection.

Quoted 7 lines

Removed python-devel and it correctly turned those off:
... libpython: [ OFF ]
... libpython-version: [ OFF ]
One possible way to speed this up a bit more would be to somehow state that if the test A fails, then don't even bother testing test B, like in the case above.

Another suggestion: To provide a short sentence with each feature stating what will be not present when something is "OFF", or even a sentence stating what the feature is about, so the user get a better picture of what is (not) being built into his tool.

Probably reusing strings we already have in the makefiles.

- Arnaldo

Arnaldo Carvalho de Melo

2013-09-30 17:39 UTC #17 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Em Mon, Sep 30, 2013 at 02:36:27PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 6 lines

Another suggestion: To provide a short sentence with each feature stating what will be not present when something is "OFF", or even a sentence stating what the feature is about, so the user get a better picture of what is (not) being built into his tool.
Probably reusing strings we already have in the makefiles.

And also reusing existing strings to tell the user which devel packages should be installed to get that feature built/enabled.

- Arnaldo

Arnaldo Carvalho de Melo

2013-09-30 17:52 UTC #18 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Em Mon, Sep 30, 2013 at 02:39:14PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 10 lines

Em Mon, Sep 30, 2013 at 02:36:27PM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 6 lines
Another suggestion: To provide a short sentence with each feature stating what will be not present when something is "OFF", or even a sentence stating what the feature is about, so the user get a better picture of what is (not) being built into his tool.
Probably reusing strings we already have in the makefiles.
And also reusing existing strings to tell the user which devel packages should be installed to get that feature built/enabled.

config/Makefile:187: No libelf found, disables 'probe' tool, please install elfutils-libelf-devel/libelf-dev config/Makefile:251: No libunwind found, disabling post unwind support. Please install libunwind-dev[el] >= 0.99 config/Makefile:279: slang not found, disables TUI support. Please install slang-devel or libslang-dev config/Makefile:292: GTK2 not found, disables GTK2 support. Please install gtk2-devel or libgtk2.0-dev config/Makefile:346: The path '/usr/bin/python-config' is not executable.
config/Makefile:350: No python-config tool was found
config/Makefile:350: Python support will not be built
config/Makefile:444: No numa.h found, disables 'perf bench numa mem' benchmark, please install numa-libs-devel or libnuma-dev

My bad, its still there, sorry for the noise 8-)

I'm testing removing devel packages, one by one, removing gtk2-devel provides the biggest reduction in build time, no surprises :)

- Arnaldo

Arnaldo Carvalho de Melo

2013-09-30 18:03 UTC #19 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Em Mon, Sep 30, 2013 at 02:46:39PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 2 lines

I'm testing removing devel packages, one by one, removing gtk2-devel provides the biggest reduction in build time, no surprises :)

And this exercise spotted a but:

make[1]: Leaving directory `/home/git/linux/tools/lib/lk'
GEN /tmp/build/perf/python/perf.so
Context.xs:22:20: fatal error: EXTERN.h: No such file or directory
compilation terminated.
make: *** [/tmp/build/perf/scripts/perl/Perf-Trace-Util/Context.o] Error 1
make: *** Waiting for unfinished jobs.... util/scripting-engines/trace-event-perl.c:29:20: fatal error: EXTERN.h: No such file or directory compilation terminated.
make: *** [/tmp/build/perf/util/scripting-engines/trace-event-perl.o] Error 1
make: Leaving directory `/home/git/linux/tools/perf'

Performance counter stats for 'make -j4 -C tools/perf O=/tmp/build/perf' (3 runs):

6.453914638 seconds time elapsed ( +- 80.66% )

[acme@zoo linux]$
[acme@zoo linux]$

I.e. I did a 'yum remove python-devel', that removed some more packages that require this package, one of them is needed for perf to build, and this is not stated on those:

config/Makefile:444: No numa.h found, disables 'perf bench numa mem' benchmark, please install numa-libs-devel or libnuma-dev

In fact this is why I thought those messages had been missing in your patchset, they don't appear when I remove perl-devel, i.e. 'perl' supporty is correctly detected as not possible to build due to missing deps, but no message is emitted for this case.

And the bug is that the feature test for perl support passes, but since it doesn't try to use EXTERN.h, like something that is enabled when the feature test passes (util/scripting-engines/trace-event-perl.c), the build fails.

Notebook battery dying, will continue later...

- Arnaldo

Ingo Molnar

author

2013-09-30 19:15 UTC #20 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Quoted 9 lines · Arnaldo Carvalho de Melo

In fact this is why I thought those messages had been missing in your patchset, they don't appear when I remove perl-devel, i.e. 'perl' supporty is correctly detected as not possible to build due to missing deps, but no message is emitted for this case.
And the bug is that the feature test for perl support passes, but since it doesn't try to use EXTERN.h, like something that is enabled when the feature test passes (util/scripting-engines/trace-event-perl.c), the build fails.

Hm, I probably messed up the lib-perl testcase - it was one of the more complex ones. I assumed that this:

... libperl: [ OFF ]

meant that I didn't have the dependencies installed - but it's the testcase that is wrong most likely.

Thanks,

Ingo

Ingo Molnar

author

2013-09-30 19:09 UTC #21 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Quoted 11 lines · Arnaldo Carvalho de Melo

Em Mon, Sep 30, 2013 at 02:27:41PM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 8 lines
Em Mon, Sep 30, 2013 at 02:12:20PM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 7 lines
Em Mon, Sep 30, 2013 at 06:42:10PM +0200, Ingo Molnar escreveu:
Quoted 2 lines
This series (with combo patch attached) implements (much) faster perf-tools feature-auto-detection.
Cool stuff!
Testing:

Quoted 8 lines

Removed python-devel and it correctly turned those off:
... libpython: [ OFF ]
... libpython-version: [ OFF ]
One possible way to speed this up a bit more would be to somehow state that if the test A fails, then don't even bother testing test B, like in the case above.

Yeah, that's possible in a way, I did it in another case, see how the 'bionic' test for Android libc gets executed separately (but still unconditionally - we should probably only execute it if the libc test fails).

The 'FEATURE_TESTS' variable holds the 'flat' list of features that we want to definitely test.

Also, if glibc test passes then perhaps the strlcpy() test does not have to be executed at all.

In the initial version I wanted to maintain the existing feature dependency.

Thanks,

Ingo

Linus Torvalds

2013-09-30 17:34 UTC #22 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Quoted 2 lines · Arnaldo Carvalho de Melo

Checking why that strlcpy failed...

I don't think glibc does strlcpy. It's not a standard C function, and it's somewhat controversial (although I dislike strncpy more with the crazy zero-padding, ugh).

Linus

Arnaldo Carvalho de Melo

2013-09-30 17:53 UTC #23 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Em Mon, Sep 30, 2013 at 10:34:27AM -0700, Linus Torvalds escreveu:

Quoted 6 lines

Quoted 2 lines · Arnaldo Carvalho de Melo
Checking why that strlcpy failed...
I don't think glibc does strlcpy. It's not a standard C function, and

My concern was more about the thinking: ``Is this red "OFF" thing a problem? I feel so much more confortable when all entries have nice green "on" lights...''

Quoted 2 lines

it's somewhat controversial (although I dislike strncpy more with the crazy zero-padding, ugh).

:-)

- Arnaldo

Ingo Molnar

author

2013-09-30 19:04 UTC #24 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Quoted 11 lines · Arnaldo Carvalho de Melo

Em Mon, Sep 30, 2013 at 10:34:27AM -0700, Linus Torvalds escreveu:
Quoted 6 lines
Quoted 2 lines · Arnaldo Carvalho de Melo
Checking why that strlcpy failed...
I don't think glibc does strlcpy. It's not a standard C function, and
My concern was more about the thinking: ``Is this red "OFF" thing a problem? I feel so much more confortable when all entries have nice green "on" lights...''

Yeah, so I think we should add our internal implementation of strlcpy() as a __weak function instead - if the libc does not provide then we provide a fallback.

That should get rid of another ~50 msecs of build overhead, as failed feature tests are the most expensive ones.

Agreed?

Thanks,

Ingo

Ingo Molnar

author

2013-10-01 11:35 UTC #25 lore raw

PATCH 1/1 perf autodep: Remove strlcpy feature check, add __weak strlcpy implementation

Quoted 9 lines · Ingo Molnar

Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Checking why that strlcpy failed...
I don't think glibc does strlcpy. It's not a standard C function, and
My concern was more about the thinking: ``Is this red "OFF" thing a problem? I feel so much more confortable when all entries have nice green "on" lights...''

Quoted 6 lines

Yeah, so I think we should add our internal implementation of strlcpy() as a __weak function instead - if the libc does not provide then we provide a fallback.
That should get rid of another ~50 msecs of build overhead, as failed feature tests are the most expensive ones.

The patch below implements that. I haven't actually tested it on a system with a in-libc strlcpy implementation, but it Should Just Work (tm) ;-)

Overhead is down from 0.600 secs to 0.540 secs. The only remaining thing is the libperl bug, I'll have a look at that next.

( I also couldn't resist fixing up perf's version of compiler.h a bit,
will split that out into a separate patch later on. )

Thanks,

Ingo

=====================>
Subject: perf autodep: Remove strlcpy feature check, add __weak strlcpy implementation
From: Ingo Molnar <[email protected]>
Date: Tue Oct 1 13:26:13 CEST 2013

---
 tools/perf/config/Makefile                      |    8 +-------
 tools/perf/config/feature-checks/Makefile       |    3 ---
 tools/perf/config/feature-checks/test-strlcpy.c |    8 --------
 tools/perf/util/cache.h                         |    3 +--
 tools/perf/util/include/linux/compiler.h        |   19 ++++++++++++++-----
 tools/perf/util/path.c                          |   10 +++++++---
 6 files changed, 23 insertions(+), 28 deletions(-)

tools/perf/config/Makefile +1 -7

Index: tip/tools/perf/config/Makefile
===================================================================
@@ -101,7 +101,7 @@ $(info )
 $(info Auto-detecting system features:)
 $(shell make -i -j -C config/feature-checks >/dev/null 2>&1)
 
-FEATURE_TESTS = stackprotector-all volatile-register-var fortify-source libelf libelf-mmap glibc dwarf libelf-getphdrnum libunwind libaudit libslang gtk2 gtk2-infobar libperl libpython libpython-version libbfd strlcpy on-exit backtrace libnuma
+FEATURE_TESTS = stackprotector-all volatile-register-var fortify-source libelf libelf-mmap glibc dwarf libelf-getphdrnum libunwind libaudit libslang gtk2 gtk2-infobar libperl libpython libpython-version libbfd on-exit backtrace libnuma
 
 $(foreach feat,$(FEATURE_TESTS),$(call feature_check,$(feat)))

@@ -421,12 +421,6 @@ else
   endif
 endif
 
-ifndef NO_STRLCPY
-  ifeq ($(feature-strlcpy), 1)
-    CFLAGS += -DHAVE_STRLCPY_SUPPORT
-  endif
-endif
-
 ifndef NO_ON_EXIT
   ifeq ($(feature-on-exit), 1)
     CFLAGS += -DHAVE_ON_EXIT_SUPPORT

tools/perf/config/feature-checks/Makefile +0 -3

Index: tip/tools/perf/config/feature-checks/Makefile
===================================================================
@@ -93,9 +93,6 @@ test-libpython-version: test-libpython-v
 test-libbfd: test-libbfd.c
 	$(CC) -o $@ [email protected] -DPACKAGE='perf' -DPACKAGE=perf -lbfd -ldl
 
-test-strlcpy: test-strlcpy.c
-	$(CC) -o $@ [email protected]
-
 test-on-exit: test-on-exit.c
 	$(CC) -o $@ [email protected]

tools/perf/config/feature-checks/test-strlcpy.c deleted +0 -8

Index: tip/tools/perf/config/feature-checks/test-strlcpy.c
===================================================================
@@ -1,8 +0,0 @@
-#include <stdlib.h>
-extern size_t strlcpy(char *dest, const char *src, size_t size);
-
-int main(void)
-{
-	strlcpy(NULL, NULL, 0);
-	return 0;
-}

tools/perf/util/cache.h +1 -2

Index: tip/tools/perf/util/cache.h
===================================================================
@@ -70,8 +70,7 @@ extern char *perf_path(const char *fmt,
 extern char *perf_pathdup(const char *fmt, ...)
 	__attribute__((format (printf, 1, 2)));
 
-#ifndef HAVE_STRLCPY_SUPPORT
+/* Matches the libc/libbsd function attribute so we declare this unconditionally: */
 extern size_t strlcpy(char *dest, const char *src, size_t size);
-#endif
 
 #endif /* __PERF_CACHE_H */

tools/perf/util/include/linux/compiler.h +14 -5

Index: tip/tools/perf/util/include/linux/compiler.h
===================================================================
@@ -2,20 +2,29 @@
 #define _PERF_LINUX_COMPILER_H_
 
 #ifndef __always_inline
-#define __always_inline	inline
+# define __always_inline	inline __attribute__((always_inline))
 #endif
+
 #define __user
+
 #ifndef __attribute_const__
-#define __attribute_const__
+# define __attribute_const__
 #endif
 
 #ifndef __maybe_unused
-#define __maybe_unused		__attribute__((unused))
+# define __maybe_unused		__attribute__((unused))
+#endif
+
+#ifndef __packed
+# define __packed		__attribute__((__packed__))
 #endif
-#define __packed	__attribute__((__packed__))
 
 #ifndef __force
-#define __force
+# define __force
+#endif
+
+#ifndef __weak
+# define __weak			__attribute__((weak))
 #endif
 
 #endif

tools/perf/util/path.c +7 -3

Index: tip/tools/perf/util/path.c
===================================================================
@@ -22,19 +22,23 @@ static const char *get_perf_dir(void)
 	return ".";
 }
 
-#ifndef HAVE_STRLCPY_SUPPORT
-size_t strlcpy(char *dest, const char *src, size_t size)
+/*
+ * If libc has strlcpy() then that version will override this
+ * implementation:
+ */
+size_t __weak strlcpy(char *dest, const char *src, size_t size)
 {
 	size_t ret = strlen(src);
 
 	if (size) {
 		size_t len = (ret >= size) ? size - 1 : ret;
+
 		memcpy(dest, src, len);
 		dest[len] = '\0';
 	}
+
 	return ret;
 }
-#endif
 
 static char *get_pathname(void)
 {

Ingo Molnar

author

2013-10-01 12:05 UTC #26 lore raw

Re: [PATCH] perf autodep: Remove strlcpy feature check, add __weak strlcpy implementation

Quoted 2 lines · Ingo Molnar

Overhead is down from 0.600 secs to 0.540 secs. The only remaining thing is the libperl bug, I'll have a look at that next.

So, libperl detection works fine here, once I've installed the prereq package on Fedora, "perl-ExtUtils-Embed":

comet:~/tip/tools/perf> make Makefile

Auto-detecting system features:

...            stackprotector-all: [ on  ]
...         volatile-register-var: [ on  ]
...                fortify-source: [ on  ]
...                        libelf: [ on  ]
...                   libelf-mmap: [ on  ]
...                         glibc: [ on  ]
...                         dwarf: [ on  ]
...             libelf-getphdrnum: [ on  ]
...                     libunwind: [ on  ]
...                      libaudit: [ on  ]
...                      libslang: [ on  ]
...                          gtk2: [ on  ]
...                  gtk2-infobar: [ on  ]
...                       libperl: [ on  ]
...                     libpython: [ on  ]
...             libpython-version: [ on  ]
...                        libbfd: [ on  ]
...                       on-exit: [ on  ]
...                     backtrace: [ on  ]
...                       libnuma: [ on  ]

Time is down to 0.480 sec because there are no build failures now, only Make re-checking the dependencies of already built binaries.

And the actual feature check is roughly 0.330 msecs of that:

comet:~/tip/tools/perf/config/feature-checks> time ( make -j >/dev/null; \
for N in stackprotector-all volatile-register-var fortify-source libelf \
libelf-mmap glibc dwarf libelf-getphdrnum libunwind libaudit libslang gtk2 \
gtk2-infobar libperl libpython libpython-version libbfd on-exit backtrace \
libnuma; do make test-$N >/dev/null; done )

 real    0m0.330s
 user    0m0.290s
 sys     0m0.031s

With 0.150 msecs spent elsewhere.

So there's more speedups possible I think, for example we could construct an 'optimistic' testcase that is generated live and includes a concatenation of all the testcases.

If the build of that file succeeds then we have a really efficient fast-path both in the first-build and in the repeat-build case.

If that build fails then we do the more finegrained feature check.

Thoughts?

Thanks,

Ingo

Arnaldo Carvalho de Melo

2013-10-01 12:48 UTC #27 lore raw

Re: [PATCH] perf autodep: Remove strlcpy feature check, add __weak strlcpy implementation

Em Tue, Oct 01, 2013 at 02:04:55PM +0200, Ingo Molnar escreveu:

Quoted 10 lines

So there's more speedups possible I think, for example we could construct an 'optimistic' testcase that is generated live and includes a concatenation of all the testcases.
If the build of that file succeeds then we have a really efficient fast-path both in the first-build and in the repeat-build case.
If that build fails then we do the more finegrained feature check.
Thoughts?

Lets get what you have merged and continue from there ;-)

- Arnaldo

Ingo Molnar

author

2013-10-01 12:51 UTC #28 lore raw

PATCH 1/1 perf autodep: Speed up the 'all features are present' case

Quoted 18 lines · Ingo Molnar

And the actual feature check is roughly 0.330 msecs of that:
comet:~/tip/tools/perf/config/feature-checks> time ( make -j >/dev/null; \
for N in stackprotector-all volatile-register-var fortify-source libelf \
libelf-mmap glibc dwarf libelf-getphdrnum libunwind libaudit libslang gtk2 \
gtk2-infobar libperl libpython libpython-version libbfd on-exit backtrace \
libnuma; do make test-$N >/dev/null; done )
 real    0m0.330s
 user    0m0.290s
 sys     0m0.031s
With 0.150 msecs spent elsewhere.
So there's more speedups possible I think, for example we could construct an 'optimistic' testcase that is generated live and includes a concatenation of all the testcases.

Quoted 6 lines

If the build of that file succeeds then we have a really efficient fast-path both in the first-build and in the repeat-build case.
If that build fails then we do the more finegrained feature check.
Thoughts?

So, something like the patch below. It contains test-all.c "testcase from hell" which will succeed on a well configured system.

With this final trick I got a ridiculous speedup in auto-detection speed, for the rebuild-again case:

comet:~/tip/tools/perf> perf stat --null --repeat 5 make Makefile

Auto-detecting system features:

 ...            stackprotector-all: [ on  ]
 ...         volatile-register-var: [ on  ]
 ...                fortify-source: [ on  ]
 ...                        libelf: [ on  ]
 ...                   libelf-mmap: [ on  ]
 ...                         glibc: [ on  ]
 ...                         dwarf: [ on  ]
 ...             libelf-getphdrnum: [ on  ]
 ...                     libunwind: [ on  ]
 ...                      libaudit: [ on  ]
 ...                      libslang: [ on  ]
 ...                          gtk2: [ on  ]
 ...                  gtk2-infobar: [ on  ]
 ...                       libperl: [ on  ]
 ...                     libpython: [ on  ]
 ...             libpython-version: [ on  ]
 ...                        libbfd: [ on  ]
 ...                       on-exit: [ on  ]
 ...                     backtrace: [ on  ]
 ...                       libnuma: [ on  ]

make: Nothing to be done for `Makefile'.

Performance counter stats for 'make Makefile' (5 runs):

0.183843005 seconds time elapsed ( +- 0.21% )

So the cached build time is down from 3.300 secs to 0.183 secs, an 18-fold speedup.

It's still full auto-detection, because the -MD dependency generation works for test-all.c as well. For example once I remove the 'perl-ExtUtils-Embed' package, I immediately get the correct, adapted auto-dep output with the 'libperl' testcase showing 'OFF':

comet:~/tip/tools/perf> make Makefile

Auto-detecting system features:

...            stackprotector-all: [ on  ]
...         volatile-register-var: [ on  ]
...                fortify-source: [ on  ]
...                        libelf: [ on  ]
...                   libelf-mmap: [ on  ]
...                         glibc: [ on  ]
...                         dwarf: [ on  ]
...             libelf-getphdrnum: [ on  ]
...                     libunwind: [ on  ]
...                      libaudit: [ on  ]
...                      libslang: [ on  ]
...                          gtk2: [ on  ]
...                  gtk2-infobar: [ on  ]
...                       libperl: [ OFF ]
...                     libpython: [ on  ]
...             libpython-version: [ on  ]
...                        libbfd: [ on  ]
...                       on-exit: [ on  ]
...                     backtrace: [ on  ]
...                       libnuma: [ on  ]

Note that at this point this is just a proof-of-concept patch for performance testing, nothing mergable yet.

In particular I'm not happy yet with the current construction of test-all.c and its build method: it's the result of concatenation of files and build arguments plus further massaging.

That should probably be automated in some fashion, to make it easier to add new testcases and to make the whole construct more maintainable.

Thanks,

Ingo

=========================>
Subject: perf autodep: Speed up the 'all features are present' case
From: Ingo Molnar <[email protected]>
Date: Tue Oct 1 14:14:31 CEST 2013

---
 tools/perf/config/Makefile                  |   18 ++
 tools/perf/config/feature-checks/Makefile   |    3 
 tools/perf/config/feature-checks/test-all.c |  196 ++++++++++++++++++++++++++++
 3 files changed, 215 insertions(+), 2 deletions(-)

tools/perf/config/Makefile +16 -2

Index: tip/tools/perf/config/Makefile
===================================================================
@@ -94,16 +94,30 @@ define feature_check_code
   feature-$(1) := $(shell make -C config/feature-checks test-$1 >/dev/null 2>/dev/null && echo 1 || echo 0)
 endef
 
+feature_set = $(eval $(feature_set_code))
+define feature_set_code
+  feature-$(1) := 1
+endef
+
 #
 # Build the feature check binaries in parallel, ignore errors, ignore return value and suppress output:
 #
 $(info )
 $(info Auto-detecting system features:)
-$(shell make -i -j -C config/feature-checks >/dev/null 2>&1)
 
 FEATURE_TESTS = stackprotector-all volatile-register-var fortify-source libelf libelf-mmap glibc dwarf libelf-getphdrnum libunwind libaudit libslang gtk2 gtk2-infobar libperl libpython libpython-version libbfd on-exit backtrace libnuma
 
-$(foreach feat,$(FEATURE_TESTS),$(call feature_check,$(feat)))
+#
+# Special fast-path for the 'all features are available' case:
+#
+$(call feature_check,all)
+
+ifeq ($(feature-all), 1)
+  $(foreach feat,$(FEATURE_TESTS),$(call feature_set,$(feat)))
+else
+  $(shell make -i -j -C config/feature-checks >/dev/null 2>&1)
+  $(foreach feat,$(FEATURE_TESTS),$(call feature_check,$(feat)))
+endif
 
 feature_print = $(eval $(feature_print_code))

tools/perf/config/feature-checks/Makefile +3 -0

Index: tip/tools/perf/config/feature-checks/Makefile
===================================================================
@@ -99,6 +99,9 @@ test-on-exit: test-on-exit.c
 test-backtrace: test-backtrace.c
 	$(CC) -o $@ [email protected]
 
+test-all:
+	$(CC) -o $@ [email protected] -Werror -fstack-protector -Wvolatile-register-var -O2 -Werror -D_FORTIFY_SOURCE=2 -ldw -lelf -lnuma -lunwind -lunwind-x86_64 -lelf -laudit -I/usr/include/slang -lslang $(shell pkg-config --libs --cflags gtk+-2.0 2>/dev/null) $(FLAGS_PERL_EMBED) $(FLAGS_PYTHON_EMBED) -DPACKAGE='perf' -DPACKAGE=perf -lbfd -ldl
+
 -include *.d */*.d
 
 ###############################

tools/perf/config/feature-checks/test-all.c new file +196 -0

196 lines not on this page · raw message

Ingo Molnar

author

2013-10-01 14:46 UTC #29 lore raw

PATCH 1/1 perf tools: Speed up git-version test on re-make

Another build speedup, while at it.

===============>
Subject: perf tools: Speed up git-version test on re-make
From: Ingo Molnar <[email protected]>
Date: Tue Oct 1 16:28:09 CEST 2013

util/PERF-VERSION-GEN is currently executed on every build attempt, and this script can take a lot of time on trees that are at a significant git-distance from Linus's tree:

  $ time util/PERF-VERSION-GEN

  real    0m4.343s
  user    0m4.176s
  sys     0m0.140s

It also takes a lot of time if the Git repository is network attached, etc., because the commands it uses:

        TAG=$(git describe --abbrev=0 --match "v[0-9].[0-9]*" 2>/dev/null )

has to count commits from the nearest tag and thus has to access (and decompress) every git commit blob on the relevant version path.

Even on Linus's tree it takes 0.28 seconds on a fast box to count all the commits and get the git version string:

  $ time util/PERF-VERSION-GEN

  real    0m0.279s
  user    0m0.247s
  sys     0m0.025s

But the version string only has to be regenerated if the git repository's head commit changes. So add a dependency of ../../.git/HEAD and touch the file every time it's regenerated, so that Make's build rules can pick it up and cache the result:

make: `PERF-VERSION-FILE' is up to date.

  real    0m0.184s
  user    0m0.117s
  sys     0m0.026s

So the version generation overhead is completely gone for re-builds.

Signed-off-by Ingo Molnar <[email protected]>

---
 tools/perf/Makefile |   15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

tools/perf/Makefile +13 -2

Index: tip/tools/perf/Makefile
===================================================================
@@ -70,8 +70,9 @@ ifneq ($(OUTPUT),)
 #$(info Determined 'OUTPUT' to be $(OUTPUT))
 endif
 
-$(OUTPUT)PERF-VERSION-FILE: .FORCE-PERF-VERSION-FILE
+$(OUTPUT)PERF-VERSION-FILE: ../../.git/HEAD
 	@$(SHELL_PATH) util/PERF-VERSION-GEN $(OUTPUT)
+	@touch $(OUTPUT)PERF-VERSION-FILE
 
 CC = $(CROSS_COMPILE)gcc
 AR = $(CROSS_COMPILE)ar

@@ -813,7 +814,17 @@ clean: $(LIBTRACEEVENT)-clean $(LIBLK)-c
 	$(RM) $(OUTPUT)util/*-flex*
 	$(python-clean)
 
+#
+# Trick: if ../../.git does not exist - we are building out of tree for example,
+# then force version regeneration:
+#
+ifeq ($(wildcard ../../.git/HEAD),) 
+    GIT-HEAD-PHONY = ../../.git/HEAD
+else 
+    GIT-HEAD-PHONY =
+endif
+
 .PHONY: all install clean strip $(LIBTRACEEVENT) $(LIBLK)
 .PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell
-.PHONY: .FORCE-PERF-VERSION-FILE TAGS tags cscope .FORCE-PERF-CFLAGS
+.PHONY: $(GIT-HEAD-PHONY) TAGS tags cscope .FORCE-PERF-CFLAGS

Namhyung Kim

maintainer

2013-10-02 06:47 UTC #30 lore raw

Re: [PATCH] perf tools: Speed up git-version test on re-make

On Tue, 1 Oct 2013 16:46:18 +0200, Ingo Molnar wrote:

[SNIP]

Quoted 17 lines

-$(OUTPUT)PERF-VERSION-FILE: .FORCE-PERF-VERSION-FILE +$(OUTPUT)PERF-VERSION-FILE: ../../.git/HEAD

tools/perf/Makefile +6 -0

 	@$(SHELL_PATH) util/PERF-VERSION-GEN $(OUTPUT)
+	@touch $(OUTPUT)PERF-VERSION-FILE
 
 CC = $(CROSS_COMPILE)gcc
 AR = $(CROSS_COMPILE)ar

@@ -813,7 +814,17 @@ clean: $(LIBTRACEEVENT)-clean $(LIBLK)-c
 	$(RM) $(OUTPUT)util/*-flex*
 	$(python-clean)
 
+#
+# Trick: if ../../.git does not exist - we are building out of tree for example,
+# then force version regeneration:
+#
+ifeq ($(wildcard ../../.git/HEAD),)

tools/perf/Makefile clean: $(LIBTRACEEVENT)-clean $(LIBLK)-c · line 830

Quoted 10 lines

tools/perf/Makefile +6 -1

+    GIT-HEAD-PHONY = ../../.git/HEAD
+else 
+    GIT-HEAD-PHONY =
+endif
+
 .PHONY: all install clean strip $(LIBTRACEEVENT) $(LIBLK)
 .PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell
-.PHONY: .FORCE-PERF-VERSION-FILE TAGS tags cscope .FORCE-PERF-CFLAGS
+.PHONY: $(GIT-HEAD-PHONY) TAGS tags cscope .FORCE-PERF-CFLAGS

I found an unrelated bug. When I tried to build perf out of tree after 'make perf-targz-src-pkg' it cannot get any version number.

GEN common-cmds.h
* new build flags or prefix
make[1]: *** No rule to make target `kernelversion'. Stop. PERF_VERSION =

      ...
      AR libperf.a
      LINK perf

  $ ./perf --version
  perf version

It looks like we should save and pass the result of 'make kernelversion' to the tarball.

Thanks,
Namhyung

Ingo Molnar

author

2013-10-02 06:50 UTC #31 lore raw

Re: [PATCH] perf tools: Speed up git-version test on re-make

Quoted 38 lines · Namhyung Kim

On Tue, 1 Oct 2013 16:46:18 +0200, Ingo Molnar wrote:
[SNIP]
Quoted 17 lines
-$(OUTPUT)PERF-VERSION-FILE: .FORCE-PERF-VERSION-FILE +$(OUTPUT)PERF-VERSION-FILE: ../../.git/HEAD
tools/perf/Makefile +6 -0
 	@$(SHELL_PATH) util/PERF-VERSION-GEN $(OUTPUT)
+	@touch $(OUTPUT)PERF-VERSION-FILE
 
 CC = $(CROSS_COMPILE)gcc
 AR = $(CROSS_COMPILE)ar
@@ -813,7 +814,17 @@ clean: $(LIBTRACEEVENT)-clean $(LIBLK)-c
 	$(RM) $(OUTPUT)util/*-flex*
 	$(python-clean)
 
+#
+# Trick: if ../../.git does not exist - we are building out of tree for example,
+# then force version regeneration:
+#
+ifeq ($(wildcard ../../.git/HEAD),) 
tools/perf/Makefile clean: $(LIBTRACEEVENT)-clean $(LIBLK)-c · line 830
Quoted 10 lines
tools/perf/Makefile +6 -1
+    GIT-HEAD-PHONY = ../../.git/HEAD
+else 
+    GIT-HEAD-PHONY =
+endif
+
 .PHONY: all install clean strip $(LIBTRACEEVENT) $(LIBLK)
 .PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell
-.PHONY: .FORCE-PERF-VERSION-FILE TAGS tags cscope .FORCE-PERF-CFLAGS
+.PHONY: $(GIT-HEAD-PHONY) TAGS tags cscope .FORCE-PERF-CFLAGS
 
I found an unrelated bug. When I tried to build perf out of tree after 'make perf-targz-src-pkg' it cannot get any version number.
GEN common-cmds.h
* new build flags or prefix
make[1]: *** No rule to make target `kernelversion'. Stop. PERF_VERSION =

Quoted 6 lines

      ...
      AR libperf.a
      LINK perf

  $ ./perf --version
  perf version

Is this unrelated to my patches?

Quoted 2 lines

It looks like we should save and pass the result of 'make kernelversion' to the tarball.

Or we could simply generate PERF-VERSION-FILE and save it into the tarball?

Then it will be the most accurate version we can determine at the point when the tarball is generated.

Thanks,

Ingo

Namhyung Kim

maintainer

2013-10-02 08:04 UTC #32 lore raw

Re: [PATCH] perf tools: Speed up git-version test on re-make

Quoted 44 lines · Ingo Molnar

Quoted 42 lines · Namhyung Kim
On Tue, 1 Oct 2013 16:46:18 +0200, Ingo Molnar wrote:
[SNIP]
Quoted 17 lines
-$(OUTPUT)PERF-VERSION-FILE: .FORCE-PERF-VERSION-FILE +$(OUTPUT)PERF-VERSION-FILE: ../../.git/HEAD
tools/perf/Makefile +6 -0
 	@$(SHELL_PATH) util/PERF-VERSION-GEN $(OUTPUT)
+	@touch $(OUTPUT)PERF-VERSION-FILE
 
 CC = $(CROSS_COMPILE)gcc
 AR = $(CROSS_COMPILE)ar
@@ -813,7 +814,17 @@ clean: $(LIBTRACEEVENT)-clean $(LIBLK)-c
 	$(RM) $(OUTPUT)util/*-flex*
 	$(python-clean)
 
+#
+# Trick: if ../../.git does not exist - we are building out of tree for example,
+# then force version regeneration:
+#
+ifeq ($(wildcard ../../.git/HEAD),) 
tools/perf/Makefile clean: $(LIBTRACEEVENT)-clean $(LIBLK)-c · line 830
Quoted 10 lines
tools/perf/Makefile +6 -1
+    GIT-HEAD-PHONY = ../../.git/HEAD
+else 
+    GIT-HEAD-PHONY =
+endif
+
 .PHONY: all install clean strip $(LIBTRACEEVENT) $(LIBLK)
 .PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell
-.PHONY: .FORCE-PERF-VERSION-FILE TAGS tags cscope .FORCE-PERF-CFLAGS
+.PHONY: $(GIT-HEAD-PHONY) TAGS tags cscope .FORCE-PERF-CFLAGS
 
I found an unrelated bug. When I tried to build perf out of tree after 'make perf-targz-src-pkg' it cannot get any version number.
GEN common-cmds.h
* new build flags or prefix
make[1]: *** No rule to make target `kernelversion'. Stop. PERF_VERSION =
      ...
      AR libperf.a
      LINK perf

Quoted 4 lines

Quoted 2 lines
  $ ./perf --version
  perf version 
Is this unrelated to my patches?

Yes, I can reproduce it without your patch.

Quoted 9 lines

Quoted 3 lines
It looks like we should save and pass the result of 'make kernelversion' to the tarball.
Or we could simply generate PERF-VERSION-FILE and save it into the tarball?
Then it will be the most accurate version we can determine at the point when the tarball is generated.

Hmm.. I tried to do it, but it's not that simple than I thought. With your patch, the PERF-VERSION-FILE is regenerated everytime if it can't find .git/HEAD so saving the file didn't help since it'll be overwritten. Anyway, it also cannot be used with 'make O=...' build.

Thanks,
Namhyung

Ingo Molnar

author

2013-10-01 15:28 UTC #33 lore raw

Re: [PATCH] perf autodep: Remove strlcpy feature check, add __weak strlcpy implementation

Ok, this should be my final perf-build speedup patch.

With this patch and all the other patches applied perf delta-builds very fast now - an empty re-build takes just 0.2 seconds:

comet:~/tip/tools/perf> time make

  real    0m0.207s
  user    0m0.130s
  sys     0m0.034s

and the rebuild after a single .c file was changed is just 1.8 seconds:

comet:~/tip/tools/perf> touch perf.c; time make

  real    0m1.892s
  user    0m1.495s
  sys     0m0.337s

Without the changes this used to be 9.4 seconds:

comet:~/tip/tools/perf> touch perf.c; time make

  real    0m9.418s
  user    0m8.251s
  sys     0m0.996s

which was an eternity! :-)

Thanks,

Ingo

------------------------>
Subject: perf tools: Speed up the final link
From: Ingo Molnar <[email protected]>
Date: Tue Oct 1 17:17:22 CEST 2013

libtraceevent.a and liblk.a rules have always-missed dependencies, which causes python.so to be relinked at every build attempt - even if none of the affected code changes.

This slows down re-builds unnecessarily, by adding more than a second to the build time:

comet:~/tip/tools/perf> time make

...

SUBDIR /fast/mingo/tip/tools/lib/lk/
make[1]: `liblk.a' is up to date.
SUBDIR /fast/mingo/tip/tools/lib/traceevent/
LINK perf
GEN python/perf.so

  real    0m1.701s
  user    0m1.338s
  sys     0m0.301s

Add the (trivial) dependencies to not force a re-link.

This speeds up an empty re-build enormously:

comet:~/tip/tools/perf> time make

...

  real    0m0.207s
  user    0m0.134s
  sys     0m0.028s

[ This adds some coupling between the build dependencies of
libtraceevent and liblk - but until those stay relatively simple this should not be an issue. ]

Signed-off-by Ingo Molnar <[email protected]>

---
 tools/perf/Makefile |   10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

tools/perf/Makefile +7 -3

Index: tip/tools/perf/Makefile
===================================================================
@@ -669,15 +669,19 @@ $(LIB_FILE): $(LIB_OBJS)
 	$(QUIET_AR)$(RM) $@ && $(AR) rcs $@ $(LIB_OBJS)
 
 # libtraceevent.a
-$(LIBTRACEEVENT):
+TE_SOURCES = $(wildcard $(TRACE_EVENT_DIR)*.[ch])
+
+$(LIBTRACEEVENT): $(TE_SOURCES)
 	$(QUIET_SUBDIR0)$(TRACE_EVENT_DIR) $(QUIET_SUBDIR1) O=$(OUTPUT) libtraceevent.a
 
 $(LIBTRACEEVENT)-clean:
 	$(QUIET_SUBDIR0)$(TRACE_EVENT_DIR) $(QUIET_SUBDIR1) O=$(OUTPUT) clean
 
+LIBLK_SOURCES = $(wildcard $(LK_PATH)*.[ch])
+
 # if subdir is set, we've been called from above so target has been built
 # already
-$(LIBLK):
+$(LIBLK): $(LIBLK_SOURCES)
 ifeq ($(subdir),)
 	$(QUIET_SUBDIR0)$(LK_DIR) $(QUIET_SUBDIR1) O=$(OUTPUT) liblk.a
 endif

@@ -824,7 +828,7 @@ else
     GIT-HEAD-PHONY =
 endif
 
-.PHONY: all install clean strip $(LIBTRACEEVENT) $(LIBLK)
+.PHONY: all install clean strip
 .PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell
 .PHONY: $(GIT-HEAD-PHONY) TAGS tags cscope .FORCE-PERF-CFLAGS

Ingo Molnar

author

2013-10-01 15:29 UTC #34 lore raw

PATCH 1/1 perf tools: Speed up the final link

(Sent with proper subject line.)

=================>
libtraceevent.a and liblk.a rules have always-missed dependencies, which causes python.so to be relinked at every build attempt - even if none of the affected code changes.

This slows down re-builds unnecessarily, by adding more than a second to the build time:

comet:~/tip/tools/perf> time make

...

SUBDIR /fast/mingo/tip/tools/lib/lk/
make[1]: `liblk.a' is up to date.
SUBDIR /fast/mingo/tip/tools/lib/traceevent/
LINK perf
GEN python/perf.so

  real    0m1.701s
  user    0m1.338s
  sys     0m0.301s

Add the (trivial) dependencies to not force a re-link.

This speeds up an empty re-build enormously:

comet:~/tip/tools/perf> time make

...

  real    0m0.207s
  user    0m0.134s
  sys     0m0.028s

[ This adds some coupling between the build dependencies of
libtraceevent and liblk - but until those stay relatively simple this should not be an issue. ]

Signed-off-by Ingo Molnar <[email protected]>

---
 tools/perf/Makefile |   10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

tools/perf/Makefile +7 -3

Index: tip/tools/perf/Makefile
===================================================================
@@ -669,15 +669,19 @@ $(LIB_FILE): $(LIB_OBJS)
 	$(QUIET_AR)$(RM) $@ && $(AR) rcs $@ $(LIB_OBJS)
 
 # libtraceevent.a
-$(LIBTRACEEVENT):
+TE_SOURCES = $(wildcard $(TRACE_EVENT_DIR)*.[ch])
+
+$(LIBTRACEEVENT): $(TE_SOURCES)
 	$(QUIET_SUBDIR0)$(TRACE_EVENT_DIR) $(QUIET_SUBDIR1) O=$(OUTPUT) libtraceevent.a
 
 $(LIBTRACEEVENT)-clean:
 	$(QUIET_SUBDIR0)$(TRACE_EVENT_DIR) $(QUIET_SUBDIR1) O=$(OUTPUT) clean
 
+LIBLK_SOURCES = $(wildcard $(LK_PATH)*.[ch])
+
 # if subdir is set, we've been called from above so target has been built
 # already
-$(LIBLK):
+$(LIBLK): $(LIBLK_SOURCES)
 ifeq ($(subdir),)
 	$(QUIET_SUBDIR0)$(LK_DIR) $(QUIET_SUBDIR1) O=$(OUTPUT) liblk.a
 endif

@@ -824,7 +828,7 @@ else
     GIT-HEAD-PHONY =
 endif
 
-.PHONY: all install clean strip $(LIBTRACEEVENT) $(LIBLK)
+.PHONY: all install clean strip
 .PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell
 .PHONY: $(GIT-HEAD-PHONY) TAGS tags cscope .FORCE-PERF-CFLAGS

Geert Uytterhoeven

2013-10-01 07:04 UTC #35 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Quoted 8 lines · Linus Torvalds

Quoted 2 lines · Arnaldo Carvalho de Melo
Checking why that strlcpy failed...
I don't think glibc does strlcpy. It's not a standard C function, and it's somewhat controversial (although I dislike strncpy more with the crazy zero-padding, ugh).

Indeed, e.g. on Ubuntu it's in libbsd, so you also need to #include <bsd/string.h> and link with -lbsd.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

Ingo Molnar

author

2013-10-01 08:38 UTC #36 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Quoted 5 lines · Geert Uytterhoeven

Quoted 3 lines · Linus Torvalds
Quoted 1 line · Arnaldo Carvalho de Melo

Quoted 8 lines

Quoted 5 lines
Quoted 1 line
Checking why that strlcpy failed...
I don't think glibc does strlcpy. It's not a standard C function, and it's somewhat controversial (although I dislike strncpy more with the crazy zero-padding, ugh).
Indeed, e.g. on Ubuntu it's in libbsd, so you also need to #include <bsd/string.h> and link with -lbsd.

Well, the point of the feature test is not to pick up an external definition of strlcpy(), but to keep working on systems where strcpy() has been patched into _glibc_, where we cannot avoid linking to it. (ARCH Linux is such a distribution for example.)

So it's a quirk in essence - and I think marking our implementation as __weak ought to solve that.

Thanks,

Ingo

Namhyung Kim

maintainer

2013-10-02 06:05 UTC #37 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Hi Ingo,

Quoted 6 lines · Ingo Molnar

This series (with combo patch attached) implements (much) faster perf-tools feature-auto-detection.
I used 3 tricks to implement feature auto-dependencies and to speed up feature detection:

Quoted 6 lines

- standalone Makefile in config/feature-checks/ built in parallel
- split-out standalone .c files in config/feature-checks/*.c
- used GCC's auto-dependency generation feature (-MD) to track the effects of system library addition/removal.

I have a memory that this could lead to a nasty build failure. Please see the commit b6f4f804108b ("tools lib traceevent: Do not generate dependency for system header files").

The problem is that it turned out to depend on some compiler headers which are located under some directory with a version number. If so, when compiler upgraded to a new version, it cannot find the original dependencies so fail to build.

 $ cat config/feature-checks/test-libelf.d
  test-libelf: test-libelf.c /usr/include/libelf.h /usr/include/sys/types.h \
   /usr/include/features.h /usr/include/stdc-predef.h \
   /usr/include/sys/cdefs.h /usr/include/bits/wordsize.h \
   /usr/include/gnu/stubs.h /usr/include/gnu/stubs-64.h \
   /usr/include/bits/types.h /usr/include/bits/typesizes.h \
   /usr/include/time.h \
   /usr/lib/gcc/x86_64-redhat-linux/4.7.2/include/stddef.h \
   /usr/include/endian.h /usr/include/bits/endian.h \
   /usr/include/bits/byteswap.h /usr/include/bits/byteswap-16.h \
   /usr/include/sys/select.h /usr/include/bits/select.h \
   /usr/include/bits/sigset.h /usr/include/bits/time.h \
   /usr/include/sys/sysmacros.h /usr/include/bits/pthreadtypes.h \
   /usr/include/elf.h \
   /usr/lib/gcc/x86_64-redhat-linux/4.7.2/include/stdint.h \
   /usr/include/stdint.h /usr/include/bits/wchar.h

In this case we are using this for feature-checking, so I guess it'd fail to check the feature after upgrade.

Thanks,
Namhyung

Ingo Molnar

author

2013-10-02 06:28 UTC #38 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Quoted 11 lines · Namhyung Kim

Hi Ingo,
Quoted 8 lines · Ingo Molnar
This series (with combo patch attached) implements (much) faster perf-tools feature-auto-detection.
I used 3 tricks to implement feature auto-dependencies and to speed up feature detection:
- standalone Makefile in config/feature-checks/ built in parallel

Quoted 8 lines

Quoted 4 lines
- split-out standalone .c files in config/feature-checks/*.c
- used GCC's auto-dependency generation feature (-MD) to track the effects of system library addition/removal.
I have a memory that this could lead to a nasty build failure. Please see the commit b6f4f804108b ("tools lib traceevent: Do not generate dependency for system header files").

I think that at least the 'make clean' failure was just a buggy Makefile. To quote the build error from the commit:

comet:~/tip/tools/lib/traceevent> make clean
make: *** No rule to make target /usr/lib/gcc/x86_64-redhat-linux/4.7.0/include/stddef.h', needed by .trace-seq

It suggests that the 'clean' target depended on .d dependency files - that's a fundamentally incorrect use of -M/-MD auto-dependencies.

Quoted 22 lines

 $ cat config/feature-checks/test-libelf.d
  test-libelf: test-libelf.c /usr/include/libelf.h /usr/include/sys/types.h \
   /usr/include/features.h /usr/include/stdc-predef.h \
   /usr/include/sys/cdefs.h /usr/include/bits/wordsize.h \
   /usr/include/gnu/stubs.h /usr/include/gnu/stubs-64.h \
   /usr/include/bits/types.h /usr/include/bits/typesizes.h \
   /usr/include/time.h \
   /usr/lib/gcc/x86_64-redhat-linux/4.7.2/include/stddef.h \
   /usr/include/endian.h /usr/include/bits/endian.h \
   /usr/include/bits/byteswap.h /usr/include/bits/byteswap-16.h \
   /usr/include/sys/select.h /usr/include/bits/select.h \
   /usr/include/bits/sigset.h /usr/include/bits/time.h \
   /usr/include/sys/sysmacros.h /usr/include/bits/pthreadtypes.h \
   /usr/include/elf.h \
   /usr/lib/gcc/x86_64-redhat-linux/4.7.2/include/stdint.h \
   /usr/include/stdint.h /usr/include/bits/wchar.h

Quoted 2 lines

In this case we are using this for feature-checking, so I guess it'd fail to check the feature after upgrade.

The dependencies are re-made by GCC if a target fails and is rebuilt - and that should include the new header locations.

I checked out the parent commit (8f7c1d07ade5) which still had full -M, and this is how it utilized dependencies:

# let .d file also depends on the source and header files
define check_deps
                @set -e; $(RM) $@; \
                $(CC) -M $(CFLAGS) $< > $@.$$$$; \
                sed 's,\($*\)\.o[ :]*,\1.o $@ : ,g' < $@.$$$$ > $@; \
                $(RM) $@.$$$$
endef

that's not a very robust method either: .d files should be generated via -MD not via -M and should be included directly into the Makefile, like I did it in my patch:

-include *.d */*.d

and the .d files themselves are never added as dependencies - they are re-made by compilation automatically, not by any explicit Makefile rule. Adding them as dependencies risks circular dependencies, because the only method to rebuild a .d file is to actually meet the dependencies of a .c target.

So if done properly I don't think the build failure cited in that changelog can trigger.

Now, I cannot vouch for -MD blindly, without having seen a lot more testing, so we might still be forced to disable or limit that auto-dep trick, but the reasons cited in b6f4f804108b don't seem to be a GCC bug but a Makefile bug - they just weren't fully understood back then.

Thanks,

Ingo

Jiri Olsa

2013-10-02 09:26 UTC #39 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Quoted 17 lines · Ingo Molnar

This series (with combo patch attached) implements (much) faster perf-tools feature-auto-detection.
I used 3 tricks to implement feature auto-dependencies and to speed up feature detection:
- standalone Makefile in config/feature-checks/ built in parallel
- split-out standalone .c files in config/feature-checks/*.c
- used GCC's auto-dependency generation feature (-MD) to track the effects of system library addition/removal.
Before the changes a fully cached re-build of an already built tree took 2.6 second:

Quoted 7 lines

    $ perf stat --null --sync --repeat 3 -a make Makefile
2.669467209 seconds time elapsed ( +- 0.07% )
After the changes it takes only 0.6 seconds:
0.599161560 seconds time elapsed ( +- 0.25% )

nice!

Quoted 3 lines

So the empty build got 4.4x faster.

SNIP

Quoted 26 lines

ifdef NO_DEMANGLE
CFLAGS += -DNO_DEMANGLE
else

tools/perf/config/Makefile +4 -5

-  ifdef HAVE_CPLUS_DEMANGLE
+  ifdef HAVE_CPLUS_DEMANGLE_SUPPORT
     EXTLIBS += -liberty
-    CFLAGS += -DHAVE_CPLUS_DEMANGLE
+    CFLAGS += -DHAVE_CPLUS_DEMANGLE_SUPPORT
   else
     FLAGS_BFD=$(CFLAGS) $(LDFLAGS) $(EXTLIBS) -DPACKAGE='perf' -lbfd
-    has_bfd := $(call try-cc,$(SOURCE_BFD),$(FLAGS_BFD),libbfd)
-    ifeq ($(has_bfd),y)
+    ifeq ($(feature-libbfd), 1)
       EXTLIBS += -lbfd
     else
       FLAGS_BFD_IBERTY=$(FLAGS_BFD) -liberty

@@ -388,7 +410,7 @@ else
           has_cplus_demangle := $(call try-cc,$(SOURCE_CPLUS_DEMANGLE),$(FLAGS_CPLUS_DEMANGLE),demangle)
           ifeq ($(has_cplus_demangle),y)
             EXTLIBS += -liberty
-            CFLAGS += -DHAVE_CPLUS_DEMANGLE
+            CFLAGS += -DHAVE_CPLUS_DEMANGLE_SUPPORT
           else
             msg := $(warning No bfd.h/libbfd found, install binutils-dev[el]/zlib-static to gain symbol demangling)
             CFLAGS += -DNO_DEMANGLE

tools/perf/config/Makefile else · line 422

Quoted 1 line

tools/perf/config/Makefile +0 -0
@@ -400,30 +422,29 @@ else
 endif

Why not add tests for has_bfd_iberty/has_bfd_iberty_z/has_cplus_demangle?

We could remove the config/feature-tests.mak completely and other cleanup like remove try-cc.

jirka

Ingo Molnar

author

2013-10-02 10:11 UTC #40 lore raw

Re: [PATCH] perf auto-dep: Speed up feature tests by building them in parallel

Quoted 6 lines · Jiri Olsa

Quoted 6 lines

tools/perf/config/Makefile +1 -1

@@ -388,7 +410,7 @@ else
           has_cplus_demangle := $(call try-cc,$(SOURCE_CPLUS_DEMANGLE),$(FLAGS_CPLUS_DEMANGLE),demangle)
           ifeq ($(has_cplus_demangle),y)
             EXTLIBS += -liberty
-            CFLAGS += -DHAVE_CPLUS_DEMANGLE
+            CFLAGS += -DHAVE_CPLUS_DEMANGLE_SUPPORT

Quoted 8 lines

tools/perf/config/Makefile else · line 422
Quoted 5 lines
tools/perf/config/Makefile +0 -0
           else
             msg := $(warning No bfd.h/libbfd found, install binutils-dev[el]/zlib-static to gain symbol demangling)
             CFLAGS += -DNO_DEMANGLE
@@ -400,30 +422,29 @@ else
 endif
Why not add tests for has_bfd_iberty/has_bfd_iberty_z/has_cplus_demangle?

Yeah, did not get to that stage yet: I only converted the feature tests that trigger on my system (which is about 80% of them). Will complete the transition.

Quoted 2 lines

We could remove the config/feature-tests.mak completely and other cleanup like remove try-cc.

Yeah - that will be a nice simplification!

Thanks,

Ingo

Ingo Molnar

author

2013-09-12 20:19 UTC #41 lore raw

In reply to David Ahern

Quoted 7 lines · David Ahern

Quoted 2 lines · Arnaldo Carvalho de Melo
Its something that annoys me as well, but not so much as to make me figure out how to make those be done only if some source file changed.
Jiri and I have both taken stabs at a config-based build rather than probing. Just need to finish it.

Mind outlining the approach you are thinking about?

Firstly, please don't even think about autotools. (Just forget it exists.)

Secondly, the way perf tries to build by auto-detecting the build environment and auto-disabling bits it cannot build just yet is pretty powerful. The core bits will build on just about any system, and our fallbacks are really good.

The result is that perf will build on just about any random system, without the user having to install any dependency. It would be really sad to lose that aspect.

What I think would work best is what I outlined in the previous mail: to cache successful feature test results and only re-do unsuccessful tests:
those are the ones that are expected to turn into successful tests in the future, once the missing dependencies are installed.

Since most tests succeed even on a sparsely installed system, this trick alone will speed up the checks big time.

Furthermore, this method would encourage people to install the dependencies - and perf developers, who do many repeat builds after trivial one-file changes, will typically have all dependencies installed anyway, so for them such a caching feature would result in totally cached feature tests and very fast build times.

Thanks,

Ingo

Arnaldo Carvalho de Melo

2013-09-12 20:38 UTC #42 lore raw

Em Thu, Sep 12, 2013 at 10:18:55PM +0200, Ingo Molnar escreveu:

Quoted 7 lines · David Ahern

Quoted 6 lines · Arnaldo Carvalho de Melo
Quoted 3 lines
Its something that annoys me as well, but not so much as to make me figure out how to make those be done only if some source file changed.

Quoted 6 lines

Quoted 2 lines
Jiri and I have both taken stabs at a config-based build rather than probing. Just need to finish it.
Mind outlining the approach you are thinking about?
Firstly, please don't even think about autotools. (Just forget it exists.)

hehe, no, that wasn't considered.

Quoted 4 lines

Secondly, the way perf tries to build by auto-detecting the build environment and auto-disabling bits it cannot build just yet is pretty powerful. The core bits will build on just about any system, and our fallbacks are really good.

That would remain as:

make -C tools/perf autoconfig

Quoted 3 lines

The result is that perf will build on just about any random system, without the user having to install any dependency. It would be really sad to lose that aspect.

we will not

Quoted 4 lines

What I think would work best is what I outlined in the previous mail: to cache successful feature test results and only re-do unsuccessful tests:
those are the ones that are expected to turn into successful tests in the future, once the missing dependencies are installed.

that is an optimization to 'make autoconfig', i.e. what we have now, improved.

Quoted 8 lines

Since most tests succeed even on a sparsely installed system, this trick alone will speed up the checks big time.
Furthermore, this method would encourage people to install the dependencies - and perf developers, who do many repeat builds after trivial one-file changes, will typically have all dependencies installed anyway, so for them such a caching feature would result in totally cached feature tests and very fast build times.

What he mentioned is the multiple attempts at doing:

make -C tools/perf menuconfig

and use kbuild to allow one to select what he/she wants to build, i.e. using the kernel config system in perf.

In that case the feature checks would be triggered only for the features selected, not for all perf currently selectable features.

- Arnaldo

Ingo Molnar

author

2013-09-12 20:46 UTC #43 lore raw

Quoted 8 lines · Arnaldo Carvalho de Melo

Em Thu, Sep 12, 2013 at 10:18:55PM +0200, Ingo Molnar escreveu:
Quoted 7 lines · David Ahern
Quoted 6 lines · Arnaldo Carvalho de Melo
Quoted 3 lines
Its something that annoys me as well, but not so much as to make me figure out how to make those be done only if some source file changed.

Quoted 8 lines

Quoted 6 lines
Quoted 2 lines
Jiri and I have both taken stabs at a config-based build rather than probing. Just need to finish it.
Mind outlining the approach you are thinking about?
Firstly, please don't even think about autotools. (Just forget it exists.)
hehe, no, that wasn't considered.

/phew! :-)

Quoted 7 lines

Quoted 4 lines
Secondly, the way perf tries to build by auto-detecting the build environment and auto-disabling bits it cannot build just yet is pretty powerful. The core bits will build on just about any system, and our fallbacks are really good.
That would remain as:

Quoted 7 lines

make -C tools/perf autoconfig
Quoted 3 lines
The result is that perf will build on just about any random system, without the user having to install any dependency. It would be really sad to lose that aspect.
we will not

But it would be nice to keep building as simple as 'make'.

So I don't think splitting out the feature tests into a separate pass, to be done manually by the user, is a step forward.

Speeding them up by caching their results, while cleaning up the presentation of the testcases, on the other hand, would be a (big!) step forward.

Thanks,

Ingo

David Ahern

2013-09-12 21:09 UTC #44 lore raw

Quoted 7 lines · Ingo Molnar

Quoted 5 lines
Quoted 3 lines
Mind outlining the approach you are thinking about?
Firstly, please don't even think about autotools. (Just forget it exists.)
hehe, no, that wasn't considered.
/phew! :-)

kconf approach of course:
https://lkml.org/lkml/2013/4/1/600
(minus the manual steps in that RFC).

David

Ingo Molnar

author

2013-09-12 21:18 UTC #45 lore raw

Quoted 5 lines · David Ahern

Quoted 4 lines · Ingo Molnar
Quoted 4 lines
Quoted 3 lines
Mind outlining the approach you are thinking about?
Firstly, please don't even think about autotools. (Just forget it exists.)

Quoted 7 lines

Quoted 3 lines
Quoted 1 line
hehe, no, that wasn't considered.
/phew! :-)
kconf approach of course:
https://lkml.org/lkml/2013/4/1/600
(minus the manual steps in that RFC).

I'm not sure what the end stage is where you'd like to arrive, but I don't think that forcing a separate configuration pass is an improvement :-/

By default a simple 'make' should build perf to the maximum extent possible, with no other input required from the user - with warnings displayed as package install suggestions.

This:

Enable newt-based TUI (NEWT) [N/y] (NEW) y
Enable GTK-based UI (GTK2) [N/y] (NEW) n
Enable support for Bionic (e.g., Android platform) (BIONIC) [N/y] (NEW) Development support for libc is available - glibc or bionic (LIBC) [N/y] (NEW) y
Enable support for libelf (LIBELF) [N/y] (NEW) y
Enable support for libunwind (LIBUNWIND) [N/y] (NEW) y Enable support for dwarf (DWARF) [N/y] (NEW) y
Enable support for demangle (DEMANGLE) [N/y] (NEW) y
Enable support for perl scripting engine (LIBPERL) [N/y] (NEW) y Enable support for python scripting engine (LIBPYTHON) [N/y] (NEW) y Enable support for libaudit (LIBAUDIT) [N/y/?] (NEW) y Enable support for libnuma (LIBNUMA) [N/y/?] (NEW) y
Enable support for stack backtrace debugging (BACKTRACE) [N/y] (NEW) y

would be useful only as long as each listed option is actually _buildable_. I.e. it should not be possible for the user to configure perf in a way that makes the build fail.

I.e. this should go on top of a feature detection logic, allowing further customization, for features that the user might want to turn off.

Thanks,

Ingo

David Ahern

2013-09-12 22:10 UTC #46 lore raw

Quoted 6 lines · Ingo Molnar

Quoted 3 lines
kconf approach of course:
   https://lkml.org/lkml/2013/4/1/600
(minus the manual steps in that RFC).
I'm not sure what the end stage is where you'd like to arrive, but I don't think that forcing a separate configuration pass is an improvement :-/

once I have a working config -- say one for a target (minimal build) and one for analysis (more but not all features -- eg., no gtk) I never see auto-probing again. At best some new feature (config) comes along and the config has to be updated again, but kconf only inquires about new features.

Quoted 4 lines

By default a simple 'make' should build perf to the maximum extent possible, with no other input required from the user - with warnings displayed as package install suggestions.

By default there is no config. Autoprobing generates a first one or a user can specify a defconfig.

David

Ingo Molnar

author

2013-09-13 05:09 UTC #47 lore raw

Quoted 6 lines · David Ahern

Quoted 3 lines
By default a simple 'make' should build perf to the maximum extent possible, with no other input required from the user - with warnings displayed as package install suggestions.
By default there is no config. Autoprobing generates a first one or a user can specify a defconfig.

This could work if there's not two but three states for individual features:

- autoprobe
- on
- off

and if autoprobe, if a system feature has been probed successfully, automatically turned 'autoprobe' entries into 'on'.

That would give us the best of all worlds - autodetection, configurability and caching:

- initial user types 'make' and gets a .config that has almost all entries 'on', a few 'autoprobe'.

- once the user installs a dependency, the corresponding .config entry turns into 'on'.

- the regular user or developers would have libraries that turn all entries in the .config to 'on'.

- if a user is genuinely uninterested in a feature, he can mark it 'off', which would then stay off permanently. This could also be used by embedded/specialized builds.

- other specialized users, like distro builds, could use a .config with all entries 'on' and could enforce the presence of all dependencies for a successful build. [We could add 'make allyesconfig' to help that.]

Thanks,

Ingo

Jean Pihet

2013-09-13 09:32 UTC #48 lore raw

Hi,

Quoted 32 lines · Ingo Molnar

Quoted 6 lines · David Ahern
Quoted 3 lines
By default a simple 'make' should build perf to the maximum extent possible, with no other input required from the user - with warnings displayed as package install suggestions.
By default there is no config. Autoprobing generates a first one or a user can specify a defconfig.
This could work if there's not two but three states for individual features:
- autoprobe
- on
- off
and if autoprobe, if a system feature has been probed successfully, automatically turned 'autoprobe' entries into 'on'.
That would give us the best of all worlds - autodetection, configurability and caching:
- initial user types 'make' and gets a .config that has almost all entries 'on', a few 'autoprobe'.
- once the user installs a dependency, the corresponding .config entry turns into 'on'.
- the regular user or developers would have libraries that turn all entries in the .config to 'on'.

Quoted 7 lines

- if a user is genuinely uninterested in a feature, he can mark it 'off', which would then stay off permanently. This could also be used by embedded/specialized builds.
- other specialized users, like distro builds, could use a .config with all entries 'on' and could enforce the presence of all dependencies for a successful build. [We could add 'make allyesconfig' to help that.]

Is there a way to detect the presence of a dependency and _also_ check its version? Some new features are depending on a recent version of a library, e.g. dwarf unwinding depends on libunwind >= 1.1 (cf.
http://www.spinics.net/lists/kernel/msg1598951.html).

Thanks,
Jean

Quoted 9 lines

Thanks,
Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Ingo Molnar

author

2013-09-13 09:46 UTC #49 lore raw

Quoted 39 lines · Jean Pihet

Hi,
Quoted 36 lines · Ingo Molnar
Quoted 6 lines · David Ahern
Quoted 3 lines
By default a simple 'make' should build perf to the maximum extent possible, with no other input required from the user - with warnings displayed as package install suggestions.
By default there is no config. Autoprobing generates a first one or a user can specify a defconfig.
This could work if there's not two but three states for individual features:
- autoprobe
- on
- off
and if autoprobe, if a system feature has been probed successfully, automatically turned 'autoprobe' entries into 'on'.
That would give us the best of all worlds - autodetection, configurability and caching:
- initial user types 'make' and gets a .config that has almost all entries 'on', a few 'autoprobe'.
- once the user installs a dependency, the corresponding .config entry turns into 'on'.
- the regular user or developers would have libraries that turn all entries in the .config to 'on'.
- if a user is genuinely uninterested in a feature, he can mark it 'off', which would then stay off permanently. This could also be used by embedded/specialized builds.

Quoted 8 lines

Quoted 3 lines
- other specialized users, like distro builds, could use a .config with all entries 'on' and could enforce the presence of all dependencies for a successful build. [We could add 'make allyesconfig' to help that.]
Is there a way to detect the presence of a dependency and _also_ check its version? Some new features are depending on a recent version of a library, e.g. dwarf unwinding depends on libunwind >= 1.1 (cf.
http://www.spinics.net/lists/kernel/msg1598951.html).

Yeah, see the testcases in tools/perf/config/feature-tests.mak, they typically include the latest library API usages, which will fail on older versions.

That kind of 'does it actually work?' test is a lot more robust than explicit version checks, and combined with caching it should be fast and parallelizable as well. (One of the problems of the current simple implementation of the feature tests is that they are 20 serial tests with no parallelization.)

Thanks,

Ingo

Jean Pihet

2013-09-13 17:15 UTC #50 lore raw

Quoted 46 lines · Ingo Molnar

Quoted 43 lines · Jean Pihet
Hi,
Quoted 39 lines · Ingo Molnar
Quoted 6 lines · David Ahern
Quoted 3 lines
By default a simple 'make' should build perf to the maximum extent possible, with no other input required from the user - with warnings displayed as package install suggestions.
By default there is no config. Autoprobing generates a first one or a user can specify a defconfig.
This could work if there's not two but three states for individual features:
- autoprobe
- on
- off
and if autoprobe, if a system feature has been probed successfully, automatically turned 'autoprobe' entries into 'on'.
That would give us the best of all worlds - autodetection, configurability and caching:
- initial user types 'make' and gets a .config that has almost all entries 'on', a few 'autoprobe'.
- once the user installs a dependency, the corresponding .config entry turns into 'on'.
- the regular user or developers would have libraries that turn all entries in the .config to 'on'.
- if a user is genuinely uninterested in a feature, he can mark it 'off', which would then stay off permanently. This could also be used by embedded/specialized builds.
- other specialized users, like distro builds, could use a .config with all entries 'on' and could enforce the presence of all dependencies for a successful build. [We could add 'make allyesconfig' to help that.]

Quoted 8 lines

Quoted 4 lines
Is there a way to detect the presence of a dependency and _also_ check its version? Some new features are depending on a recent version of a library, e.g. dwarf unwinding depends on libunwind >= 1.1 (cf.
http://www.spinics.net/lists/kernel/msg1598951.html).
Yeah, see the testcases in tools/perf/config/feature-tests.mak, they typically include the latest library API usages, which will fail on older versions.

Ok!
I just sent a patch to feature-tests.mak for the newly added dwarf unwinding feature ('perf tools: Check libunwind for availability of dwarf parsing feature').

Thanks,
Jean

Quoted 10 lines

That kind of 'does it actually work?' test is a lot more robust than explicit version checks, and combined with caching it should be fast and parallelizable as well. (One of the problems of the current simple implementation of the feature tests is that they are 20 serial tests with no parallelization.)
Thanks,
Ingo

Linus Torvalds

2013-09-12 18:51 UTC #51 lore raw

Quoted 2 lines · Linus Torvalds

But at least the "make install" problem is repeatable, though.

And now this new problem is repeatable too:

# On a fully built kernel tree
perf record -g -e cycles:pp make -j

results in

[ perf record: Woken up 27 times to write data ]
[ perf record: Captured and wrote 7.980 MB perf.data (~348659 samples) ] 0x1b4e0 [0]: failed to process type: -1970637019

where that number changes randomly, ie I get

0x28dc58 [0]: failed to process type: 99257493
0x4100 [0]: failed to process type: -1144359783
0x29050 [0]: failed to process type: -972156963
..

looks like perhaps some uninitialized variable somewhere?

Recording performance profiles of other (simpler?) loads still seems to work. So it's something about that "make -j" that makes it crap out (note that the tree is fully built, so not a lot actually gets *done*, and the thing only takes a few seconds)

Linus

Ingo Molnar

author

2013-09-12 20:34 UTC #52 lore raw

Quoted 23 lines · Linus Torvalds

Quoted 2 lines · Linus Torvalds
But at least the "make install" problem is repeatable, though.
And now this new problem is repeatable too:
# On a fully built kernel tree
perf record -g -e cycles:pp make -j
results in
[ perf record: Woken up 27 times to write data ]
[ perf record: Captured and wrote 7.980 MB perf.data (~348659 samples) ] 0x1b4e0 [0]: failed to process type: -1970637019
where that number changes randomly, ie I get
0x28dc58 [0]: failed to process type: 99257493
0x4100 [0]: failed to process type: -1144359783
0x29050 [0]: failed to process type: -972156963
..

Quoted 6 lines

looks like perhaps some uninitialized variable somewhere?
Recording performance profiles of other (simpler?) loads still seems to work. So it's something about that "make -j" that makes it crap out (note that the tree is fully built, so not a lot actually gets *done*, and the thing only takes a few seconds)

Hm, just to make sure, are you running a (very fresh) kernel that has this fix included:

d008d5258e9c perf: Fix up MMAP2 buffer space reservation

Thanks,

Ingo

Linus Torvalds

2013-09-12 20:38 UTC #53 lore raw

Quoted 5 lines · Ingo Molnar

Hm, just to make sure, are you running a (very fresh) kernel that has this fix included:
d008d5258e9c perf: Fix up MMAP2 buffer space reservation

I've rebooted about five times since that report, so I'm not sure..

[ Short time passes ]

Hmm. That "reproducible" error is no longer reproducible for me on current git (68f0d9d92e54). So if you fixed something recently, it's possible I was running a pre-perf-pull kernel and an after-perf-pull "perf" binary.

Linus

Ingo Molnar

author

2013-09-12 20:49 UTC #54 lore raw

Quoted 7 lines · Linus Torvalds

Quoted 5 lines · Ingo Molnar
Hm, just to make sure, are you running a (very fresh) kernel that has this fix included:
d008d5258e9c perf: Fix up MMAP2 buffer space reservation

Quoted 8 lines

I've rebooted about five times since that report, so I'm not sure..
[ Short time passes ]
Hmm. That "reproducible" error is no longer reproducible for me on current git (68f0d9d92e54). So if you fixed something recently, it's possible I was running a pre-perf-pull kernel and an after-perf-pull "perf" binary.

Yeah, so I think prior merging d008d5258e9c you'd see those artifacts with new, MMAP2-aware perf tooling. Both older kernels would work fine with any version of the tooling, and old tooling would work fine. Only new tooling used on a kernel within the breakage window would be exposed to this.

I think the window of breakage is relatively small (you managed to hit it), but should we add a quirk for this?

Thanks,

Ingo

Linus Torvalds

2013-09-12 20:52 UTC #55 lore raw

Quoted 4 lines · Ingo Molnar

I think the window of breakage is relatively small (you managed to hit it), but should we add a quirk for this?

Was the breakage entirely inside the merge window? ie no released kernels?

If so, just ignore it.

Linus

Ingo Molnar

author

2013-09-12 21:01 UTC #56 lore raw

Quoted 8 lines · Linus Torvalds

Quoted 4 lines · Ingo Molnar
I think the window of breakage is relatively small (you managed to hit it), but should we add a quirk for this?
Was the breakage entirely inside the merge window? ie no released kernels?

Absolutely! Starts at commit 0d99b7087324, stops at commit d008d5258e9c, all inside the merge window.

Quoted 1 line

If so, just ignore it.

Ok, great!

Thanks,

Ingo

Ingo Molnar

author

2013-09-12 20:11 UTC #57 lore raw

Quoted 10 lines · Linus Torvalds

Quoted 3 lines · Linus Torvalds
When I compiled "perf" at the same time as doing a big kernel compile, the kernel compile failed
Oops. That may actually have been me being a bit *too* eager with a "make allmodconfig" build. I can't reproduce it, and I'm starting to suspect that I instead had two kernel compiles going, not one kernel compile and a tools/perf/ compile.

Btw., building perf in parallel to an ongoing kernel compile is something I do almost daily, and I never saw problems due to that. A perf build is not supposed to cause any side effect on the kernel build and vice versa.

So in general it should just work.

Quoted 3 lines

"Yo Dawg, I heard you like kernel compiles, so I put a kernel compile
in your kernel compile so that you can compile the kernel while you compile the kernel".

Heh, accidental dual kernel builds within the same kernel tree is something I have a _lot_ of involuntary experience with, and I usually recognize the error message patterns straight away ;-)

As a kernel hacker one learns useful skills all the time!

Thanks,

Ingo

Ingo Molnar

author

2013-10-02 07:31 UTC #58 lore raw

PATCH 1/1 tools/perf: Fix double/triple-build of the feature detection logic during 'make install' et al