Skip to content

RFC tools/perf: Speed up the build system

Ingo Molnar lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest50 patches 29 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/50tools/perf: Standardize feature support define names to: HAVE_{FEATURE}_SUPPORTtools/perf/arch/x86/include/perf_regs.h tools/perf/arch/x86/util/unwind.c tools/perf/bench/mem-memcpy-arch.h +27 more+86 -860 replies
2/50tools/perf/build: Add feature check core codetools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-hello.c+38 -00 replies
3/50tools/perf/build: Add 'autodep' functionality, generate feature test dependencies automaticallytools/perf/config/feature-checks/Makefile+6 -20 replies
4/50tools/perf/build: Split out feature check: 'libnuma'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-libnuma.c+18 -40 replies
5/50tools/perf/build: Split out feature check: 'stackprotector-all'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-stackprotector-all.c+12 -10 replies
6/50tools/perf/build: Split out feature check: 'stackprotector'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-stackprotector.c+12 -10 replies
7/50tools/perf/build: Split out feature check: 'volatile-register-var'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-volatile-register-var.c+12 -10 replies
8/50tools/perf/build: Split out feature check: 'fortify-source'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-fortify-source.c+12 -10 replies
9/50tools/perf/build: Split out feature check: 'bionic'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-bionic.c+16 -50 replies
10/50tools/perf/build: Clean up the libelf logic in config/Makefiletools/perf/config/Makefile+29 -290 replies
11/50tools/perf/build: Split out feature check: 'libelf'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-libelf.c+13 -20 replies
12/50tools/perf/build: Split out feature check: 'glibc'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-glibc.c+14 -20 replies
13/50tools/perf/build: Split out feature check: 'dwarf'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-dwarf.c+15 -20 replies
14/50tools/perf/build: Clean up the mmap logic in config/Makefiletools/perf/config/Makefile+26 -250 replies
15/50tools/perf/build: Split out feature check: 'libelf-mmap'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-libelf-mmap.c+14 -30 replies
16/50tools/perf/build: Split out feature check: 'libelf-getphdrnum'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-libelf-getphdrnum.c+14 -20 replies
17/50tools/perf/build: Clean up the libunwind logic in config/Makefiletools/perf/config/Makefile+12 -120 replies
18/50tools/perf/build: Split out feature check: 'libunwind'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-libunwind.c+27 -100 replies
19/50tools/perf/build: Split out feature check: 'libaudit'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-libaudit.c+14 -32 replies
20/50tools/perf/build: Split out feature check: 'libslang'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-libslang.c+13 -32 replies
21/50tools/perf/build: Split out feature check: 'gtk2'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-gtk2.c+15 -00 replies
22/50tools/perf/build: Split out feature check: 'gtk2-infobar'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-gtk2-infobar.c+16 -10 replies
23/50tools/perf/build: Split out feature check: 'libperl'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-libperl.c+24 -10 replies
24/50tools/perf/build: Split out feature check: 'libpython'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-libpython.c+27 -10 replies
25/50tools/perf/build: Split out feature check: 'libpython-version'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-libpython-version.c+16 -10 replies
26/50tools/perf/build: Split out feature check: 'libbfd'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-libbfd.c +1 more+25 -33 replies
27/50tools/perf/build: Split out feature check: 'on-exit'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-on-exit.c+12 -10 replies
28/50tools/perf/build: Split out feature check: 'backtrace'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-backtrace.c+16 -10 replies
29/50tools/perf: Clean up util/include/linux/compiler.htools/perf/util/include/linux/compiler.h+10 -50 replies
30/50tools/perf: Turn strlcpy() into a __weak functiontools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-strlcpy.c +3 more+12 -220 replies
31/50tools/perf/build: Speed up auto-detection of features by adding a 'test-all' targettools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-all.c+240 -73 replies
32/50tools/perf/build: Speed up git-version test on re-maketools/perf/Makefile+13 -20 replies
33/50tools/perf/build: Speed up the final linktools/perf/Makefile+7 -30 replies
34/50tools/perf: Fix double/triple-build of the feature detection logic during 'make install' et altools/perf/Documentation/Makefile+4 -130 replies
35/50tools/perf/build: Invoke feature-checks 'clean' target from the main Makefiletools/perf/Makefile+9 -20 replies
36/50tools/perf/build: Speed up auto-detectiontools/perf/config/Makefile tools/perf/config/feature-checks/Makefile+44 -352 replies
37/50tools/perf/build: Improve printout-of auto-detected featurestools/perf/config/Makefile+33 -95 replies
38/50tools/perf/build: Automatically build in parallel, based on number of CPUs in the systemtools/perf/Makefile.parallel+26 -00 replies
39/50tools/perf/build: Flip Makefile.parallel and Makefile.perftools/perf/Makefile tools/perf/Makefile.parallel tools/perf/Makefile.perf+855 -8550 replies
40/50tools/perf/build: Standardize the various messages output by parallel maketools/perf/Makefile tools/perf/config/Makefile+22 -50 replies
41/50tools/perf/build: Split out feature checks: 'liberty', 'liberty-z', 'cplus-demangle'tools/perf/config/Makefile tools/perf/config/feature-checks/Makefile tools/perf/config/feature-checks/test-cplus-demangle.c+28 -100 replies
42/50tools/perf/build: Remove unused config/feature-tests.maktools/perf/config/Makefile tools/perf/config/feature-tests.mak tools/perf/config/utilities.mak+1 -2600 replies
43/50tools/perf/build: Clean up various testcasestools/perf/config/feature-checks/test-backtrace.c tools/perf/config/feature-checks/test-cplus-demangle.c tools/perf/config/feature-checks/test-gtk2-infobar.c +5 more+47 -90 replies
44/50tools/perf/build: Collapse the test-all.c testcasetools/perf/config/feature-checks/test-all.c tools/perf/config/feature-checks/test-backtrace.c tools/perf/config/feature-checks/test-libunwind.c+102 -1932 replies
45/50tools/perf/build: Pass through all targets to Makefile.perftools/perf/Makefile+7 -00 replies
46/50tools/perf/build: Make sure autodep feature binaries honor the O= settingtools/perf/config/Makefile+8 -30 replies
47/50tools/perf/build: Exclude MAKEFLAGS from nested invocationtools/perf/Makefile+13 -30 replies
48/50tools/perf/build: Fix non-canonical directory names in O=tools/perf/Makefile+8 -10 replies
49/50tools/perf/build: Fix O=/some/dir perf.o type of targetstools/perf/Documentation/Makefile tools/perf/Makefile.perf+17 -12 replies
50/50tools/perf/build: Harmonize the style of the feature testcasestools/perf/config/feature-checks/test-backtrace.c tools/perf/config/feature-checks/test-cplus-demangle.c tools/perf/config/feature-checks/test-dwarf.c +9 more+11 -90 replies

Reviews trailers per patch

None of the 50 patches has a Reviewed-by or Acked-by.

PatchDavid Ahern
SeriesT-b
1/50
2/50
3/50
4/50
5/50
6/50
7/50
8/50
9/50
10/50
11/50
12/50
13/50
14/50
15/50
16/50
17/50
18/50
19/50
20/50
21/50
22/50
23/50
24/50
25/50
26/50
27/50
28/50
29/50
30/50
31/50
32/50
33/50
34/50
35/50
36/50
37/50
38/50
39/50
40/50
41/50
42/50
43/50
44/50
45/50
46/50
47/50
48/50
49/50
50/50

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [RFC PATCH 00/50] tools/perf: Speed up the build system

This series improves various aspects of the tools/perf build system.

The biggest change is the rewrite of the feature detection framework and a conversion of all feature testcases to the new method. The new feature detection machinery relies on three new methods:

- split-out, independent testcases that allow "make -j" parallel builds [ the old build system built the 20+ testcases serially, greatly slowing down the build. ]

- dependency auto-generation via gcc's -MD feature, which allows the 'caching' of testcases.

- a 'test-all' testcase that contains all the other testcases and allows a fast check for the typical 'all dependencies are installed' case.

The series also includes a number of fixes and convenience features:

- auto-detect the number of CPUs on the system and add the -j<cpus> option automatically

- Fix 'make O=/some/dir/ perf.o' type of rules to work in the $(OUTPUT) directory

- Improve documentation builds/install to not require feature-detection

- speed up linking, eliminate spurious rebuilds

Altogether build speed on my testsystem improved by a factor of 2.1x:

[BEFORE]

    $ cd tools/perf
    $ time make -j12 install
    ...
      real    0m27.522s
      user    1m59.478s
      sys     0m9.714s

[AFTER]

    $ cd tools/perf
    $ time make install
    ...
      real    0m12.658s
      user    1m39.507s
      sys     0m7.706s

Re-build speed, which is important to perf developers, increased 5.0x:

[BEFORE]

    $ touch perf.c
    $ time make
    ...
      real    0m9.368s
      user    0m8.166s
      sys     0m1.015s

[AFTER]

    $ touch perf.c
    $ time make
    BUILD: Doing 'make -j12' parallel build
    CC perf.o
    LINK perf
      real    0m1.857s
      user    0m1.522s
      sys     0m0.295s

Empty build speed of 'make' in an already built tree - often done by developers to see whether they are testing the latest binary, increased more than 50-fold:

[BEFORE]

    $ time make
    ...
      real    0m9.105s
      user    0m7.975s
      sys     0m0.949s

[AFTER]

    $ time make
    $ time make
    BUILD: Doing 'make -j12' parallel build
      real    0m0.172s
      user    0m0.119s
      sys     0m0.029s

Thanks,

Ingo

=============>
Ingo Molnar (50):
tools/perf: Standardize feature support define names to: HAVE_{FEATURE}_SUPPORT tools/perf/build: Add feature check core code
tools/perf/build: Add 'autodep' functionality, generate feature test dependencies automatically tools/perf/build: Split out feature check: 'libnuma'
tools/perf/build: Split out feature check: 'stackprotector-all' tools/perf/build: Split out feature check: 'stackprotector'
tools/perf/build: Split out feature check: 'volatile-register-var' tools/perf/build: Split out feature check: 'fortify-source'
tools/perf/build: Split out feature check: 'bionic'
tools/perf/build: Clean up the libelf logic in config/Makefile tools/perf/build: Split out feature check: 'libelf'
tools/perf/build: Split out feature check: 'glibc'
tools/perf/build: Split out feature check: 'dwarf'
tools/perf/build: Clean up the mmap logic in config/Makefile
tools/perf/build: Split out feature check: 'libelf-mmap'
tools/perf/build: Split out feature check: 'libelf-getphdrnum' tools/perf/build: Clean up the libunwind logic in config/Makefile tools/perf/build: Split out feature check: 'libunwind'
tools/perf/build: Split out feature check: 'libaudit'
tools/perf/build: Split out feature check: 'libslang'
tools/perf/build: Split out feature check: 'gtk2'
tools/perf/build: Split out feature check: 'gtk2-infobar'
tools/perf/build: Split out feature check: 'libperl'
tools/perf/build: Split out feature check: 'libpython'
tools/perf/build: Split out feature check: 'libpython-version' tools/perf/build: Split out feature check: 'libbfd'
tools/perf/build: Split out feature check: 'on-exit'
tools/perf/build: Split out feature check: 'backtrace'
tools/perf: Clean up util/include/linux/compiler.h
tools/perf: Turn strlcpy() into a __weak function
tools/perf/build: Speed up auto-detection of features by adding a 'test-all' target tools/perf/build: Speed up git-version test on re-make
tools/perf/build: Speed up the final link
tools/perf: Fix double/triple-build of the feature detection logic during 'make install' et al tools/perf/build: Invoke feature-checks 'clean' target from the main Makefile tools/perf/build: Speed up auto-detection
tools/perf/build: Improve printout-of auto-detected features
tools/perf/build: Automatically build in parallel, based on number of CPUs in the system tools/perf/build: Flip Makefile.parallel and Makefile.perf
tools/perf/build: Standardize the various messages output by parallel make tools/perf/build: Split out feature checks: 'liberty', 'liberty-z', 'cplus-demangle' tools/perf/build: Remove unused config/feature-tests.mak
tools/perf/build: Clean up various testcases
tools/perf/build: Collapse the test-all.c testcase
tools/perf/build: Pass through all targets to Makefile.perf
tools/perf/build: Make sure autodep feature binaries honor the O= setting tools/perf/build: Exclude MAKEFLAGS from nested invocation
tools/perf/build: Fix non-canonical directory names in O=
tools/perf/build: Fix O=/some/dir perf.o type of targets
tools/perf/build: Harmonize the style of the feature testcases

 tools/perf/Documentation/Makefile                             |  19 +-
 tools/perf/Makefile                                           | 840 ++---------------------------
 tools/perf/Makefile.perf                                      | 855 ++++++++++++++++++++++++++++++
 tools/perf/arch/x86/include/perf_regs.h                       |   6 +-
 tools/perf/arch/x86/util/unwind.c                             |   4 +-
 tools/perf/bench/mem-memcpy-arch.h                            |   2 +-
 tools/perf/bench/mem-memcpy.c                                 |   2 +-
 tools/perf/bench/mem-memset-arch.h                            |   2 +-
 tools/perf/bench/mem-memset.c                                 |   2 +-
 tools/perf/builtin-bench.c                                    |   4 +-
 tools/perf/builtin-inject.c                                   |   2 +-
 tools/perf/builtin-probe.c                                    |  14 +-
 tools/perf/builtin-record.c                                   |  12 +-
 tools/perf/config/Makefile                                    | 326 +++++++-----
 tools/perf/config/feature-checks/Makefile                     | 148 ++++++
 tools/perf/config/feature-checks/test-all.c                   | 106 ++++
 tools/perf/config/feature-checks/test-backtrace.c             |  13 +
 tools/perf/config/feature-checks/test-bionic.c                |   6 +
 tools/perf/config/feature-checks/test-cplus-demangle.c        |  14 +
 tools/perf/config/feature-checks/test-dwarf.c                 |  10 +
 tools/perf/config/feature-checks/test-fortify-source.c        |   6 +
 tools/perf/config/feature-checks/test-glibc.c                 |   8 +
 tools/perf/config/feature-checks/test-gtk2-infobar.c          |  11 +
 tools/perf/config/feature-checks/test-gtk2.c                  |  10 +
 tools/perf/config/feature-checks/test-hello.c                 |   6 +
 tools/perf/config/feature-checks/test-libaudit.c              |  10 +
 tools/perf/config/feature-checks/test-libbfd.c                |  15 +
 tools/perf/config/feature-checks/test-libelf-getphdrnum.c     |   8 +
 tools/perf/config/feature-checks/test-libelf-mmap.c           |   8 +
 tools/perf/config/feature-checks/test-libelf.c                |   8 +
 tools/perf/config/feature-checks/test-libnuma.c               |   9 +
 tools/perf/config/feature-checks/test-libperl.c               |   9 +
 tools/perf/config/feature-checks/test-libpython-version.c     |  10 +
 tools/perf/config/feature-checks/test-libpython.c             |   8 +
 tools/perf/config/feature-checks/test-libslang.c              |   6 +
 tools/perf/config/feature-checks/test-libunwind.c             |  27 +
 tools/perf/config/feature-checks/test-on-exit.c               |  15 +
 tools/perf/config/feature-checks/test-stackprotector-all.c    |   6 +
 tools/perf/config/feature-checks/test-stackprotector.c        |   6 +
 tools/perf/config/feature-checks/test-volatile-register-var.c |   6 +
 tools/perf/config/feature-tests.mak                           | 246 ---------
 tools/perf/config/utilities.mak                               |  14 -
 tools/perf/perf.c                                             |   4 +-
 tools/perf/ui/gtk/browser.c                                   |   2 +-
 tools/perf/ui/gtk/gtk.h                                       |   4 +-
 tools/perf/ui/gtk/util.c                                      |   4 +-
 tools/perf/ui/ui.h                                            |   4 +-
 tools/perf/util/annotate.h                                    |   4 +-
 tools/perf/util/cache.h                                       |   3 +-
 tools/perf/util/generate-cmdlist.sh                           |   4 +-
 tools/perf/util/hist.h                                        |   4 +-
 tools/perf/util/include/dwarf-regs.h                          |   2 +-
 tools/perf/util/include/linux/compiler.h                      |  19 +-
 tools/perf/util/map.c                                         |   2 +-
 tools/perf/util/path.c                                        |  10 +-
 tools/perf/util/perf_regs.h                                   |   4 +-
 tools/perf/util/probe-event.c                                 |   4 +-
 tools/perf/util/probe-finder.h                                |   4 +-
 tools/perf/util/symbol-elf.c                                  |   2 +-
 tools/perf/util/symbol.h                                      |   8 +-
 tools/perf/util/unwind.h                                      |   4 +-
 tools/perf/util/util.c                                        |   4 +-
 62 files changed, 1678 insertions(+), 1257 deletions(-)
 create mode 100644 tools/perf/Makefile.perf
 create mode 100644 tools/perf/config/feature-checks/Makefile
 create mode 100644 tools/perf/config/feature-checks/test-all.c
 create mode 100644 tools/perf/config/feature-checks/test-backtrace.c
 create mode 100644 tools/perf/config/feature-checks/test-bionic.c
 create mode 100644 tools/perf/config/feature-checks/test-cplus-demangle.c
 create mode 100644 tools/perf/config/feature-checks/test-dwarf.c
 create mode 100644 tools/perf/config/feature-checks/test-fortify-source.c
 create mode 100644 tools/perf/config/feature-checks/test-glibc.c
 create mode 100644 tools/perf/config/feature-checks/test-gtk2-infobar.c
 create mode 100644 tools/perf/config/feature-checks/test-gtk2.c
 create mode 100644 tools/perf/config/feature-checks/test-hello.c
 create mode 100644 tools/perf/config/feature-checks/test-libaudit.c
 create mode 100644 tools/perf/config/feature-checks/test-libbfd.c
 create mode 100644 tools/perf/config/feature-checks/test-libelf-getphdrnum.c
 create mode 100644 tools/perf/config/feature-checks/test-libelf-mmap.c
 create mode 100644 tools/perf/config/feature-checks/test-libelf.c
 create mode 100644 tools/perf/config/feature-checks/test-libnuma.c
 create mode 100644 tools/perf/config/feature-checks/test-libperl.c
 create mode 100644 tools/perf/config/feature-checks/test-libpython-version.c
 create mode 100644 tools/perf/config/feature-checks/test-libpython.c
 create mode 100644 tools/perf/config/feature-checks/test-libslang.c
 create mode 100644 tools/perf/config/feature-checks/test-libunwind.c
 create mode 100644 tools/perf/config/feature-checks/test-on-exit.c
 create mode 100644 tools/perf/config/feature-checks/test-stackprotector-all.c
 create mode 100644 tools/perf/config/feature-checks/test-stackprotector.c
 create mode 100644 tools/perf/config/feature-checks/test-volatile-register-var.c
 delete mode 100644 tools/perf/config/feature-tests.mak
Signature · 1 line
1.7.11.7