tools: Add clean targets for tools directory

hi,
sending some tools build fixes,

patches 5 and 6 are RFC and I'd like to have it
tested/confirmed by Wang Nan it suits his needs.

All of them available on top of my perf/fixes branch at:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git

thanks,
jirka

---
Jiri Olsa (6):
tools: Add clean targets for tools directory tools bpf: Add *.cmd files clean up
tools lockdep: Add *.cmd files clean up perf tools: Add missing sources in perf's MANIFEST perf build: Add feature-dump target
perf build: Introduce FEATURES_DUMP make variable

 tools/Makefile             | 11 ++++++++++-
 tools/lib/bpf/Makefile     |  2 +-
 tools/lib/lockdep/Makefile |  2 +-
 tools/perf/MANIFEST        |  2 ++
 tools/perf/Makefile.perf   | 25 ++++++++++++++++++++++++-
 tools/perf/config/Makefile |  4 ++++
 6 files changed, 42 insertions(+), 4 deletions(-)

Adding missing clean targets for following tools directories:
lib/bpf
lib/subcmd
build

This are now cleaned via 'make -C tools clean' command.

---
 tools/Makefile | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

index 0ba0df3b516f..4e8e10755e0d 100644
@@ -127,6 +127,12 @@ liblockdep_clean:
 libapi_clean:
 	$(call descend,lib/api,clean)
 
+libbpf_clean:
+	$(call descend,lib/bpf,clean)
+
+libsubcmd_clean:
+	$(call descend,lib/subcmd,clean)
+
 perf_clean:
 	$(call descend,$(@:_clean=),clean)
 

@@ -142,9 +148,12 @@ tmon_clean:
 freefall_clean:
 	$(call descend,laptop/freefall,clean)
 
+build_clean:
+	$(call descend,build,clean)
+
 clean: acpi_clean cgroup_clean cpupower_clean hv_clean firewire_clean lguest_clean \
 		perf_clean selftests_clean turbostat_clean usb_clean virtio_clean \
 		vm_clean net_clean iio_clean x86_energy_perf_policy_clean tmon_clean \
-		freefall_clean
+		freefall_clean build_clean libbpf_clean libsubcmd_clean liblockdep_clean
 
 .PHONY: FORCE

2.4.3

Add *.cmd files to be removed within clean target.

---
 tools/lib/bpf/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 919b71780710..84e0e986ade4 100644
@@ -192,7 +192,7 @@ config-clean:
 	$(Q)$(MAKE) -C $(srctree)/tools/build/feature/ clean >/dev/null
 
 clean:
-	$(call QUIET_CLEAN, libbpf) $(RM) *.o *~ $(TARGETS) *.a *.so $(VERSION_FILES) .*.d \
+	$(call QUIET_CLEAN, libbpf) $(RM) *.o *~ $(TARGETS) *.a *.so $(VERSION_FILES) .*.d .*.cmd \
 		$(RM) LIBBPF-CFLAGS
 	$(call QUIET_CLEAN, core-gen) $(RM) $(OUTPUT)FEATURE-DUMP.libbpf
 

2.4.3

Add *.cmd files to be removed within clean target.

---
 tools/lib/lockdep/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 7e319afac78a..90d2baeb621a 100644
@@ -149,7 +149,7 @@ install_lib: all_cmd
 install: install_lib
 
 clean:
-	$(RM) *.o *~ $(TARGETS) *.a *liblockdep*.so* $(VERSION_FILES) .*.d
+	$(RM) *.o *~ $(TARGETS) *.a *liblockdep*.so* $(VERSION_FILES) .*.d .*.cmd
 	$(RM) tags TAGS
 
 PHONY += force

2.4.3

From: Jiri Olsa <[email protected]>

Adding missing bitmap.[ch] sources to the MINIFEST file.

---
 tools/perf/MANIFEST | 2 ++
 1 file changed, 2 insertions(+)

index ddf922f93aa1..2e1fa2357528 100644
@@ -28,6 +28,7 @@ tools/lib/string.c
 tools/lib/symbol/kallsyms.c
 tools/lib/symbol/kallsyms.h
 tools/lib/find_bit.c
+tools/lib/bitmap.c
 tools/include/asm/atomic.h
 tools/include/asm/barrier.h
 tools/include/asm/bug.h

@@ -57,6 +58,7 @@ tools/include/linux/rbtree_augmented.h
 tools/include/linux/string.h
 tools/include/linux/types.h
 tools/include/linux/err.h
+tools/include/linux/bitmap.h
 include/asm-generic/bitops/arch_hweight.h
 include/asm-generic/bitops/const_hweight.h
 include/asm-generic/bitops/fls64.h

2.4.3

To provide FEATURE-DUMP into $(FEATURE_DUMP_COPY)
if defined, with no further action.

Get feature dump of the current build:

  $ make feature-dump
    BUILD:   Doing 'make -j4' parallel build

Auto-detecting system features:
... dwarf: [ on ]

FEATURE-DUMP file available in FEATURE-DUMP

Get feature dump static build into /tmp/fd file:

  $ make feature-dump FEATURE_DUMP_COPY=/tmp/fd LDFLAGS=-static
    BUILD:   Doing 'make -j4' parallel build

Auto-detecting system features:
... dwarf: [ OFF ]

SNIP

FEATURE-DUMP file copied into /tmp/fd

---
 tools/perf/Makefile.perf | 11 +++++++++++
 1 file changed, 11 insertions(+)

index 0a22407e1d7d..f758a72df1b3 100644
@@ -611,6 +611,17 @@ clean: $(LIBTRACEEVENT)-clean $(LIBAPI)-clean $(LIBBPF)-clean $(LIBSUBCMD)-clean
 	$(python-clean)
 
 #
+# To provide FEATURE-DUMP into $(FEATURE_DUMP_COPY)
+# file if defined, with no further action.
+feature-dump:
+ifdef FEATURE_DUMP_COPY
+	@cp $(OUTPUT)FEATURE-DUMP $(FEATURE_DUMP_COPY)
+	@echo "FEATURE-DUMP file copied into $(FEATURE_DUMP_COPY)"
+else
+	@echo "FEATURE-DUMP file available in $(OUTPUT)FEATURE-DUMP"
+endif
+
+#
 # Trick: if ../../.git does not exist - we are building out of tree for example,
 # then force version regeneration:
 #

2.4.3

Introducing FEATURES_DUMP make variable to provide features detection dump file and bypass the feature detection.

The intention is to use this during build tests to skip repeated features detection, like:

Get feature dump static build into /tmp/fd file:

  $ make feature-dump FEATURE_DUMP_COPY=/tmp/fd LDFLAGS=-static
    BUILD:   Doing 'make -j4' parallel build

Auto-detecting system features:
... dwarf: [ OFF ]

SNIP

FEATURE-DUMP file copied into /tmp/fd

Use /tmp/fd to build perf:

  $ make FEATURES_DUMP=/tmp/fd LDFLAGS=-static

  $ file perf
  perf: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for ...

---
 tools/perf/Makefile.perf   | 14 +++++++++++++-
 tools/perf/config/Makefile |  4 ++++
 2 files changed, 17 insertions(+), 1 deletion(-)

index f758a72df1b3..5d34815c7ccb 100644
@@ -77,6 +77,9 @@ include config/utilities.mak
 # Define NO_AUXTRACE if you do not want AUX area tracing support
 #
 # Define NO_LIBBPF if you do not want BPF support
+#
+# Define FEATURES_DUMP to provide features detection dump file
+# and bypass the feature detection
 
 # As per kernel Makefile, avoid funny character set dependencies
 unexport LC_ALL

@@ -166,6 +169,15 @@ ifeq ($(config),1)
 include config/Makefile
 endif
 
+# The FEATURE_DUMP_EXPORT holds location of the actual
+# FEATURE_DUMP file to be used to bypass feature detection
+# (for bpf or any other subproject)
+ifeq ($(FEATURES_DUMP),)
+FEATURE_DUMP_EXPORT := $(realpath $(OUTPUT)FEATURE-DUMP)
+else
+FEATURE_DUMP_EXPORT := $(FEATURES_DUMP)
+endif
+
 export prefix bindir sharedir sysconfdir DESTDIR
 
 # sparse is architecture-neutral, which means that we need to tell it

@@ -436,7 +448,7 @@ $(LIBAPI)-clean:
 	$(Q)$(MAKE) -C $(LIB_DIR) O=$(OUTPUT) clean >/dev/null
 
 $(LIBBPF): fixdep FORCE
-	$(Q)$(MAKE) -C $(BPF_DIR) O=$(OUTPUT) $(OUTPUT)libbpf.a FEATURES_DUMP=$(realpath $(OUTPUT)FEATURE-DUMP)
+	$(Q)$(MAKE) -C $(BPF_DIR) O=$(OUTPUT) $(OUTPUT)libbpf.a FEATURES_DUMP=$(FEATURE_DUMP_EXPORT)
 
 $(LIBBPF)-clean:
 	$(call QUIET_CLEAN, libbpf)

index 254d06e39bea..5b2f12d6be3e 100644
@@ -181,7 +181,11 @@ LDFLAGS += -Wl,-z,noexecstack
 
 EXTLIBS = -lpthread -lrt -lm -ldl
 
+ifeq ($(FEATURES_DUMP),)
 include $(srctree)/tools/build/Makefile.feature
+else
+include $(FEATURES_DUMP)
+endif
 
 ifeq ($(feature-stackprotector-all), 1)
   CFLAGS += -fstack-protector-all

2.4.3

Thank you for these two patch. However I found a small limitation. I must skip 'make clean' between two feature-dump, or the result is wrong.

For example:

$ make feature-dump LDFLAGS="-static"
  BUILD:   Doing 'make -j24' parallel build

Auto-detecting system features:
<SNIP>
... zlib: [ on ]
... lzma: [ OFF ] <--- looks good. I don't have static lzma library
... get_cpuid: [ on ]
... bpf: [ on ]

$ make feature-dump
  BUILD:   Doing 'make -j24' parallel build

Auto-detecting system features:
<SNIP>

...                          zlib: [ on  ]
...                          lzma: [ on  ]   <--- also good
...                     get_cpuid: [ on  ]
...                           bpf: [ on  ]

$ make feature-dump LDFLAGS="-static"
  BUILD:   Doing 'make -j24' parallel build

Auto-detecting system features:
<SNIP>

...                          zlib: [ on  ]
...                          lzma: [ on  ]     <--- Bad...
...                     get_cpuid: [ on  ]
...                           bpf: [ on  ]

Thank you.

On Tue, Jan 12, 2016 at 05:28:28PM +0800, Wangnan (F) wrote:

SNIP

right, it's because feature is not under build framework and compilation does not care about flags change..

I'll try to check on that.. does this patch help to improve the current speed or do you need to solve this to get substantial speedup?

thanks,
jirka

I'm working on it. I have already done it, but still have some problem. On some machine 'make_static' test target lost '-liberty -lz' in its linker options. Seems something wrong in this part:

ifeq ($(feature-libbfd), 1)
EXTLIBS += -lbfd

  # call all detections now so we get correct
  # status in VF output
  $(call feature_check,liberty)
  $(call feature_check,liberty-z)
  $(call feature_check,cplus-demangle)

ifeq ($(feature-liberty), 1)

    EXTLIBS += -liberty
  else
    ifeq ($(feature-liberty-z), 1)
      EXTLIBS += -liberty -lz
    endif

endif
endif

On a 24 core machine the build-test takes about 16 mins. Faster than before:

real 16m7.642s
user 171m47.488s
sys 19m10.692s

On Wed, Jan 13, 2016 at 05:46:13PM +0800, Wangnan (F) wrote:

SNIP

IIRC we had issues with this before.. different systems provide those libs in different way, like:
2cf9040714f3 perf tools: Fix bfd dependency libraries detection

you might want to check feature detection code if it gets built properly on your system

jirka

No this problem. There's no feature-libiberty and feature-libiberty-z in output feature-dump file. This is the key reason.

Finished.

On a 24 core machine:

Without this feature, a full build-test takes:

real 21m16.579s
user 176m12.989s
sys 20m54.950s

With this feature:

real 16m0.619s
user 172m53.896s
sys 19m22.777s

Speedup: 24.75%.

So this feature is really useful.

Thank you.

looks great, thanks

jirka