perf/core fixes

Hi Ingo,

Please consider pulling,

- Arnaldo

The following changes since commit 3eb9ede23bdd96e9ba60e2b4d4d17a7c35d58448:

Merge tag 'perf-core-for-mingo' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux into perf/core (2016-01-09 17:17:33 +0100)

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git tags/perf-core-for-mingo

for you to fetch changes up to b0500c169b4069e40f03391c7280cd6eaf849e49:

perf test: Reset err after using it hold errcode in hist testcases (2016-01-11 19:22:22 -0300)

----------------------------------------------------------------
perf/core fixes:

Developer stuff:

- Fix a few clean targets in tools/ (Jiri Olsa)

- Add missing sources to perf's MANIFEST, fixing the out of tree build with 'make perf-tar*-src-pkg' tarballs (Jiri Olsa)

- Fix bpf related build problems in PowerPC (Naveen N. Rao, Wang Nan)

- 'make -C tools/perf build-test' fixes (Wang Nan)

- Fix 'perf test hist' entry (Wang Nan)

- Add BPF feature check to test-all, as in an environment with all other features enabled, BPF would be considered enabled without doing real feature check. (Wang Nan)

----------------------------------------------------------------
Jiri Olsa (4):
tools: Add clean targets for tools directory tools bpf: Add *.cmd files clean up
tools lockdep: Add *.cmd files clean up perf tools: Add missing sources to perf's MANIFEST

Naveen N. Rao (1):
perf bpf: Fix build breakage due to libbpf

Wang Nan (7):
perf tools: Add -lutil in python lib list for broken python-config perf tools: Fix phony build target for build-test
perf tools: Fix PowerPC native building
tools: Move Makefile.arch from perf/config to tools/scripts tools build: Add BPF feature check to test-all
perf test: Fix false TEST_OK result for 'perf test hist' perf test: Reset err after using it hold errcode in hist testcases

 tools/Makefile                               | 11 ++++++++++-
 tools/build/feature/test-all.c               |  5 +++++
 tools/build/feature/test-bpf.c               | 20 +++++++++++++++++++-
 tools/lib/bpf/Makefile                       | 18 +++++++++---------
 tools/lib/bpf/bpf.c                          |  4 ++--
 tools/lib/lockdep/Makefile                   |  2 +-
 tools/perf/MANIFEST                          |  2 ++
 tools/perf/config/Makefile                   |  4 ++--
 tools/perf/tests/hists_common.c              |  5 -----
 tools/perf/tests/hists_cumulate.c            |  1 +
 tools/perf/tests/hists_filter.c              |  1 +
 tools/perf/tests/hists_link.c                |  1 +
 tools/perf/tests/hists_output.c              |  1 +
 tools/perf/tests/make                        | 18 ++++++++++++++++--
 tools/{perf/config => scripts}/Makefile.arch |  0
 15 files changed, 70 insertions(+), 23 deletions(-)
 rename tools/{perf/config => scripts}/Makefile.arch (100%)

From: Jiri Olsa <[email protected]>

Adding missing clean targets for following tools directories:

lib/bpf
lib/subcmd
build

This are now cleaned via 'make -C tools clean' command.

---
 tools/Makefile | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

index 0ba0df3b516f..4e8e10755e0d 100644
@@ -127,6 +127,12 @@ liblockdep_clean:
 libapi_clean:
 	$(call descend,lib/api,clean)
 
+libbpf_clean:
+	$(call descend,lib/bpf,clean)
+
+libsubcmd_clean:
+	$(call descend,lib/subcmd,clean)
+
 perf_clean:
 	$(call descend,$(@:_clean=),clean)
 

@@ -142,9 +148,12 @@ tmon_clean:
 freefall_clean:
 	$(call descend,laptop/freefall,clean)
 
+build_clean:
+	$(call descend,build,clean)
+
 clean: acpi_clean cgroup_clean cpupower_clean hv_clean firewire_clean lguest_clean \
 		perf_clean selftests_clean turbostat_clean usb_clean virtio_clean \
 		vm_clean net_clean iio_clean x86_energy_perf_policy_clean tmon_clean \
-		freefall_clean
+		freefall_clean build_clean libbpf_clean libsubcmd_clean liblockdep_clean
 
 .PHONY: FORCE

2.1.0

From: Jiri Olsa <[email protected]>

Add *.cmd files to be removed within clean target.

---
 tools/lib/bpf/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 919b71780710..84e0e986ade4 100644
@@ -192,7 +192,7 @@ config-clean:
 	$(Q)$(MAKE) -C $(srctree)/tools/build/feature/ clean >/dev/null
 
 clean:
-	$(call QUIET_CLEAN, libbpf) $(RM) *.o *~ $(TARGETS) *.a *.so $(VERSION_FILES) .*.d \
+	$(call QUIET_CLEAN, libbpf) $(RM) *.o *~ $(TARGETS) *.a *.so $(VERSION_FILES) .*.d .*.cmd \
 		$(RM) LIBBPF-CFLAGS
 	$(call QUIET_CLEAN, core-gen) $(RM) $(OUTPUT)FEATURE-DUMP.libbpf
 

2.1.0

From: Jiri Olsa <[email protected]>

Add *.cmd files to be removed within clean target.

---
 tools/lib/lockdep/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 7e319afac78a..90d2baeb621a 100644
@@ -149,7 +149,7 @@ install_lib: all_cmd
 install: install_lib
 
 clean:
-	$(RM) *.o *~ $(TARGETS) *.a *liblockdep*.so* $(VERSION_FILES) .*.d
+	$(RM) *.o *~ $(TARGETS) *.a *liblockdep*.so* $(VERSION_FILES) .*.d .*.cmd
 	$(RM) tags TAGS
 
 PHONY += force

2.1.0

From: Jiri Olsa <[email protected]>

Adding missing bitmap.[ch] sources to the MANIFEST file. Fixes building 'make perf-*-src-pkg' generated tarballs.

---
 tools/perf/MANIFEST | 2 ++
 1 file changed, 2 insertions(+)

index ddf922f93aa1..2e1fa2357528 100644
@@ -28,6 +28,7 @@ tools/lib/string.c
 tools/lib/symbol/kallsyms.c
 tools/lib/symbol/kallsyms.h
 tools/lib/find_bit.c
+tools/lib/bitmap.c
 tools/include/asm/atomic.h
 tools/include/asm/barrier.h
 tools/include/asm/bug.h

@@ -57,6 +58,7 @@ tools/include/linux/rbtree_augmented.h
 tools/include/linux/string.h
 tools/include/linux/types.h
 tools/include/linux/err.h
+tools/include/linux/bitmap.h
 include/asm-generic/bitops/arch_hweight.h
 include/asm-generic/bitops/const_hweight.h
 include/asm-generic/bitops/fls64.h

2.1.0

From: Wang Nan <[email protected]>

On some system the perf-config is broken, causes link failure like this:

/usr/lib64/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_forkpty':
/opt/wangnan/yocto-build/tmp-eglibc/work/x86_64-oe-linux/python/2.7.3-r0.3.1/Python-2.7.3/./Modules/posixmodule.c:3816: undefined reference to forkpty' /usr/lib64/python2.7/config/libpython2.7.a(posixmodule.o): In function posix_openpty':
/opt/wangnan/yocto-build/tmp-eglibc/work/x86_64-oe-linux/python/2.7.3-r0.3.1/Python-2.7.3/./Modules/posixmodule.c:3756: undefined reference to `openpty'
collect2: error: ld returned 1 exit status
make[1]: *** [/home/wangnan/kernel-hydrogen/tools/perf/out/perf] Error 1
make: *** [all] Error 2

  $ python-config --libs
  -lpthread -ldl -lpthread -lutil -lm -lpython2.7

In this case a '-lutil' should be appended to -lpython2.7.

(I know we have --start-group and --end-group. I can see them in command line of collect2 by strace. However it doesn't work. Seems I have a broken environment?)

---
 tools/perf/config/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 254d06e39bea..0793c7654858 100644
@@ -493,7 +493,7 @@ else
 
       PYTHON_EMBED_LDOPTS := $(shell $(PYTHON_CONFIG_SQ) --ldflags 2>/dev/null)
       PYTHON_EMBED_LDFLAGS := $(call strip-libs,$(PYTHON_EMBED_LDOPTS))
-      PYTHON_EMBED_LIBADD := $(call grep-libs,$(PYTHON_EMBED_LDOPTS))
+      PYTHON_EMBED_LIBADD := $(call grep-libs,$(PYTHON_EMBED_LDOPTS)) -lutil
       PYTHON_EMBED_CCOPTS := $(shell $(PYTHON_CONFIG_SQ) --cflags 2>/dev/null)
       FLAGS_PYTHON_EMBED := $(PYTHON_EMBED_CCOPTS) $(PYTHON_EMBED_LDOPTS)
 

2.1.0

From: Wang Nan <[email protected]>

make_kernelsrc and make_kernelsrc_tools are skiped if a previous build-test is done, because 'make build-test' creates two files with same names. To avoid this, they should be included in .PHONY list.

---
 tools/perf/tests/make | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index c1fbb8e884c0..130be7c88824 100644
@@ -280,5 +280,5 @@ all: $(run) $(run_O) tarpkg make_kernelsrc make_kernelsrc_tools
 out: $(run_O)
 	@echo OK
 
-.PHONY: all $(run) $(run_O) tarpkg clean
+.PHONY: all $(run) $(run_O) tarpkg clean make_kernelsrc make_kernelsrc_tools
 endif # ifndef MK

2.1.0

From: Wang Nan <[email protected]>

Checks BPF syscall number, turn off libbpf building on platform doesn't correctly support sys_bpf instead of blocking compiling.

---
 tools/build/feature/test-bpf.c | 20 +++++++++++++++++++-
 tools/lib/bpf/bpf.c            |  4 ++--
 2 files changed, 21 insertions(+), 3 deletions(-)

index 062bac811af9..b389026839b9 100644
@@ -1,9 +1,23 @@
+#include <asm/unistd.h>
 #include <linux/bpf.h>
+#include <unistd.h>
+
+#ifndef __NR_bpf
+# if defined(__i386__)
+#  define __NR_bpf 357
+# elif defined(__x86_64__)
+#  define __NR_bpf 321
+# elif defined(__aarch64__)
+#  define __NR_bpf 280
+#  error __NR_bpf not defined. libbpf does not support your arch.
+# endif
+#endif
 
 int main(void)
 {
 	union bpf_attr attr;
 
+	/* Check fields in attr */
 	attr.prog_type = BPF_PROG_TYPE_KPROBE;
 	attr.insn_cnt = 0;
 	attr.insns = 0;

@@ -14,5 +28,9 @@ int main(void)
 	attr.kern_version = 0;
 
 	attr = attr;
-	return 0;
+	/*
+	 * Test existence of __NR_bpf and BPF_PROG_LOAD.
+	 * This call should fail if we run the testcase.
+	 */
+	return syscall(__NR_bpf, BPF_PROG_LOAD, attr, sizeof(attr));
 }

index 5bdc6eab6852..1f91cc941b7c 100644
@@ -14,8 +14,8 @@
 #include "bpf.h"
 
 /*
- * When building perf, unistd.h is override. Define __NR_bpf is
- * required to be defined.
+ * When building perf, unistd.h is overrided. __NR_bpf is
+ * required to be defined explicitly.
  */
 #ifndef __NR_bpf
 # if defined(__i386__)

2.1.0

From: Wang Nan <[email protected]>

After this patch other directories can use this architecture detector without directly including it from perf's directory. Libbpf would utilize it to get proper $(ARCH) so it can receive correct uapi include directory.

---
 tools/perf/config/Makefile      |  2 +-
 tools/perf/config/Makefile.arch | 18 ------------------
 tools/perf/tests/make           | 16 +++++++++++++++-
 tools/scripts/Makefile.arch     | 18 ++++++++++++++++++
 4 files changed, 34 insertions(+), 20 deletions(-)
 delete mode 100644 tools/perf/config/Makefile.arch
 create mode 100644 tools/scripts/Makefile.arch

index 0793c7654858..7545ba60053e 100644
@@ -17,7 +17,7 @@ detected_var = $(shell echo "$(1)=$($(1))" >> $(OUTPUT).config-detected)
 
 CFLAGS := $(EXTRA_CFLAGS) $(EXTRA_WARNINGS)
 
-include $(src-perf)/config/Makefile.arch
+include $(srctree)/tools/scripts/Makefile.arch
 
 $(call detected_var,ARCH)
 

deleted file mode 100644
index e11fbd6fae78..000000000000
@@ -1,18 +0,0 @@
-ifndef ARCH
-ARCH := $(shell uname -m 2>/dev/null || echo not)
-endif
-
-ARCH := $(shell echo $(ARCH) | sed -e s/i.86/x86/ -e s/x86_64/x86/ \
-                                  -e s/sun4u/sparc/ -e s/sparc64/sparc/ \
-                                  -e /arm64/!s/arm.*/arm/ -e s/sa110/arm/ \
-                                  -e s/s390x/s390/ -e s/parisc64/parisc/ \
-                                  -e s/ppc.*/powerpc/ -e s/mips.*/mips/ \
-                                  -e s/sh[234].*/sh/ -e s/aarch64.*/arm64/ \
-                                  -e s/tile.*/tile/ )
-
-LP64 := $(shell echo __LP64__ | ${CC} ${CFLAGS} -E -x c - | tail -n 1)
-ifeq ($(LP64), 1)
-  IS_64_BIT := 1
-else
-  IS_64_BIT := 0
-endif

index 130be7c88824..df38decc48c3 100644
@@ -1,3 +1,5 @@
+include ../scripts/Makefile.include
+
 ifndef MK
 ifeq ($(MAKECMDGOALS),)
 # no target specified, trigger the whole suite

@@ -12,7 +14,19 @@ endif
 else
 PERF := .
 
-include config/Makefile.arch
+# As per kernel Makefile, avoid funny character set dependencies
+unexport LC_ALL
+LC_COLLATE=C
+LC_NUMERIC=C
+export LC_COLLATE LC_NUMERIC
+
+ifeq ($(srctree),)
+srctree := $(patsubst %/,%,$(dir $(shell pwd)))
+srctree := $(patsubst %/,%,$(dir $(srctree)))
+#$(info Determined 'srctree' to be $(srctree))
+endif
+
+include $(srctree)/tools/scripts/Makefile.arch
 
 # FIXME looks like x86 is the only arch running tests ;-)
 # we need some IS_(32/64) flag to make this generic

new file mode 100644
index 000000000000..e11fbd6fae78
@@ -0,0 +1,18 @@
+ifndef ARCH
+ARCH := $(shell uname -m 2>/dev/null || echo not)
+endif
+
+ARCH := $(shell echo $(ARCH) | sed -e s/i.86/x86/ -e s/x86_64/x86/ \
+                                  -e s/sun4u/sparc/ -e s/sparc64/sparc/ \
+                                  -e /arm64/!s/arm.*/arm/ -e s/sa110/arm/ \
+                                  -e s/s390x/s390/ -e s/parisc64/parisc/ \
+                                  -e s/ppc.*/powerpc/ -e s/mips.*/mips/ \
+                                  -e s/sh[234].*/sh/ -e s/aarch64.*/arm64/ \
+                                  -e s/tile.*/tile/ )
+
+LP64 := $(shell echo __LP64__ | ${CC} ${CFLAGS} -E -x c - | tail -n 1)
+ifeq ($(LP64), 1)
+  IS_64_BIT := 1
+else
+  IS_64_BIT := 0
+endif

2.1.0

From: "Naveen N. Rao" <[email protected]>

perf build is currently (v4.4-rc5) broken on powerpc:

bpf.c:28:4: error: #error __NR_bpf not defined. libbpf does not support your arch.
# error __NR_bpf not defined. libbpf does not support your arch.

      ^

Fix this by including tools/scripts/Makefile.arch for the proper $ARCH macro. While at it, remove redundant LP64 macro definition.

Also, since libbpf require $(srctree) now, detect the path of srctree like perf.

---
 tools/lib/bpf/Makefile | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

index 84e0e986ade4..fc1bc75ae56d 100644
@@ -6,6 +6,12 @@ BPF_EXTRAVERSION = 1
 
 MAKEFLAGS += --no-print-directory
 
+ifeq ($(srctree),)
+srctree := $(patsubst %/,%,$(dir $(shell pwd)))
+srctree := $(patsubst %/,%,$(dir $(srctree)))
+srctree := $(patsubst %/,%,$(dir $(srctree)))
+#$(info Determined 'srctree' to be $(srctree))
+endif
 
 # Makefiles suck: This macro sets a default value of $(2) for the
 # variable named by $(1), unless the variable has been set by

@@ -31,7 +37,8 @@ INSTALL = install
 DESTDIR ?=
 DESTDIR_SQ = '$(subst ','\'',$(DESTDIR))'
 
-LP64 := $(shell echo __LP64__ | ${CC} ${CFLAGS} -E -x c - | tail -n 1)
+include $(srctree)/tools/scripts/Makefile.arch
+
 ifeq ($(LP64), 1)
   libdir_relative = lib64
 else

@@ -57,13 +64,6 @@ ifndef VERBOSE
   VERBOSE = 0
 endif
 
-ifeq ($(srctree),)
-srctree := $(patsubst %/,%,$(dir $(shell pwd)))
-srctree := $(patsubst %/,%,$(dir $(srctree)))
-srctree := $(patsubst %/,%,$(dir $(srctree)))
-#$(info Determined 'srctree' to be $(srctree))
-endif
-
 FEATURE_USER = .libbpf
 FEATURE_TESTS = libelf libelf-getphdrnum libelf-mmap bpf
 FEATURE_DISPLAY = libelf bpf

2.1.0

From: Wang Nan <[email protected]>

The test-all.c file doesn't check BPF related features. For an environment with all other features enabled, BPF would be considered enabled without doing real feature check.

This patch adds test-bpf.c into test-all.c.

---
 tools/build/feature/test-all.c | 5 +++++
 1 file changed, 5 insertions(+)

index 33cf6f20bd4e..81025cade45f 100644
@@ -125,6 +125,10 @@
 # include "test-get_cpuid.c"
 #undef main
 
+#define main main_test_bpf
+# include "test-bpf.c"
+#undef main
+
 int main(int argc, char *argv[])
 {
 	main_test_libpython();

@@ -153,6 +157,7 @@ int main(int argc, char *argv[])
 	main_test_pthread_attr_setaffinity_np();
 	main_test_lzma();
 	main_test_get_cpuid();
+	main_test_bpf();
 
 	return 0;
 }

2.1.0

From: Wang Nan <[email protected]>

Commit 71d6de64fedd ("perf test: Fix hist testcases when kptr_restrict is on") solves a double free problem when 'perf test hist' calling setup_fake_machine(). However, the result is still incorrect. For example:

  $ ./perf test -v 'filtering hist entries'
  25: Test filtering hist entries                              :
  --- start ---
  test child forked, pid 4186
  Cannot create kernel maps
  test child finished with 0
  ---- end ----
  Test filtering hist entries: Ok

In this case the body of this test is not get executed at all, but the result is 'Ok'.

Actually, in setup_fake_machine() there's no need to create real kernel maps. What we want are the fake maps. This patch removes the machine__create_kernel_maps() in setup_fake_machine(), so it won't be affected by kptr_restrict setting.

Test result:

  $ cat /proc/sys/kernel/kptr_restrict
  1
  $ ~/perf test -v hist
  15: Test matching and linking multiple hists                 :
  --- start ---
  test child forked, pid 24031
  test child finished with 0
  ---- end ----
  Test matching and linking multiple hists: Ok
  [SNIP]

---
 tools/perf/tests/hists_common.c | 5 -----
 1 file changed, 5 deletions(-)

index bcfd081ee1d2..071a8b5f5232 100644
@@ -87,11 +87,6 @@ struct machine *setup_fake_machine(struct machines *machines)
 		return NULL;
 	}
 
-	if (machine__create_kernel_maps(machine)) {
-		pr_debug("Cannot create kernel maps\n");
-		return NULL;
-	}
-
 	for (i = 0; i < ARRAY_SIZE(fake_threads); i++) {
 		struct thread *thread;
 

2.1.0

From: Wang Nan <[email protected]>

All hists test cases forget to reset err after using it to hold an error code. If error occure in setup_fake_machine() it incorrectly return TEST_OK.

This patch fixes it.

---
 tools/perf/tests/hists_cumulate.c | 1 +
 tools/perf/tests/hists_filter.c   | 1 +
 tools/perf/tests/hists_link.c     | 1 +
 tools/perf/tests/hists_output.c   | 1 +
 4 files changed, 4 insertions(+)

index e36089212061..5e6a86e50fb9 100644
@@ -706,6 +706,7 @@ int test__hists_cumulate(int subtest __maybe_unused)
 	err = parse_events(evlist, "cpu-clock", NULL);
 	if (err)
 		goto out;
+	err = TEST_FAIL;
 
 	machines__init(&machines);
 

index 2a784befd9ce..351a42463444 100644
@@ -120,6 +120,7 @@ int test__hists_filter(int subtest __maybe_unused)
 	err = parse_events(evlist, "task-clock", NULL);
 	if (err)
 		goto out;
+	err = TEST_FAIL;
 
 	/* default sort order (comm,dso,sym) will be used */
 	if (setup_sorting(NULL) < 0)

index c764d69ac6ef..64b257d8d557 100644
@@ -293,6 +293,7 @@ int test__hists_link(int subtest __maybe_unused)
 	if (err)
 		goto out;
 
+	err = TEST_FAIL;
 	/* default sort order (comm,dso,sym) will be used */
 	if (setup_sorting(NULL) < 0)
 		goto out;

index ebe6cd485b5d..b231265148d8 100644
@@ -597,6 +597,7 @@ int test__hists_output(int subtest __maybe_unused)
 	err = parse_events(evlist, "cpu-clock", NULL);
 	if (err)
 		goto out;
+	err = TEST_FAIL;
 
 	machines__init(&machines);
 

2.1.0

Pulled into tip:perf/urgent, thanks a lot Arnaldo!

Ingo