Skip to contentSkip to conversation

Copy hashmap to tools/perf/util, use in perf expr

PATCH In review Ian Rogers 3 participants 18 messages May 15, 2020 – May 20, 2020

Subsystems (5)
lore
Share & tools
lore

Ian Rogers

author reviewer
#0 lore raw

v3 cover Copy hashmap to tools/perf/util, use in perf expr

Perf's expr code currently builds an array of strings then removes duplicates. The array is larger than necessary and has recently been increased in size. When this was done it was commented that a hashmap would be preferable.

libbpf has a hashmap but libbpf isn't currently required to build perf. To satisfy various concerns this change copies libbpf's hashmap into tools/perf/util, it then adds a check in perf that the two are in sync.

Andrii's patch to hashmap from bpf-next is brought into this set to fix issues with hashmap__clear.

Two minor changes to libbpf's hashmap are made that remove an unused dependency and fix a compiler warning.

Two perf test changes are also brought in as they need refactoring to account for the expr API change and it is expected they will land ahead of this.
https://lore.kernel.org/lkml/[email protected]/

Tested with 'perf test' and 'make -C tools/perf build-test'.

The hashmap change was originally part of an RFC:
https://lore.kernel.org/lkml/[email protected]/

v3. addresses review feedback from Andrii Nakryiko <[email protected]> and Jiri Olsa <[email protected]>.
v2. moves hashmap into tools/perf/util rather than libapi, to allow hashmap's libbpf symbols to be visible when built statically for testing.

Andrii Nakryiko (1):
libbpf: Fix memory leak and possible double-free in hashmap__clear

Ian Rogers (6):
libbpf hashmap: Remove unused #include
libbpf hashmap: Fix signedness warnings
tools lib/api: Copy libbpf hashmap to tools/perf/util perf test: Provide a subtest callback to ask for the reason for
skipping a subtest
perf test: Improve pmu event metric testing
perf expr: Migrate expr ids table to a hashmap

 tools/lib/bpf/hashmap.c         |  10 +-
 tools/lib/bpf/hashmap.h         |   1 -
 tools/perf/check-headers.sh     |   4 +
 tools/perf/tests/builtin-test.c |  18 ++-
 tools/perf/tests/expr.c         |  44 +++---
 tools/perf/tests/pmu-events.c   | 169 ++++++++++++++++++++++-
 tools/perf/tests/tests.h        |   4 +
 tools/perf/util/Build           |   4 +
 tools/perf/util/expr.c          | 129 +++++++++--------
 tools/perf/util/expr.h          |  26 ++--
 tools/perf/util/expr.y          |  22 +--
 tools/perf/util/hashmap.c       | 238 ++++++++++++++++++++++++++++++++
 tools/perf/util/hashmap.h       | 177 ++++++++++++++++++++++++
 tools/perf/util/metricgroup.c   |  92 ++++++------
 tools/perf/util/stat-shadow.c   |  49 ++++---
 15 files changed, 798 insertions(+), 189 deletions(-)
 create mode 100644 tools/perf/util/hashmap.c
 create mode 100644 tools/perf/util/hashmap.h
Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
author reviewer
#1 lore raw

PATCH v3 1/7 libbpf: Fix memory leak and possible double-free in hashmap__clear

From: Andrii Nakryiko <[email protected]>

Fix memory leak in hashmap_clear() not freeing hashmap_entry structs for each of the remaining entries. Also NULL-out bucket list to prevent possible double-free between hashmap__clear() and hashmap__free().

Running test_progs-asan flavor clearly showed this problem.

---
 tools/lib/bpf/hashmap.c | 7 +++++++
 1 file changed, 7 insertions(+)
tools/lib/bpf/hashmap.c +7 -0
index 54c30c802070..cffb96202e0d 100644
@@ -59,7 +59,14 @@ struct hashmap *hashmap__new(hashmap_hash_fn hash_fn,
 
 void hashmap__clear(struct hashmap *map)
 {
+	struct hashmap_entry *cur, *tmp;
+	int bkt;
+
+	hashmap__for_each_entry_safe(map, cur, tmp, bkt) {
+		free(cur);
+	}
 	free(map->buckets);
+	map->buckets = NULL;
 	map->cap = map->cap_bits = map->sz = 0;
 }
 
Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
author reviewer
#2 lore raw

PATCH v3 2/7 libbpf hashmap: Remove unused #include

In reply to Ian Rogers

Remove #include of libbpf_internal.h that is unused. Discussed in this thread:
https://lore.kernel.org/lkml/CAEf4BzZRmiEds_8R8g4vaAeWvJzPb4xYLnpF0X2VNY8oTzkphQ@mail.gmail.com/

---
 tools/lib/bpf/hashmap.h | 1 -
 1 file changed, 1 deletion(-)
tools/lib/bpf/hashmap.h +0 -1
index bae8879cdf58..e823b35e7371 100644
@@ -15,7 +15,6 @@
 #else
 #include <bits/reg.h>
 #endif
-#include "libbpf_internal.h"
 
 static inline size_t hash_bits(size_t h, int bits)
 {
Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
author reviewer
#3 lore raw

PATCH v3 3/7 libbpf hashmap: Fix signedness warnings

In reply to Ian Rogers

Fixes the following warnings:

hashmap.c: In function ‘hashmap__clear’:
hashmap.h:150:20: error: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Werror=sign-compare]
150 | for (bkt = 0; bkt < map->cap; bkt++) \

hashmap.c: In function ‘hashmap_grow’:
hashmap.h:150:20: error: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Werror=sign-compare]
150 | for (bkt = 0; bkt < map->cap; bkt++) \

---
 tools/lib/bpf/hashmap.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
tools/lib/bpf/hashmap.c +2 -3
index cffb96202e0d..a405dad068f5 100644
@@ -60,7 +60,7 @@ struct hashmap *hashmap__new(hashmap_hash_fn hash_fn,
 void hashmap__clear(struct hashmap *map)
 {
 	struct hashmap_entry *cur, *tmp;
-	int bkt;
+	size_t bkt;
 
 	hashmap__for_each_entry_safe(map, cur, tmp, bkt) {
 		free(cur);
@@ -100,8 +100,7 @@ static int hashmap_grow(struct hashmap *map)
 	struct hashmap_entry **new_buckets;
 	struct hashmap_entry *cur, *tmp;
 	size_t new_cap_bits, new_cap;
-	size_t h;
-	int bkt;
+	size_t h, bkt;
 
 	new_cap_bits = map->cap_bits + 1;
 	if (new_cap_bits < HASHMAP_MIN_CAP_BITS)
Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
author reviewer
#4 lore raw

PATCH v3 4/7 tools lib/api: Copy libbpf hashmap to tools/perf/util

In reply to Ian Rogers

Allow use of hashmap in perf. Modify perf's check-headers.sh script to check that the files are kept in sync, in the same way kernel headers are checked. This will warn if they are out of sync at the start of a perf build.

---
 tools/perf/check-headers.sh |   4 +
 tools/perf/util/Build       |   4 +
 tools/perf/util/hashmap.c   | 238 ++++++++++++++++++++++++++++++++++++
 tools/perf/util/hashmap.h   | 177 +++++++++++++++++++++++++++
 4 files changed, 423 insertions(+)
 create mode 100644 tools/perf/util/hashmap.c
 create mode 100644 tools/perf/util/hashmap.h
tools/perf/check-headers.sh +4 -0
index cf147db4e5ca..94c2bc22c2bb 100755
@@ -128,4 +128,8 @@ check arch/x86/lib/insn.c             '-I "^#include [\"<]\(../include/\)*asm/in
 # diff non-symmetric files
 check_2 tools/perf/arch/x86/entry/syscalls/syscall_64.tbl arch/x86/entry/syscalls/syscall_64.tbl
 
+# check duplicated library files
+check_2 tools/perf/util/hashmap.h tools/lib/bpf/hashmap.h
+check_2 tools/perf/util/hashmap.c tools/lib/bpf/hashmap.c
+
 cd tools/perf
tools/perf/util/Build +4 -0
index ca07a162d602..880fcdd1ab11 100644
@@ -136,6 +136,10 @@ perf-$(CONFIG_LIBELF) += symbol-elf.o
 perf-$(CONFIG_LIBELF) += probe-file.o
 perf-$(CONFIG_LIBELF) += probe-event.o
 
+ifndef CONFIG_LIBBPF
+perf-y += hashmap.o
+endif
+
 ifndef CONFIG_LIBELF
 perf-y += symbol-minimal.o
 endif
tools/perf/util/hashmap.c new file +238 -0

238 lines not on this page · raw message

tools/perf/util/hashmap.h new file +177 -0

177 lines not on this page · raw message

Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
author reviewer
#5 lore raw

PATCH v3 5/7 perf test: Provide a subtest callback to ask for the reason for skipping a subtest

In reply to Ian Rogers

Now subtests can inform why a test was skipped. The upcoming patch improvint PMU event metric testing will use it.

Cc Adrian Hunter <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jin Yao <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc John Garry <[email protected]>
Cc Kajol Jain <[email protected]>
Cc Kan Liang <[email protected]>
Cc Leo Yan <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Paul Clarke <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephane Eranian <[email protected]>
[ split from a larger patch ]
---
 tools/perf/tests/builtin-test.c | 11 +++++++++--
 tools/perf/tests/tests.h        |  1 +
 2 files changed, 10 insertions(+), 2 deletions(-)
tools/perf/tests/builtin-test.c +9 -2
index 3471ec52ea11..baee735e6aa5 100644
@@ -429,8 +429,15 @@ static int test_and_print(struct test *t, bool force_skip, int subtest)
 	case TEST_OK:
 		pr_info(" Ok\n");
 		break;
-	case TEST_SKIP:
-		color_fprintf(stderr, PERF_COLOR_YELLOW, " Skip\n");
+	case TEST_SKIP: {
+		const char *skip_reason = NULL;
+		if (t->subtest.skip_reason)
+			skip_reason = t->subtest.skip_reason(subtest);
+		if (skip_reason)
+			color_fprintf(stderr, PERF_COLOR_YELLOW, " Skip (%s)\n", skip_reason);
+		else
+			color_fprintf(stderr, PERF_COLOR_YELLOW, " Skip\n");
+	}
 		break;
 	case TEST_FAIL:
 	default:
tools/perf/tests/tests.h +1 -0
index d6d4ac34eeb7..88e45aeab94f 100644
@@ -34,6 +34,7 @@ struct test {
 		bool skip_if_fail;
 		int (*get_nr)(void);
 		const char *(*get_desc)(int subtest);
+		const char *(*skip_reason)(int subtest);
 	} subtest;
 	bool (*is_supported)(void);
 	void *priv;
Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
author reviewer
#6 lore raw

PATCH v3 6/7 perf test: Improve pmu event metric testing

In reply to Ian Rogers

Break pmu-events test into 2 and add a test to verify that all pmu metric expressions simply parse. Try to parse all metric ids/events, skip/warn if metrics for the current architecture fail to parse. To support warning for a skip, and an ability for a subtest to describe why it skips.

Tested on power9, skylakex, haswell, broadwell, westmere, sandybridge and ivybridge.

May skip/warn on other architectures if metrics are invalid. In particular s390 is untested, but its expressions are trivial. The untested architectures with expressions are power8, cascadelakex, tremontx, skylake, jaketown, ivytown and variants of haswell and broadwell.

v3. addresses review comments from John Garry <[email protected]>, Jiri Olsa <[email protected]> and Arnaldo Carvalho de Melo <[email protected]>.
v2. changes the commit message as event parsing errors no longer cause the test to fail.

Cc Adrian Hunter <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jin Yao <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc John Garry <[email protected]>
Cc Kajol Jain <[email protected]>
Cc Kan Liang <[email protected]>
Cc Leo Yan <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Paul Clarke <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephane Eranian <[email protected]>
[ split from a larger patch ]
---
 tools/perf/tests/builtin-test.c |   7 ++
 tools/perf/tests/pmu-events.c   | 168 ++++++++++++++++++++++++++++++--
 tools/perf/tests/tests.h        |   3 +
 3 files changed, 172 insertions(+), 6 deletions(-)
tools/perf/tests/builtin-test.c +7 -0
index baee735e6aa5..9553f8061772 100644
@@ -75,6 +75,13 @@ static struct test generic_tests[] = {
 	{
 		.desc = "PMU events",
 		.func = test__pmu_events,
+		.subtest = {
+			.skip_if_fail	= false,
+			.get_nr		= test__pmu_events_subtest_get_nr,
+			.get_desc	= test__pmu_events_subtest_get_desc,
+			.skip_reason	= test__pmu_events_subtest_skip_reason,
+		},
+
 	},
 	{
 		.desc = "DSO data read",
tools/perf/tests/pmu-events.c +162 -6

192 lines not on this page · raw message

tools/perf/tests/tests.h +3 -0
index 88e45aeab94f..6c6c4b6a4796 100644
@@ -51,6 +51,9 @@ int test__perf_evsel__tp_sched_test(struct test *test, int subtest);
 int test__syscall_openat_tp_fields(struct test *test, int subtest);
 int test__pmu(struct test *test, int subtest);
 int test__pmu_events(struct test *test, int subtest);
+const char *test__pmu_events_subtest_get_desc(int subtest);
+const char *test__pmu_events_subtest_skip_reason(int subtest);
+int test__pmu_events_subtest_get_nr(void);
 int test__attr(struct test *test, int subtest);
 int test__dso_data(struct test *test, int subtest);
 int test__dso_data_cache(struct test *test, int subtest);
Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
#7 lore raw

Re: [PATCH v3 6/7] perf test: Improve pmu event metric testing

Em Fri, May 15, 2020 at 03:17:31PM -0700, Ian Rogers escreveu:

Quoted 105 lines

Break pmu-events test into 2 and add a test to verify that all pmu metric expressions simply parse. Try to parse all metric ids/events, skip/warn if metrics for the current architecture fail to parse. To support warning for a skip, and an ability for a subtest to describe why it skips.

Tested on power9, skylakex, haswell, broadwell, westmere, sandybridge and ivybridge.

May skip/warn on other architectures if metrics are invalid. In particular s390 is untested, but its expressions are trivial. The untested architectures with expressions are power8, cascadelakex, tremontx, skylake, jaketown, ivytown and variants of haswell and broadwell.

v3. addresses review comments from John Garry <[email protected]>, Jiri Olsa <[email protected]> and Arnaldo Carvalho de Melo <[email protected]>.
v2. changes the commit message as event parsing errors no longer cause the test to fail.

Cc Adrian Hunter <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jin Yao <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc John Garry <[email protected]>
Cc Kajol Jain <[email protected]>
Cc Kan Liang <[email protected]>
Cc Leo Yan <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Paul Clarke <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephane Eranian <[email protected]>
[ split from a larger patch ]
---
 tools/perf/tests/builtin-test.c |   7 ++
 tools/perf/tests/pmu-events.c   | 168 ++++++++++++++++++++++++++++++--
 tools/perf/tests/tests.h        |   3 +
 3 files changed, 172 insertions(+), 6 deletions(-)
tools/perf/tests/builtin-test.c +7 -0
index baee735e6aa5..9553f8061772 100644
@@ -75,6 +75,13 @@ static struct test generic_tests[] = {
 	{
 		.desc = "PMU events",
 		.func = test__pmu_events,
+		.subtest = {
+			.skip_if_fail	= false,
+			.get_nr		= test__pmu_events_subtest_get_nr,
+			.get_desc	= test__pmu_events_subtest_get_desc,
+			.skip_reason	= test__pmu_events_subtest_skip_reason,
+		},
+
 	},
 	{
 		.desc = "DSO data read",
tools/perf/tests/pmu-events.c +6 -6
index d64261da8bf7..e21f0addcfbb 100644
@@ -8,6 +8,9 @@
 #include <linux/zalloc.h>
 #include "debug.h"
 #include "../pmu-events/pmu-events.h"
+#include "util/evlist.h"
+#include "util/expr.h"
+#include "util/parse-events.h"
 
 struct perf_pmu_test_event {
 	struct pmu_event event;
@@ -144,7 +147,7 @@ static struct pmu_events_map *__test_pmu_get_events_map(void)
 }
 
 /* Verify generated events from pmu-events.c is as expected */
-static int __test_pmu_event_table(void)
+static int test_pmu_event_table(void)
 {
 	struct pmu_events_map *map = __test_pmu_get_events_map();
 	struct pmu_event *table;
@@ -347,14 +350,11 @@ static int __test__pmu_event_aliases(char *pmu_name, int *count)
 	return res;
 }
 
-int test__pmu_events(struct test *test __maybe_unused,
-		     int subtest __maybe_unused)
+
+static int test_aliases(void)
 {
 	struct perf_pmu *pmu = NULL;
 
-	if (__test_pmu_event_table())
-		return -1;
-
 	while ((pmu = perf_pmu__scan(pmu)) != NULL) {
 		int count = 0;
 
@@ -377,3 +377,159 @@ int test__pmu_events(struct test *test __maybe_unused,
 

tools/perf/tests/pmu-events.c int test__pmu_events(struct test *test __maybe_unused, · line 387

Quoted 10 lines
tools/perf/tests/pmu-events.c +8 -0
 	return 0;
 }
+
+static bool is_number(const char *str)
+{
+	char *end_ptr;
+
+	strtod(str, &end_ptr);
+	return end_ptr != str;
+}

So, this breaks in some systems:

cc1: warnings being treated as errors
tests/pmu-events.c: In function 'is_number':
tests/pmu-events.c:385: error: ignoring return value of 'strtod', declared with attribute warn_unused_result
mv: cannot stat `/tmp/build/perf/tests/.pmu-events.o.tmp': No such file or director

So I'm changing it to verify the result of strtod() which is, humm, interesting, please check:

tools/perf/tests/pmu-events.c +6 -1
index 3de59564deb0..6c58c3a89e6b 100644
@@ -1,4 +1,5 @@
 // SPDX-License-Identifier: GPL-2.0
+#include "math.h"
 #include "parse-events.h"
 #include "pmu.h"
 #include "tests.h"
@@ -381,8 +382,12 @@ static int test_aliases(void)
 static bool is_number(const char *str)
 {
 	char *end_ptr;
+	double v;
 
-	strtod(str, &end_ptr);
+	errno = 0;
+	v = strtod(str, &end_ptr);
+	if ((errno == ERANGE && (v == HUGE_VAL || v == -HUGE_VAL)) || (errno != 0 && v == 0.0))
+		return false;
 	return end_ptr != str;
 }
 
author reviewer
#8 lore raw

Re: [PATCH v3 6/7] perf test: Improve pmu event metric testing

Quoted 120 lines · Arnaldo Carvalho de Melo

Em Fri, May 15, 2020 at 03:17:31PM -0700, Ian Rogers escreveu:

Quoted 105 lines

Break pmu-events test into 2 and add a test to verify that all pmu metric expressions simply parse. Try to parse all metric ids/events, skip/warn if metrics for the current architecture fail to parse. To support warning for a skip, and an ability for a subtest to describe why it skips.

Tested on power9, skylakex, haswell, broadwell, westmere, sandybridge and ivybridge.

May skip/warn on other architectures if metrics are invalid. In particular s390 is untested, but its expressions are trivial. The untested architectures with expressions are power8, cascadelakex, tremontx, skylake, jaketown, ivytown and variants of haswell and broadwell.

v3. addresses review comments from John Garry <[email protected]>, Jiri Olsa <[email protected]> and Arnaldo Carvalho de Melo <[email protected]>.
v2. changes the commit message as event parsing errors no longer cause the test to fail.

Cc Adrian Hunter <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jin Yao <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc John Garry <[email protected]>
Cc Kajol Jain <[email protected]>
Cc Kan Liang <[email protected]>
Cc Leo Yan <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Paul Clarke <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephane Eranian <[email protected]>
[ split from a larger patch ]
---
 tools/perf/tests/builtin-test.c |   7 ++
 tools/perf/tests/pmu-events.c   | 168 ++++++++++++++++++++++++++++++--
 tools/perf/tests/tests.h        |   3 +
 3 files changed, 172 insertions(+), 6 deletions(-)
tools/perf/tests/builtin-test.c +7 -0
index baee735e6aa5..9553f8061772 100644
@@ -75,6 +75,13 @@ static struct test generic_tests[] = {
      {
              .desc = "PMU events",
              .func = test__pmu_events,
+             .subtest = {
+                     .skip_if_fail   = false,
+                     .get_nr         = test__pmu_events_subtest_get_nr,
+                     .get_desc       = test__pmu_events_subtest_get_desc,
+                     .skip_reason    = test__pmu_events_subtest_skip_reason,
+             },
+
      },
      {
              .desc = "DSO data read",
tools/perf/tests/pmu-events.c +6 -6
index d64261da8bf7..e21f0addcfbb 100644
@@ -8,6 +8,9 @@
 #include <linux/zalloc.h>
 #include "debug.h"
 #include "../pmu-events/pmu-events.h"
+#include "util/evlist.h"
+#include "util/expr.h"
+#include "util/parse-events.h"
 
 struct perf_pmu_test_event {
      struct pmu_event event;
@@ -144,7 +147,7 @@ static struct pmu_events_map *__test_pmu_get_events_map(void)
 }
 
 /* Verify generated events from pmu-events.c is as expected */
-static int __test_pmu_event_table(void)
+static int test_pmu_event_table(void)
 {
      struct pmu_events_map *map = __test_pmu_get_events_map();
      struct pmu_event *table;
@@ -347,14 +350,11 @@ static int __test__pmu_event_aliases(char *pmu_name, int *count)
      return res;
 }
 
-int test__pmu_events(struct test *test __maybe_unused,
-                  int subtest __maybe_unused)
+
+static int test_aliases(void)
 {
      struct perf_pmu *pmu = NULL;
 
-     if (__test_pmu_event_table())
-             return -1;
-
      while ((pmu = perf_pmu__scan(pmu)) != NULL) {
              int count = 0;
 
@@ -377,3 +377,159 @@ int test__pmu_events(struct test *test __maybe_unused,
 

tools/perf/tests/pmu-events.c int test__pmu_events(struct test *test __maybe_unused, · line 387

Quoted 10 lines
tools/perf/tests/pmu-events.c +8 -0
      return 0;
 }
+
+static bool is_number(const char *str)
+{
+     char *end_ptr;
+
+     strtod(str, &end_ptr);
+     return end_ptr != str;
+}

So, this breaks in some systems:

Quoted 7 lines

cc1: warnings being treated as errors
tests/pmu-events.c: In function 'is_number':
tests/pmu-events.c:385: error: ignoring return value of 'strtod', declared with attribute warn_unused_result
mv: cannot stat `/tmp/build/perf/tests/.pmu-events.o.tmp': No such file or director

So I'm changing it to verify the result of strtod() which is, humm, interesting, please check:

Thanks Arnaldo and sorry for the difficulty. This looks like a good fix.

Quoted 11 lines
tools/perf/tests/pmu-events.c +1 -0
index 3de59564deb0..6c58c3a89e6b 100644
@@ -1,4 +1,5 @@
 // SPDX-License-Identifier: GPL-2.0
+#include "math.h"
 #include "parse-events.h"
 #include "pmu.h"
 #include "tests.h"

tools/perf/tests/pmu-events.c static int test_aliases(void) · line 389

Quoted 9 lines
tools/perf/tests/pmu-events.c +4 -1
@@ -381,8 +382,12 @@ static int test_aliases(void)
 static bool is_number(const char *str)
 {
        char *end_ptr;
+       double v;
 
-       strtod(str, &end_ptr);
+       errno = 0;
+       v = strtod(str, &end_ptr);
+       if ((errno == ERANGE && (v == HUGE_VAL || v == -HUGE_VAL)) || (errno != 0 && v == 0.0))

errno can either be 0 or ERANGE here, but we test both. Perhaps use errno != 0 for both cases as the man page notes suggest doing this. The tests using v are necessary to avoid the unused result, but presumably any errno case should return false here? I guess testing that is redundant as the return below will catch it. Perhaps this should be:

errno = 0;
v = strtod(str, &end_ptr);
(void)v; /* We don't care for the value of the double, just that it converts. Avoid unused result warnings. */
return errno == 0 && end_ptr != str;

Thanks,
Ian

Quoted 4 lines
tools/perf/tests/pmu-events.c +1 -0
+               return false;
        return end_ptr != str;
 }
 
#9 lore raw

Re: [PATCH v3 6/7] perf test: Improve pmu event metric testing

Em Tue, May 19, 2020 at 01:15:41PM -0700, Ian Rogers escreveu:

Quoted 160 lines
Quoted 120 lines · Arnaldo Carvalho de Melo

Em Fri, May 15, 2020 at 03:17:31PM -0700, Ian Rogers escreveu:

Quoted 105 lines

Break pmu-events test into 2 and add a test to verify that all pmu metric expressions simply parse. Try to parse all metric ids/events, skip/warn if metrics for the current architecture fail to parse. To support warning for a skip, and an ability for a subtest to describe why it skips.

Tested on power9, skylakex, haswell, broadwell, westmere, sandybridge and ivybridge.

May skip/warn on other architectures if metrics are invalid. In particular s390 is untested, but its expressions are trivial. The untested architectures with expressions are power8, cascadelakex, tremontx, skylake, jaketown, ivytown and variants of haswell and broadwell.

v3. addresses review comments from John Garry <[email protected]>, Jiri Olsa <[email protected]> and Arnaldo Carvalho de Melo <[email protected]>.
v2. changes the commit message as event parsing errors no longer cause the test to fail.

Cc Adrian Hunter <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jin Yao <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc John Garry <[email protected]>
Cc Kajol Jain <[email protected]>
Cc Kan Liang <[email protected]>
Cc Leo Yan <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Paul Clarke <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephane Eranian <[email protected]>
[ split from a larger patch ]
---
 tools/perf/tests/builtin-test.c |   7 ++
 tools/perf/tests/pmu-events.c   | 168 ++++++++++++++++++++++++++++++--
 tools/perf/tests/tests.h        |   3 +
 3 files changed, 172 insertions(+), 6 deletions(-)
tools/perf/tests/builtin-test.c +7 -0
index baee735e6aa5..9553f8061772 100644
@@ -75,6 +75,13 @@ static struct test generic_tests[] = {
      {
              .desc = "PMU events",
              .func = test__pmu_events,
+             .subtest = {
+                     .skip_if_fail   = false,
+                     .get_nr         = test__pmu_events_subtest_get_nr,
+                     .get_desc       = test__pmu_events_subtest_get_desc,
+                     .skip_reason    = test__pmu_events_subtest_skip_reason,
+             },
+
      },
      {
              .desc = "DSO data read",
tools/perf/tests/pmu-events.c +6 -6
index d64261da8bf7..e21f0addcfbb 100644
@@ -8,6 +8,9 @@
 #include <linux/zalloc.h>
 #include "debug.h"
 #include "../pmu-events/pmu-events.h"
+#include "util/evlist.h"
+#include "util/expr.h"
+#include "util/parse-events.h"
 
 struct perf_pmu_test_event {
      struct pmu_event event;
@@ -144,7 +147,7 @@ static struct pmu_events_map *__test_pmu_get_events_map(void)
 }
 
 /* Verify generated events from pmu-events.c is as expected */
-static int __test_pmu_event_table(void)
+static int test_pmu_event_table(void)
 {
      struct pmu_events_map *map = __test_pmu_get_events_map();
      struct pmu_event *table;
@@ -347,14 +350,11 @@ static int __test__pmu_event_aliases(char *pmu_name, int *count)
      return res;
 }
 
-int test__pmu_events(struct test *test __maybe_unused,
-                  int subtest __maybe_unused)
+
+static int test_aliases(void)
 {
      struct perf_pmu *pmu = NULL;
 
-     if (__test_pmu_event_table())
-             return -1;
-
      while ((pmu = perf_pmu__scan(pmu)) != NULL) {
              int count = 0;
 
@@ -377,3 +377,159 @@ int test__pmu_events(struct test *test __maybe_unused,
 

tools/perf/tests/pmu-events.c int test__pmu_events(struct test *test __maybe_unused, · line 387

Quoted 10 lines
tools/perf/tests/pmu-events.c +8 -0
      return 0;
 }
+
+static bool is_number(const char *str)
+{
+     char *end_ptr;
+
+     strtod(str, &end_ptr);
+     return end_ptr != str;
+}

So, this breaks in some systems:

Quoted 7 lines

cc1: warnings being treated as errors
tests/pmu-events.c: In function 'is_number':
tests/pmu-events.c:385: error: ignoring return value of 'strtod', declared with attribute warn_unused_result
mv: cannot stat `/tmp/build/perf/tests/.pmu-events.o.tmp': No such file or director

So I'm changing it to verify the result of strtod() which is, humm, interesting, please check:

Thanks Arnaldo and sorry for the difficulty. This looks like a good fix.

Quoted 11 lines
tools/perf/tests/pmu-events.c +1 -0
index 3de59564deb0..6c58c3a89e6b 100644
@@ -1,4 +1,5 @@
 // SPDX-License-Identifier: GPL-2.0
+#include "math.h"
 #include "parse-events.h"
 #include "pmu.h"
 #include "tests.h"

tools/perf/tests/pmu-events.c static int test_aliases(void) · line 389

Quoted 9 lines
tools/perf/tests/pmu-events.c +4 -1
@@ -381,8 +382,12 @@ static int test_aliases(void)
 static bool is_number(const char *str)
 {
        char *end_ptr;
+       double v;
 
-       strtod(str, &end_ptr);
+       errno = 0;
+       v = strtod(str, &end_ptr);
+       if ((errno == ERANGE && (v == HUGE_VAL || v == -HUGE_VAL)) || (errno != 0 && v == 0.0))

errno can either be 0 or ERANGE here, but we test both. Perhaps use errno != 0 for both cases as the man page notes suggest doing this. The tests using v are necessary to avoid the unused result, but presumably any errno case should return false here? I guess testing that is redundant as the return below will catch it. Perhaps this should be:

Quoted 5 lines

errno = 0;
v = strtod(str, &end_ptr);
(void)v; /* We don't care for the value of the double, just that it converts. Avoid unused result warnings. */
return errno == 0 && end_ptr != str;

Ok, I'll try that one.

- Arnaldo

Quoted 7 lines

Thanks,
Ian

Quoted 4 lines
tools/perf/tests/pmu-events.c +1 -0
+               return false;
        return end_ptr != str;
 }
 
Signature · 1 line
- Arnaldo
#10 lore raw

Re: [PATCH v3 6/7] perf test: Improve pmu event metric testing

Em Tue, May 19, 2020 at 10:15:48PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 8 lines

Em Tue, May 19, 2020 at 01:15:41PM -0700, Ian Rogers escreveu:

Quoted 7 lines

On Tue, May 19, 2020 at 12:06 PM Arnaldo Carvalho de Melo errno != 0 for both cases as the man page notes suggest doing this. The tests using v are necessary to avoid the unused result, but presumably any errno case should return false here? I guess testing that is redundant as the return below will catch it. Perhaps this should be:

Quoted 7 lines
Quoted 5 lines

errno = 0;
v = strtod(str, &end_ptr);
(void)v; /* We don't care for the value of the double, just that it converts. Avoid unused result warnings. */
return errno == 0 && end_ptr != str;

Ok, I'll try that one.

What I have is in my tmp.perf/core branch, this and the hashmap.h __WORDSIZE fixups, with those all the alpine Linux containers, that use Musl libc passed, waiting for the remaining 80 other containers to finish, now lots of these containers build with and without NO_LIBBPF=1 to make sure we test both variants,

- Arnaldo

author reviewer
#11 lore raw

PATCH v3 7/7 perf expr: Migrate expr ids table to a hashmap

In reply to Ian Rogers

Use a hashmap between a char* string and a double* value. While bpf's hashmap entries are size_t in size, we can't guarantee sizeof(size_t) >= sizeof(double). Avoid a memory allocation when gathering ids by making 0.0 a special value encoded as NULL.

Original map suggestion by Andi Kleen:
https://lore.kernel.org/lkml/[email protected]/
and seconded by Jiri Olsa:
https://lore.kernel.org/lkml/20200423112915.GH1136647@krava/

---
 tools/perf/tests/expr.c       |  44 ++++++------
 tools/perf/tests/pmu-events.c |  25 +++----
 tools/perf/util/expr.c        | 129 +++++++++++++++++++---------------
 tools/perf/util/expr.h        |  26 +++----
 tools/perf/util/expr.y        |  22 +-----
 tools/perf/util/metricgroup.c |  92 +++++++++++-------------
 tools/perf/util/stat-shadow.c |  49 ++++++++-----
 7 files changed, 197 insertions(+), 190 deletions(-)
tools/perf/tests/expr.c +23 -21
index 3f742612776a..13350c500e34 100644
@@ -19,15 +19,13 @@ static int test(struct expr_parse_ctx *ctx, const char *e, double val2)
 int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused)
 {
 	const char *p;
-	const char **other;
-	double val;
-	int i, ret;
+	double val, *val_ptr;
+	int ret;
 	struct expr_parse_ctx ctx;
-	int num_other;
 
 	expr__ctx_init(&ctx);
-	expr__add_id(&ctx, "FOO", 1);
-	expr__add_id(&ctx, "BAR", 2);
+	expr__add_id(&ctx, strdup("FOO"), 1);
+	expr__add_id(&ctx, strdup("BAR"), 2);
 
 	ret = test(&ctx, "1+1", 2);
 	ret |= test(&ctx, "FOO+BAR", 3);
@@ -52,25 +50,29 @@ int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused)
 	ret = expr__parse(&val, &ctx, p, 1);
 	TEST_ASSERT_VAL("missing operand", ret == -1);
 
+	expr__ctx_clear(&ctx);
 	TEST_ASSERT_VAL("find other",
-			expr__find_other("FOO + BAR + BAZ + BOZO", "FOO", &other, &num_other, 1) == 0);
-	TEST_ASSERT_VAL("find other", num_other == 3);
-	TEST_ASSERT_VAL("find other", !strcmp(other[0], "BAR"));
-	TEST_ASSERT_VAL("find other", !strcmp(other[1], "BAZ"));
-	TEST_ASSERT_VAL("find other", !strcmp(other[2], "BOZO"));
-	TEST_ASSERT_VAL("find other", other[3] == NULL);
+			expr__find_other("FOO + BAR + BAZ + BOZO", "FOO",
+					 &ctx, 1) == 0);
+	TEST_ASSERT_VAL("find other", hashmap__size(&ctx.ids) == 3);
+	TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "BAR",
+						    (void **)&val_ptr));
+	TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "BAZ",
+						    (void **)&val_ptr));
+	TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "BOZO",
+						    (void **)&val_ptr));
 
+	expr__ctx_clear(&ctx);
 	TEST_ASSERT_VAL("find other",
-			expr__find_other("EVENT1\\,param\\=?@ + EVENT2\\,param\\=?@", NULL,
-				   &other, &num_other, 3) == 0);
-	TEST_ASSERT_VAL("find other", num_other == 2);
-	TEST_ASSERT_VAL("find other", !strcmp(other[0], "EVENT1,param=3/"));
-	TEST_ASSERT_VAL("find other", !strcmp(other[1], "EVENT2,param=3/"));
-	TEST_ASSERT_VAL("find other", other[2] == NULL);
+			expr__find_other("EVENT1\\,param\\=?@ + EVENT2\\,param\\=?@",
+					 NULL, &ctx, 3) == 0);
+	TEST_ASSERT_VAL("find other", hashmap__size(&ctx.ids) == 2);
+	TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "EVENT1,param=3/",
+						    (void **)&val_ptr));
+	TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "EVENT2,param=3/",
+						    (void **)&val_ptr));
 
-	for (i = 0; i < num_other; i++)
-		zfree(&other[i]);
-	free((void *)other);
+	expr__ctx_clear(&ctx);
 
 	return 0;
 }
tools/perf/tests/pmu-events.c +13 -12
index e21f0addcfbb..3de59564deb0 100644
@@ -433,8 +433,6 @@ static int test_parsing(void)
 	struct pmu_events_map *map;
 	struct pmu_event *pe;
 	int i, j, k;
-	const char **ids;
-	int idnum;
 	int ret = 0;
 	struct expr_parse_ctx ctx;
 	double result;
@@ -446,29 +444,34 @@ static int test_parsing(void)
 			break;
 		j = 0;
 		for (;;) {
+			struct hashmap_entry *cur;
+			size_t bkt;
+
 			pe = &map->table[j++];
 			if (!pe->name && !pe->metric_group && !pe->metric_name)
 				break;
 			if (!pe->metric_expr)
 				continue;
-			if (expr__find_other(pe->metric_expr, NULL,
-						&ids, &idnum, 0) < 0) {
+			expr__ctx_init(&ctx);
+			if (expr__find_other(pe->metric_expr, NULL, &ctx, 0)
+				  < 0) {
 				expr_failure("Parse other failed", map, pe);
 				ret++;
 				continue;
 			}
-			expr__ctx_init(&ctx);
 
 			/*
 			 * Add all ids with a made up value. The value may
 			 * trigger divide by zero when subtracted and so try to
 			 * make them unique.
 			 */
-			for (k = 0; k < idnum; k++)
-				expr__add_id(&ctx, ids[k], k + 1);
+			k = 1;
+			hashmap__for_each_entry((&ctx.ids), cur, bkt)
+				expr__add_id(&ctx, strdup(cur->key), k++);
 
-			for (k = 0; k < idnum; k++) {
-				if (check_parse_id(ids[k], map == cpus_map, pe))
+			hashmap__for_each_entry((&ctx.ids), cur, bkt) {
+				if (check_parse_id(cur->key, map == cpus_map,
+						   pe))
 					ret++;
 			}
 
@@ -476,9 +479,7 @@ static int test_parsing(void)
 				expr_failure("Parse failed", map, pe);
 				ret++;
 			}
-			for (k = 0; k < idnum; k++)
-				zfree(&ids[k]);
-			free(ids);
+			expr__ctx_clear(&ctx);
 		}
 	}
 	/* TODO: fail when not ok */
tools/perf/util/expr.c +72 -57

158 lines not on this page · raw message

tools/perf/util/expr.h +13 -13
index 40fc452b0f2b..d60a8feaf50b 100644
@@ -2,17 +2,14 @@
 #ifndef PARSE_CTX_H
 #define PARSE_CTX_H 1
 
-#define EXPR_MAX_OTHER 64
-#define MAX_PARSE_ID EXPR_MAX_OTHER
-
-struct expr_parse_id {
-	const char *name;
-	double val;
-};
+#ifdef HAVE_LIBBPF_SUPPORT
+#include <bpf/hashmap.h>
+#else
+#include "hashmap.h"
+#endif
 
 struct expr_parse_ctx {
-	int num_ids;
-	struct expr_parse_id ids[MAX_PARSE_ID];
+	struct hashmap ids;
 };
 
 struct expr_scanner_ctx {
@@ -21,9 +18,12 @@ struct expr_scanner_ctx {
 };
 
 void expr__ctx_init(struct expr_parse_ctx *ctx);
-void expr__add_id(struct expr_parse_ctx *ctx, const char *id, double val);
-int expr__parse(double *final_val, struct expr_parse_ctx *ctx, const char *expr, int runtime);
-int expr__find_other(const char *expr, const char *one, const char ***other,
-		int *num_other, int runtime);
+void expr__ctx_clear(struct expr_parse_ctx *ctx);
+int expr__add_id(struct expr_parse_ctx *ctx, const char *id, double val);
+int expr__get_id(struct expr_parse_ctx *ctx, const char *id, double *val_ptr);
+int expr__parse(double *final_val, struct expr_parse_ctx *ctx,
+		const char *expr, int runtime);
+int expr__find_other(const char *expr, const char *one,
+		struct expr_parse_ctx *ids, int runtime);
 
 #endif
tools/perf/util/expr.y +2 -20
index 21e82a1e11a2..ec5a48bf5f34 100644
@@ -46,19 +46,6 @@ static void expr_error(double *final_val __maybe_unused,
 	pr_debug("%s\n", s);
 }
 
-static int lookup_id(struct expr_parse_ctx *ctx, char *id, double *val)
-{
-	int i;
-
-	for (i = 0; i < ctx->num_ids; i++) {
-		if (!strcasecmp(ctx->ids[i].name, id)) {
-			*val = ctx->ids[i].val;
-			return 0;
-		}
-	}
-	return -1;
-}
-
 %}
 %%
 
@@ -72,12 +59,7 @@ all_other: all_other other
 
 other: ID
 {
-	if (ctx->num_ids + 1 >= EXPR_MAX_OTHER) {
-		pr_err("failed: way too many variables");
-		YYABORT;
-	}
-
-	ctx->ids[ctx->num_ids++].name = $1;
+	expr__add_id(ctx, $1, 0.0);
 }
 |
 MIN | MAX | IF | ELSE | SMT_ON | NUMBER | '|' | '^' | '&' | '-' | '+' | '*' | '/' | '%' | '(' | ')' | ','
@@ -92,7 +74,7 @@ if_expr:
 	;
 
 expr:	  NUMBER
-	| ID			{ if (lookup_id(ctx, $1, &$$) < 0) {
+	| ID			{ if (expr__get_id(ctx, $1, &$$)) {
 					pr_debug("%s not found\n", $1);
 					YYABORT;
 				  }
tools/perf/util/metricgroup.c +42 -50

187 lines not on this page · raw message

tools/perf/util/stat-shadow.c +32 -17

116 lines not on this page · raw message

Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
#12 lore raw

Re: [PATCH v3 7/7] perf expr: Migrate expr ids table to a hashmap

Em Fri, May 15, 2020 at 03:17:32PM -0700, Ian Rogers escreveu:

Quoted 9 lines

Use a hashmap between a char* string and a double* value. While bpf's hashmap entries are size_t in size, we can't guarantee sizeof(size_t) >= sizeof(double). Avoid a memory allocation when gathering ids by making 0.0 a special value encoded as NULL.

Original map suggestion by Andi Kleen:
https://lore.kernel.org/lkml/[email protected]/
and seconded by Jiri Olsa:
https://lore.kernel.org/lkml/20200423112915.GH1136647@krava/

I'm having trouble here when building it with:

make -C tools/perf O=/tmp/build/perf

    CC       /tmp/build/perf/tests/expr.o
    INSTALL  trace_plugins
    CC       /tmp/build/perf/util/metricgroup.o
  In file included from /home/acme/git/perf/tools/lib/bpf/hashmap.h:18,
                   from /home/acme/git/perf/tools/perf/util/expr.h:6,
                   from tests/expr.c:3:
  /home/acme/git/perf/tools/lib/bpf/libbpf_internal.h:63: error: "pr_info" redefined [-Werror]
     63 | #define pr_info(fmt, ...) __pr(LIBBPF_INFO, fmt, ##__VA_ARGS__)
        |

In file included from tests/expr.c:2:
/home/acme/git/perf/tools/perf/util/debug.h:24: note: this is the location of the previous definition

It looks like libbpf's hashmap.h is being used instead of the one in tools/perf/util/, yeah, as intended, but then since I don't have the fixes you added to the BPF tree, the build fails, if I instead unconditionally use

#include "util/hashmap.h"

It works. Please ack.

I.e. with the patch below, further tests:

[acme@five perf]$ perf -vv | grep -i bpf
bpf: [ on ] # HAVE_LIBBPF_SUPPORT
[acme@five perf]$ nm ~/bin/perf | grep -i libbpf_ | wc -l 39
[acme@five perf]$ nm ~/bin/perf | grep -i hashmap_ | wc -l 17
[acme@five perf]$

Explicitely building without LIBBPF:

[acme@five perf]$ perf -vv | grep -i bpf
bpf: [ OFF ] # HAVE_LIBBPF_SUPPORT
[acme@five perf]$
[acme@five perf]$ nm ~/bin/perf | grep -i libbpf_ | wc -l 0
[acme@five perf]$ nm ~/bin/perf | grep -i hashmap_ | wc -l 9
[acme@five perf]$

Works,

- Arnaldo

tools/perf/util/expr.h +8 -5
index d60a8feaf50b..8a2c1074f90f 100644
@@ -2,11 +2,14 @@
 #ifndef PARSE_CTX_H
 #define PARSE_CTX_H 1
 
-#ifdef HAVE_LIBBPF_SUPPORT
-#include <bpf/hashmap.h>
-#else
-#include "hashmap.h"
-#endif
+// There are fixes that need to land upstream before we can use libbpf's headers,
+// for now use our copy unconditionally, since the data structures at this point
+// are exactly the same, no problem.
+//#ifdef HAVE_LIBBPF_SUPPORT
+//#include <bpf/hashmap.h>
+//#else
+#include "util/hashmap.h"
+//#endif
 
 struct expr_parse_ctx {
 	struct hashmap ids;
author reviewer
#13 lore raw

Re: [PATCH v3 7/7] perf expr: Migrate expr ids table to a hashmap

Quoted 58 lines · Arnaldo Carvalho de Melo

Em Fri, May 15, 2020 at 03:17:32PM -0700, Ian Rogers escreveu:

Quoted 9 lines

Use a hashmap between a char* string and a double* value. While bpf's hashmap entries are size_t in size, we can't guarantee sizeof(size_t) >= sizeof(double). Avoid a memory allocation when gathering ids by making 0.0 a special value encoded as NULL.

Original map suggestion by Andi Kleen:
https://lore.kernel.org/lkml/[email protected]/
and seconded by Jiri Olsa:
https://lore.kernel.org/lkml/20200423112915.GH1136647@krava/

I'm having trouble here when building it with:

make -C tools/perf O=/tmp/build/perf

    CC       /tmp/build/perf/tests/expr.o
    INSTALL  trace_plugins
    CC       /tmp/build/perf/util/metricgroup.o
  In file included from /home/acme/git/perf/tools/lib/bpf/hashmap.h:18,
                   from /home/acme/git/perf/tools/perf/util/expr.h:6,
                   from tests/expr.c:3:
  /home/acme/git/perf/tools/lib/bpf/libbpf_internal.h:63: error: "pr_info" redefined [-Werror]
     63 | #define pr_info(fmt, ...) __pr(LIBBPF_INFO, fmt, ##__VA_ARGS__)
        |

In file included from tests/expr.c:2:
/home/acme/git/perf/tools/perf/util/debug.h:24: note: this is the location of the previous definition

It looks like libbpf's hashmap.h is being used instead of the one in tools/perf/util/, yeah, as intended, but then since I don't have the fixes you added to the BPF tree, the build fails, if I instead unconditionally use

#include "util/hashmap.h"

It works. Please ack.

I.e. with the patch below, further tests:

[acme@five perf]$ perf -vv | grep -i bpf
bpf: [ on ] # HAVE_LIBBPF_SUPPORT
[acme@five perf]$ nm ~/bin/perf | grep -i libbpf_ | wc -l 39
[acme@five perf]$ nm ~/bin/perf | grep -i hashmap_ | wc -l 17
[acme@five perf]$

Explicitely building without LIBBPF:

[acme@five perf]$ perf -vv | grep -i bpf
bpf: [ OFF ] # HAVE_LIBBPF_SUPPORT
[acme@five perf]$
[acme@five perf]$ nm ~/bin/perf | grep -i libbpf_ | wc -l 0
[acme@five perf]$ nm ~/bin/perf | grep -i hashmap_ | wc -l 9
[acme@five perf]$

Quoted 3 lines

Works,

- Arnaldo

Hi Arnaldo,

this build issue sounds like this patch is missing:
https://lore.kernel.org/lkml/[email protected]/
The commit message there could have explicitly said having this #include causes the conflicting definitions between perf's debug.h and libbpf_internal.h's definitions of pr_info, etc.

Let me know how else to help and sorry for the confusion. Thanks, Ian

Quoted 24 lines
tools/perf/util/expr.h +8 -5
index d60a8feaf50b..8a2c1074f90f 100644
@@ -2,11 +2,14 @@
 #ifndef PARSE_CTX_H
 #define PARSE_CTX_H 1
 
-#ifdef HAVE_LIBBPF_SUPPORT
-#include <bpf/hashmap.h>
-#else
-#include "hashmap.h"
-#endif
+// There are fixes that need to land upstream before we can use libbpf's headers,
+// for now use our copy unconditionally, since the data structures at this point
+// are exactly the same, no problem.
+//#ifdef HAVE_LIBBPF_SUPPORT
+//#include <bpf/hashmap.h>
+//#else
+#include "util/hashmap.h"
+//#endif
 
 struct expr_parse_ctx {
        struct hashmap ids;
#14 lore raw

Re: [PATCH v3 7/7] perf expr: Migrate expr ids table to a hashmap

Em Mon, May 18, 2020 at 09:03:45AM -0700, Ian Rogers escreveu:

Quoted 64 lines
Quoted 58 lines · Arnaldo Carvalho de Melo

Em Fri, May 15, 2020 at 03:17:32PM -0700, Ian Rogers escreveu:

Quoted 9 lines

Use a hashmap between a char* string and a double* value. While bpf's hashmap entries are size_t in size, we can't guarantee sizeof(size_t) >= sizeof(double). Avoid a memory allocation when gathering ids by making 0.0 a special value encoded as NULL.

Original map suggestion by Andi Kleen:
https://lore.kernel.org/lkml/[email protected]/
and seconded by Jiri Olsa:
https://lore.kernel.org/lkml/20200423112915.GH1136647@krava/

I'm having trouble here when building it with:

make -C tools/perf O=/tmp/build/perf

    CC       /tmp/build/perf/tests/expr.o
    INSTALL  trace_plugins
    CC       /tmp/build/perf/util/metricgroup.o
  In file included from /home/acme/git/perf/tools/lib/bpf/hashmap.h:18,
                   from /home/acme/git/perf/tools/perf/util/expr.h:6,
                   from tests/expr.c:3:
  /home/acme/git/perf/tools/lib/bpf/libbpf_internal.h:63: error: "pr_info" redefined [-Werror]
     63 | #define pr_info(fmt, ...) __pr(LIBBPF_INFO, fmt, ##__VA_ARGS__)
        |

In file included from tests/expr.c:2:
/home/acme/git/perf/tools/perf/util/debug.h:24: note: this is the location of the previous definition

It looks like libbpf's hashmap.h is being used instead of the one in tools/perf/util/, yeah, as intended, but then since I don't have the fixes you added to the BPF tree, the build fails, if I instead unconditionally use

#include "util/hashmap.h"

It works. Please ack.

I.e. with the patch below, further tests:

[acme@five perf]$ perf -vv | grep -i bpf
bpf: [ on ] # HAVE_LIBBPF_SUPPORT
[acme@five perf]$ nm ~/bin/perf | grep -i libbpf_ | wc -l 39
[acme@five perf]$ nm ~/bin/perf | grep -i hashmap_ | wc -l 17
[acme@five perf]$

Explicitely building without LIBBPF:

[acme@five perf]$ perf -vv | grep -i bpf
bpf: [ OFF ] # HAVE_LIBBPF_SUPPORT
[acme@five perf]$
[acme@five perf]$ nm ~/bin/perf | grep -i libbpf_ | wc -l 0
[acme@five perf]$ nm ~/bin/perf | grep -i hashmap_ | wc -l 9
[acme@five perf]$

Quoted 3 lines

Works,

- Arnaldo

Quoted 7 lines

Hi Arnaldo,

this build issue sounds like this patch is missing:
https://lore.kernel.org/lkml/[email protected]/
The commit message there could have explicitly said having this #include causes the conflicting definitions between perf's debug.h and libbpf_internal.h's definitions of pr_info, etc.

yeah, understood, but I'm not processing patches for tools/lib/bpf/, Daniel is, I'll only get that one later, then we can go back to the way you structured it. Just an extra bit of confusion in this process ;-)

Quoted 28 lines

Let me know how else to help and sorry for the confusion. Thanks, Ian

Quoted 24 lines
tools/perf/util/expr.h +8 -5
index d60a8feaf50b..8a2c1074f90f 100644
@@ -2,11 +2,14 @@
 #ifndef PARSE_CTX_H
 #define PARSE_CTX_H 1
 
-#ifdef HAVE_LIBBPF_SUPPORT
-#include <bpf/hashmap.h>
-#else
-#include "hashmap.h"
-#endif
+// There are fixes that need to land upstream before we can use libbpf's headers,
+// for now use our copy unconditionally, since the data structures at this point
+// are exactly the same, no problem.
+//#ifdef HAVE_LIBBPF_SUPPORT
+//#include <bpf/hashmap.h>
+//#else
+#include "util/hashmap.h"
+//#endif
 
 struct expr_parse_ctx {
        struct hashmap ids;
Signature · 1 line
- Arnaldo
#15 lore raw

Re: [PATCH v3 7/7] perf expr: Migrate expr ids table to a hashmap

Em Mon, May 18, 2020 at 01:06:48PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 11 lines

Em Mon, May 18, 2020 at 09:03:45AM -0700, Ian Rogers escreveu:

Quoted 6 lines · Arnaldo Carvalho de Melo

this build issue sounds like this patch is missing:
https://lore.kernel.org/lkml/[email protected]/
The commit message there could have explicitly said having this #include causes the conflicting definitions between perf's debug.h and libbpf_internal.h's definitions of pr_info, etc.

yeah, understood, but I'm not processing patches for tools/lib/bpf/, Daniel is, I'll only get that one later, then we can go back to the way you structured it. Just an extra bit of confusion in this process ;-)

So, thiis is failing on all alpine Linux containers:

CC /tmp/build/perf/util/metricgroup.o
CC /tmp/build/perf/util/header.o
In file included from util/metricgroup.c:25:0:
/git/linux/tools/lib/api/fs/fs.h:16:0: error: "FS" redefined [-Werror]
#define FS(name) \
^
In file included from /git/linux/tools/perf/util/hashmap.h:16:0,

                 from util/expr.h:11,
                 from util/metricgroup.c:14:
/usr/include/bits/reg.h:28:0: note: this is the location of the previous definition
 #define FS     25
 ^
  CC       /tmp/build/perf/util/callchain.o
  CC       /tmp/build/perf/util/values.o
  CC       /tmp/build/perf/util/debug.o
  CC       /tmp/build/perf/util/fncache.o

cc1: all warnings being treated as errors
mv: can't rename '/tmp/build/perf/util/.metricgroup.o.tmp': No such file or directory /git/linux/tools/build/Makefile.build:96: recipe for target '/tmp/build/perf/util/metricgroup.o' failed

I'll check that soon,

- Arnaldo

author reviewer
#16 lore raw

Re: [PATCH v3 7/7] perf expr: Migrate expr ids table to a hashmap

Quoted 32 lines · Arnaldo Carvalho de Melo

Em Mon, May 18, 2020 at 01:06:48PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 11 lines

Em Mon, May 18, 2020 at 09:03:45AM -0700, Ian Rogers escreveu:

Quoted 6 lines · Arnaldo Carvalho de Melo

this build issue sounds like this patch is missing:
https://lore.kernel.org/lkml/[email protected]/
The commit message there could have explicitly said having this #include causes the conflicting definitions between perf's debug.h and libbpf_internal.h's definitions of pr_info, etc.

yeah, understood, but I'm not processing patches for tools/lib/bpf/, Daniel is, I'll only get that one later, then we can go back to the way you structured it. Just an extra bit of confusion in this process ;-)

So, thiis is failing on all alpine Linux containers:

CC /tmp/build/perf/util/metricgroup.o
CC /tmp/build/perf/util/header.o
In file included from util/metricgroup.c:25:0:
/git/linux/tools/lib/api/fs/fs.h:16:0: error: "FS" redefined [-Werror]
#define FS(name) \
^
In file included from /git/linux/tools/perf/util/hashmap.h:16:0,

                 from util/expr.h:11,
                 from util/metricgroup.c:14:
/usr/include/bits/reg.h:28:0: note: this is the location of the previous definition
 #define FS     25
 ^
  CC       /tmp/build/perf/util/callchain.o
  CC       /tmp/build/perf/util/values.o
  CC       /tmp/build/perf/util/debug.o
  CC       /tmp/build/perf/util/fncache.o
Quoted 8 lines

cc1: all warnings being treated as errors
mv: can't rename '/tmp/build/perf/util/.metricgroup.o.tmp': No such file or directory /git/linux/tools/build/Makefile.build:96: recipe for target '/tmp/build/perf/util/metricgroup.o' failed

I'll check that soon,

- Arnaldo

I had some issues here too:
https://lore.kernel.org/lkml/CAEf4BzYxTTND7T7X0dLr2CbkEvUuKtarOeoJYYROefij+qds0w@mail.gmail.com/
The only reason for the bits/reg.h inclusion is for __WORDSIZE for the hash_bits operation. As shown below:

#ifdef __GLIBC__
#include <bits/wordsize.h>
#else
#include <bits/reg.h>
#endif
static inline size_t hash_bits(size_t h, int bits) {
/* shuffle bits and return requested number of upper bits */ return (h * 11400714819323198485llu) >> (__WORDSIZE - bits); }

It'd be possible to change the definition of hash_bits and remove the #includes by:

static inline size_t hash_bits(size_t h, int bits) {
/* shuffle bits and return requested number of upper bits */ #ifdef __LP64__
int shift = 64 - bits;
#else
int shift = 32 - bits;
#endif
return (h * 11400714819323198485llu) >> shift;
}

Others may have a prefered more portable solution. A separate issue with this same function is undefined behavior getting flagged (unnecessarily) by sanitizers:
https://lore.kernel.org/lkml/[email protected]/

I was planning to come back to that once we got these changes landed.

Thanks!
Ian

#17 lore raw

Re: [PATCH v3 7/7] perf expr: Migrate expr ids table to a hashmap

Em Mon, May 18, 2020 at 09:29:06AM -0700, Ian Rogers escreveu:

Quoted 4 lines

I had some issues here too:
https://lore.kernel.org/lkml/CAEf4BzYxTTND7T7X0dLr2CbkEvUuKtarOeoJYYROefij+qds0w@mail.gmail.com/
The only reason for the bits/reg.h inclusion is for __WORDSIZE for the hash_bits operation. As shown below:

So, to have perf building in all the systems I have test build containers for I have this in:

tools/include/linux/bitops.h

#include <asm/types.h>
#include <limits.h>
#ifndef __WORDSIZE
#define __WORDSIZE (__SIZEOF_LONG__ * 8)
#endif

With that it works everywhere, Android cross NDK for arm64, older systems, cross compilers to many arches, Musl libc, uclibc, etc.

So I'm trying, just to check, that this change will make this build everywhere:

commit ef4c968ccbd52d6a02553719ac7e97f70c65ba47

Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Tue May 19 16:26:14 2020 -0300

WIP

tools/perf/util/hashmap.h +3 -4
index e823b35e7371..df59fd4fc95b 100644
@@ -10,10 +10,9 @@
 
 #include <stdbool.h>
 #include <stddef.h>
-#ifdef __GLIBC__
-#include <bits/wordsize.h>
-#else
-#include <bits/reg.h>
+#include <limits.h>
+#ifndef __WORDSIZE
+#define __WORDSIZE (__SIZEOF_LONG__ * 8)
 #endif
 
 static inline size_t hash_bits(size_t h, int bits)
Quoted 34 lines

#ifdef __GLIBC__
#include <bits/wordsize.h>
#else
#include <bits/reg.h>
#endif
static inline size_t hash_bits(size_t h, int bits) {
/* shuffle bits and return requested number of upper bits */ return (h * 11400714819323198485llu) >> (__WORDSIZE - bits); }

It'd be possible to change the definition of hash_bits and remove the #includes by:

static inline size_t hash_bits(size_t h, int bits) {
/* shuffle bits and return requested number of upper bits */ #ifdef __LP64__
int shift = 64 - bits;
#else
int shift = 32 - bits;
#endif
return (h * 11400714819323198485llu) >> shift;
}

Others may have a prefered more portable solution. A separate issue with this same function is undefined behavior getting flagged (unnecessarily) by sanitizers:
https://lore.kernel.org/lkml/[email protected]/

I was planning to come back to that once we got these changes landed.

Thanks!
Ian

Signature · 1 line
- Arnaldo