Skip to contentSkip to conversation

Copy hashmap to tools/perf/util, use in perf expr

PATCH Superseded Ian Rogers 5 participants 23 messages May 15, 2020

Subsystems (5)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Ian Rogers

author reviewer
#0 lore raw

v2 cover Copy hashmap to tools/perf/util, use in perf expr

Perf's expr code currently builds an array of strings then removes duplicates. The array is larger than necessary and has recently been increased in size. When this was done it was commented that a hashmap would be preferable.

libbpf has a hashmap but libbpf isn't currently required to build perf. To satisfy various concerns this change copies libbpf's hashmap into tools/perf/util, it then adds a check in perf that the two are in sync.

Andrii's patch to hashmap from bpf-next is brought into this set to fix issues with hashmap__clear.

Two minor changes to libbpf's hashmap are made that remove an unused dependency and fix a compiler warning.

Two perf test changes are also brought in as they need refactoring to account for the expr API change and it is expected they will land ahead of this.
https://lore.kernel.org/lkml/[email protected]/

Tested with 'perf test' and 'make -C tools/perf build-test'.

The hashmap change was originally part of an RFC:
https://lore.kernel.org/lkml/[email protected]/

v2. moves hashmap into tools/perf/util rather than libapi, to allow hashmap's libbpf symbols to be visible when built statically for testing.

Andrii Nakryiko (1):
libbpf: Fix memory leak and possible double-free in hashmap__clear

Ian Rogers (6):
libbpf hashmap: Remove unused #include
libbpf hashmap: Fix signedness warnings
tools lib/api: Copy libbpf hashmap to tools/perf/util perf test: Provide a subtest callback to ask for the reason for
skipping a subtest
perf test: Improve pmu event metric testing
perf expr: Migrate expr ids table to a hashmap

 tools/lib/bpf/hashmap.c         |  10 +-
 tools/lib/bpf/hashmap.h         |   1 -
 tools/perf/check-headers.sh     |   4 +
 tools/perf/tests/builtin-test.c |  18 ++-
 tools/perf/tests/expr.c         |  40 +++---
 tools/perf/tests/pmu-events.c   | 169 ++++++++++++++++++++++-
 tools/perf/tests/tests.h        |   4 +
 tools/perf/util/Build           |   4 +
 tools/perf/util/expr.c          | 129 +++++++++--------
 tools/perf/util/expr.h          |  26 ++--
 tools/perf/util/expr.y          |  22 +--
 tools/perf/util/hashmap.c       | 238 ++++++++++++++++++++++++++++++++
 tools/perf/util/hashmap.h       | 177 ++++++++++++++++++++++++
 tools/perf/util/metricgroup.c   |  87 ++++++------
 tools/perf/util/stat-shadow.c   |  49 ++++---
 15 files changed, 798 insertions(+), 180 deletions(-)
 create mode 100644 tools/perf/util/hashmap.c
 create mode 100644 tools/perf/util/hashmap.h
Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
author reviewer
#1 lore raw

PATCH v2 1/7 libbpf: Fix memory leak and possible double-free in hashmap__clear

From: Andrii Nakryiko <[email protected]>

Fix memory leak in hashmap_clear() not freeing hashmap_entry structs for each of the remaining entries. Also NULL-out bucket list to prevent possible double-free between hashmap__clear() and hashmap__free().

Running test_progs-asan flavor clearly showed this problem.

---
 tools/lib/bpf/hashmap.c | 7 +++++++
 1 file changed, 7 insertions(+)
tools/lib/bpf/hashmap.c +7 -0
index 54c30c802070..cffb96202e0d 100644
@@ -59,7 +59,14 @@ struct hashmap *hashmap__new(hashmap_hash_fn hash_fn,
 
 void hashmap__clear(struct hashmap *map)
 {
+	struct hashmap_entry *cur, *tmp;
+	int bkt;
+
+	hashmap__for_each_entry_safe(map, cur, tmp, bkt) {
+		free(cur);
+	}
 	free(map->buckets);
+	map->buckets = NULL;
 	map->cap = map->cap_bits = map->sz = 0;
 }
 
Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
author reviewer
#2 lore raw

PATCH v2 2/7 libbpf hashmap: Remove unused #include

In reply to Ian Rogers

Remove #include of libbpf_internal.h that is unused. Discussed in this thread:
https://lore.kernel.org/lkml/CAEf4BzZRmiEds_8R8g4vaAeWvJzPb4xYLnpF0X2VNY8oTzkphQ@mail.gmail.com/

---
 tools/lib/bpf/hashmap.h | 1 -
 1 file changed, 1 deletion(-)
tools/lib/bpf/hashmap.h +0 -1
index bae8879cdf58..e823b35e7371 100644
@@ -15,7 +15,6 @@
 #else
 #include <bits/reg.h>
 #endif
-#include "libbpf_internal.h"
 
 static inline size_t hash_bits(size_t h, int bits)
 {
Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
Acked-by on v2 2/7
#3 lore raw

Re: [PATCH v2 2/7] libbpf hashmap: Remove unused #include

Quoted 7 lines · Ian Rogers

Remove #include of libbpf_internal.h that is unused. Discussed in this thread:
https://lore.kernel.org/lkml/CAEf4BzZRmiEds_8R8g4vaAeWvJzPb4xYLnpF0X2VNY8oTzkphQ@mail.gmail.com/

---

Quoted 18 lines
 tools/lib/bpf/hashmap.h | 1 -
 1 file changed, 1 deletion(-)
tools/lib/bpf/hashmap.h +0 -2
index bae8879cdf58..e823b35e7371 100644
@@ -15,7 +15,6 @@
 #else
 #include <bits/reg.h>
 #endif
-#include "libbpf_internal.h"
 
 static inline size_t hash_bits(size_t h, int bits)
 {
--

2.26.2.761.g0e0b3e54be-goog

author reviewer
#4 lore raw

PATCH v2 3/7 libbpf hashmap: Fix signedness warnings

In reply to Ian Rogers

Fixes the following warnings:

hashmap.c: In function ‘hashmap__clear’:
hashmap.h:150:20: error: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Werror=sign-compare]
150 | for (bkt = 0; bkt < map->cap; bkt++) \

hashmap.c: In function ‘hashmap_grow’:
hashmap.h:150:20: error: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Werror=sign-compare]
150 | for (bkt = 0; bkt < map->cap; bkt++) \

---
 tools/lib/bpf/hashmap.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
tools/lib/bpf/hashmap.c +2 -3
index cffb96202e0d..a405dad068f5 100644
@@ -60,7 +60,7 @@ struct hashmap *hashmap__new(hashmap_hash_fn hash_fn,
 void hashmap__clear(struct hashmap *map)
 {
 	struct hashmap_entry *cur, *tmp;
-	int bkt;
+	size_t bkt;
 
 	hashmap__for_each_entry_safe(map, cur, tmp, bkt) {
 		free(cur);
@@ -100,8 +100,7 @@ static int hashmap_grow(struct hashmap *map)
 	struct hashmap_entry **new_buckets;
 	struct hashmap_entry *cur, *tmp;
 	size_t new_cap_bits, new_cap;
-	size_t h;
-	int bkt;
+	size_t h, bkt;
 
 	new_cap_bits = map->cap_bits + 1;
 	if (new_cap_bits < HASHMAP_MIN_CAP_BITS)
Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
Acked-by on v2 3/7
#5 lore raw

Re: [PATCH v2 3/7] libbpf hashmap: Fix signedness warnings

Quoted 7 lines · Ian Rogers

Fixes the following warnings:

hashmap.c: In function ‘hashmap__clear’:
hashmap.h:150:20: error: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Werror=sign-compare]
150 | for (bkt = 0; bkt < map->cap; bkt++) \

Quoted 6 lines

hashmap.c: In function ‘hashmap_grow’:
hashmap.h:150:20: error: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Werror=sign-compare]
150 | for (bkt = 0; bkt < map->cap; bkt++) \

---

Quoted 29 lines
 tools/lib/bpf/hashmap.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
tools/lib/bpf/hashmap.c +2 -4
index cffb96202e0d..a405dad068f5 100644
@@ -60,7 +60,7 @@ struct hashmap *hashmap__new(hashmap_hash_fn hash_fn,
 void hashmap__clear(struct hashmap *map)
 {
        struct hashmap_entry *cur, *tmp;
-       int bkt;
+       size_t bkt;
 
        hashmap__for_each_entry_safe(map, cur, tmp, bkt) {
                free(cur);
@@ -100,8 +100,7 @@ static int hashmap_grow(struct hashmap *map)
        struct hashmap_entry **new_buckets;
        struct hashmap_entry *cur, *tmp;
        size_t new_cap_bits, new_cap;
-       size_t h;
-       int bkt;
+       size_t h, bkt;
 
        new_cap_bits = map->cap_bits + 1;
        if (new_cap_bits < HASHMAP_MIN_CAP_BITS)
--

2.26.2.761.g0e0b3e54be-goog

author reviewer
#6 lore raw

PATCH v2 4/7 tools lib/api: Copy libbpf hashmap to tools/perf/util

In reply to Ian Rogers

Allow use of hashmap in perf. Modify perf's check-headers.sh script to check that the files are kept in sync, in the same way kernel headers are checked. This will warn if they are out of sync at the start of a perf build.

---
 tools/perf/check-headers.sh |   4 +
 tools/perf/util/Build       |   4 +
 tools/perf/util/hashmap.c   | 238 ++++++++++++++++++++++++++++++++++++
 tools/perf/util/hashmap.h   | 177 +++++++++++++++++++++++++++
 4 files changed, 423 insertions(+)
 create mode 100644 tools/perf/util/hashmap.c
 create mode 100644 tools/perf/util/hashmap.h
tools/perf/check-headers.sh +4 -0
index cf147db4e5ca..94c2bc22c2bb 100755
@@ -128,4 +128,8 @@ check arch/x86/lib/insn.c             '-I "^#include [\"<]\(../include/\)*asm/in
 # diff non-symmetric files
 check_2 tools/perf/arch/x86/entry/syscalls/syscall_64.tbl arch/x86/entry/syscalls/syscall_64.tbl
 
+# check duplicated library files
+check_2 tools/perf/util/hashmap.h tools/lib/bpf/hashmap.h
+check_2 tools/perf/util/hashmap.c tools/lib/bpf/hashmap.c
+
 cd tools/perf
tools/perf/util/Build +4 -0
index ca07a162d602..880fcdd1ab11 100644
@@ -136,6 +136,10 @@ perf-$(CONFIG_LIBELF) += symbol-elf.o
 perf-$(CONFIG_LIBELF) += probe-file.o
 perf-$(CONFIG_LIBELF) += probe-event.o
 
+ifndef CONFIG_LIBBPF
+perf-y += hashmap.o
+endif
+
 ifndef CONFIG_LIBELF
 perf-y += symbol-minimal.o
 endif
tools/perf/util/hashmap.c new file +238 -0

238 lines not on this page · raw message

tools/perf/util/hashmap.h new file +177 -0

177 lines not on this page · raw message

Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
Acked-by on v2 4/7
#7 lore raw

Re: [PATCH v2 4/7] tools lib/api: Copy libbpf hashmap to tools/perf/util

Quoted 8 lines · Ian Rogers

Allow use of hashmap in perf. Modify perf's check-headers.sh script to check that the files are kept in sync, in the same way kernel headers are checked. This will warn if they are out of sync at the start of a perf build.

---

Given you want to make sure they stay 1 to 1, would just creating a symlink work instead of copying the code?

Either way, I think hashmap is stable and not going to change frequently, so whichever way is fine with me.

Quoted 8 lines
 tools/perf/check-headers.sh |   4 +
 tools/perf/util/Build       |   4 +
 tools/perf/util/hashmap.c   | 238 ++++++++++++++++++++++++++++++++++++
 tools/perf/util/hashmap.h   | 177 +++++++++++++++++++++++++++
 4 files changed, 423 insertions(+)
 create mode 100644 tools/perf/util/hashmap.c
 create mode 100644 tools/perf/util/hashmap.h

[...]

author reviewer
#8 lore raw

PATCH v2 5/7 perf test: Provide a subtest callback to ask for the reason for skipping a subtest

In reply to Ian Rogers

Now subtests can inform why a test was skipped. The upcoming patch improvint PMU event metric testing will use it.

Cc Adrian Hunter <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jin Yao <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc John Garry <[email protected]>
Cc Kajol Jain <[email protected]>
Cc Kan Liang <[email protected]>
Cc Leo Yan <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Paul Clarke <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephane Eranian <[email protected]>
[ split from a larger patch ]
---
 tools/perf/tests/builtin-test.c | 11 +++++++++--
 tools/perf/tests/tests.h        |  1 +
 2 files changed, 10 insertions(+), 2 deletions(-)
tools/perf/tests/builtin-test.c +9 -2
index 3471ec52ea11..baee735e6aa5 100644
@@ -429,8 +429,15 @@ static int test_and_print(struct test *t, bool force_skip, int subtest)
 	case TEST_OK:
 		pr_info(" Ok\n");
 		break;
-	case TEST_SKIP:
-		color_fprintf(stderr, PERF_COLOR_YELLOW, " Skip\n");
+	case TEST_SKIP: {
+		const char *skip_reason = NULL;
+		if (t->subtest.skip_reason)
+			skip_reason = t->subtest.skip_reason(subtest);
+		if (skip_reason)
+			color_fprintf(stderr, PERF_COLOR_YELLOW, " Skip (%s)\n", skip_reason);
+		else
+			color_fprintf(stderr, PERF_COLOR_YELLOW, " Skip\n");
+	}
 		break;
 	case TEST_FAIL:
 	default:
tools/perf/tests/tests.h +1 -0
index d6d4ac34eeb7..88e45aeab94f 100644
@@ -34,6 +34,7 @@ struct test {
 		bool skip_if_fail;
 		int (*get_nr)(void);
 		const char *(*get_desc)(int subtest);
+		const char *(*skip_reason)(int subtest);
 	} subtest;
 	bool (*is_supported)(void);
 	void *priv;
Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
author reviewer
#9 lore raw

PATCH v2 6/7 perf test: Improve pmu event metric testing

In reply to Ian Rogers

Break pmu-events test into 2 and add a test to verify that all pmu metric expressions simply parse. Try to parse all metric ids/events, skip/warn if metrics for the current architecture fail to parse. To support warning for a skip, and an ability for a subtest to describe why it skips.

Tested on power9, skylakex, haswell, broadwell, westmere, sandybridge and ivybridge.

May skip/warn on other architectures if metrics are invalid. In particular s390 is untested, but its expressions are trivial. The untested architectures with expressions are power8, cascadelakex, tremontx, skylake, jaketown, ivytown and variants of haswell and broadwell.

v3. addresses review comments from John Garry <[email protected]>, Jiri Olsa <[email protected]> and Arnaldo Carvalho de Melo <[email protected]>.
v2. changes the commit message as event parsing errors no longer cause the test to fail.

Cc Adrian Hunter <[email protected]>
Cc Alexander Shishkin <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jin Yao <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc John Garry <[email protected]>
Cc Kajol Jain <[email protected]>
Cc Kan Liang <[email protected]>
Cc Leo Yan <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Paul Clarke <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephane Eranian <[email protected]>
[ split from a larger patch ]
---
 tools/perf/tests/builtin-test.c |   7 ++
 tools/perf/tests/pmu-events.c   | 168 ++++++++++++++++++++++++++++++--
 tools/perf/tests/tests.h        |   3 +
 3 files changed, 172 insertions(+), 6 deletions(-)
tools/perf/tests/builtin-test.c +7 -0
index baee735e6aa5..9553f8061772 100644
@@ -75,6 +75,13 @@ static struct test generic_tests[] = {
 	{
 		.desc = "PMU events",
 		.func = test__pmu_events,
+		.subtest = {
+			.skip_if_fail	= false,
+			.get_nr		= test__pmu_events_subtest_get_nr,
+			.get_desc	= test__pmu_events_subtest_get_desc,
+			.skip_reason	= test__pmu_events_subtest_skip_reason,
+		},
+
 	},
 	{
 		.desc = "DSO data read",
tools/perf/tests/pmu-events.c +162 -6

192 lines not on this page · raw message

tools/perf/tests/tests.h +3 -0
index 88e45aeab94f..6c6c4b6a4796 100644
@@ -51,6 +51,9 @@ int test__perf_evsel__tp_sched_test(struct test *test, int subtest);
 int test__syscall_openat_tp_fields(struct test *test, int subtest);
 int test__pmu(struct test *test, int subtest);
 int test__pmu_events(struct test *test, int subtest);
+const char *test__pmu_events_subtest_get_desc(int subtest);
+const char *test__pmu_events_subtest_skip_reason(int subtest);
+int test__pmu_events_subtest_get_nr(void);
 int test__attr(struct test *test, int subtest);
 int test__dso_data(struct test *test, int subtest);
 int test__dso_data_cache(struct test *test, int subtest);
Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
author reviewer
#10 lore raw

PATCH v2 7/7 perf expr: Migrate expr ids table to a hashmap

In reply to Ian Rogers

Use a hashmap between a char* string and a double* value. While bpf's hashmap entries are size_t in size, we can't guarantee sizeof(size_t) >= sizeof(double). Avoid a memory allocation when gathering ids by making 0.0 a special value encoded as NULL.

Original map suggestion by Andi Kleen:
https://lore.kernel.org/lkml/[email protected]/
and seconded by Jiri Olsa:
https://lore.kernel.org/lkml/20200423112915.GH1136647@krava/

---
 tools/perf/tests/expr.c       |  40 ++++++-----
 tools/perf/tests/pmu-events.c |  25 +++----
 tools/perf/util/expr.c        | 129 +++++++++++++++++++---------------
 tools/perf/util/expr.h        |  26 +++----
 tools/perf/util/expr.y        |  22 +-----
 tools/perf/util/metricgroup.c |  87 +++++++++++------------
 tools/perf/util/stat-shadow.c |  49 ++++++++-----
 7 files changed, 197 insertions(+), 181 deletions(-)
tools/perf/tests/expr.c +21 -19
index 3f742612776a..5e606fd5a2c6 100644
@@ -19,11 +19,9 @@ static int test(struct expr_parse_ctx *ctx, const char *e, double val2)
 int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused)
 {
 	const char *p;
-	const char **other;
-	double val;
-	int i, ret;
+	double val, *val_ptr;
+	int ret;
 	struct expr_parse_ctx ctx;
-	int num_other;
 
 	expr__ctx_init(&ctx);
 	expr__add_id(&ctx, "FOO", 1);
@@ -52,25 +50,29 @@ int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused)
 	ret = expr__parse(&val, &ctx, p, 1);
 	TEST_ASSERT_VAL("missing operand", ret == -1);
 
+	hashmap__clear(&ctx.ids);
 	TEST_ASSERT_VAL("find other",
-			expr__find_other("FOO + BAR + BAZ + BOZO", "FOO", &other, &num_other, 1) == 0);
-	TEST_ASSERT_VAL("find other", num_other == 3);
-	TEST_ASSERT_VAL("find other", !strcmp(other[0], "BAR"));
-	TEST_ASSERT_VAL("find other", !strcmp(other[1], "BAZ"));
-	TEST_ASSERT_VAL("find other", !strcmp(other[2], "BOZO"));
-	TEST_ASSERT_VAL("find other", other[3] == NULL);
+			expr__find_other("FOO + BAR + BAZ + BOZO", "FOO",
+					 &ctx, 1) == 0);
+	TEST_ASSERT_VAL("find other", hashmap__size(&ctx.ids) == 3);
+	TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "BAR",
+						    (void **)&val_ptr));
+	TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "BAZ",
+						    (void **)&val_ptr));
+	TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "BOZO",
+						    (void **)&val_ptr));
 
+	hashmap__clear(&ctx.ids);
 	TEST_ASSERT_VAL("find other",
-			expr__find_other("EVENT1\\,param\\=?@ + EVENT2\\,param\\=?@", NULL,
-				   &other, &num_other, 3) == 0);
-	TEST_ASSERT_VAL("find other", num_other == 2);
-	TEST_ASSERT_VAL("find other", !strcmp(other[0], "EVENT1,param=3/"));
-	TEST_ASSERT_VAL("find other", !strcmp(other[1], "EVENT2,param=3/"));
-	TEST_ASSERT_VAL("find other", other[2] == NULL);
+			expr__find_other("EVENT1\\,param\\=?@ + EVENT2\\,param\\=?@",
+					 NULL, &ctx, 3) == 0);
+	TEST_ASSERT_VAL("find other", hashmap__size(&ctx.ids) == 2);
+	TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "EVENT1,param=3/",
+						    (void **)&val_ptr));
+	TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "EVENT2,param=3/",
+						    (void **)&val_ptr));
 
-	for (i = 0; i < num_other; i++)
-		zfree(&other[i]);
-	free((void *)other);
+	expr__ctx_clear(&ctx);
 
 	return 0;
 }
tools/perf/tests/pmu-events.c +13 -12
index e21f0addcfbb..3de59564deb0 100644
@@ -433,8 +433,6 @@ static int test_parsing(void)
 	struct pmu_events_map *map;
 	struct pmu_event *pe;
 	int i, j, k;
-	const char **ids;
-	int idnum;
 	int ret = 0;
 	struct expr_parse_ctx ctx;
 	double result;
@@ -446,29 +444,34 @@ static int test_parsing(void)
 			break;
 		j = 0;
 		for (;;) {
+			struct hashmap_entry *cur;
+			size_t bkt;
+
 			pe = &map->table[j++];
 			if (!pe->name && !pe->metric_group && !pe->metric_name)
 				break;
 			if (!pe->metric_expr)
 				continue;
-			if (expr__find_other(pe->metric_expr, NULL,
-						&ids, &idnum, 0) < 0) {
+			expr__ctx_init(&ctx);
+			if (expr__find_other(pe->metric_expr, NULL, &ctx, 0)
+				  < 0) {
 				expr_failure("Parse other failed", map, pe);
 				ret++;
 				continue;
 			}
-			expr__ctx_init(&ctx);
 
 			/*
 			 * Add all ids with a made up value. The value may
 			 * trigger divide by zero when subtracted and so try to
 			 * make them unique.
 			 */
-			for (k = 0; k < idnum; k++)
-				expr__add_id(&ctx, ids[k], k + 1);
+			k = 1;
+			hashmap__for_each_entry((&ctx.ids), cur, bkt)
+				expr__add_id(&ctx, strdup(cur->key), k++);
 
-			for (k = 0; k < idnum; k++) {
-				if (check_parse_id(ids[k], map == cpus_map, pe))
+			hashmap__for_each_entry((&ctx.ids), cur, bkt) {
+				if (check_parse_id(cur->key, map == cpus_map,
+						   pe))
 					ret++;
 			}
 
@@ -476,9 +479,7 @@ static int test_parsing(void)
 				expr_failure("Parse failed", map, pe);
 				ret++;
 			}
-			for (k = 0; k < idnum; k++)
-				zfree(&ids[k]);
-			free(ids);
+			expr__ctx_clear(&ctx);
 		}
 	}
 	/* TODO: fail when not ok */
tools/perf/util/expr.c +72 -57

158 lines not on this page · raw message

tools/perf/util/expr.h +13 -13
index 40fc452b0f2b..d60a8feaf50b 100644
@@ -2,17 +2,14 @@
 #ifndef PARSE_CTX_H
 #define PARSE_CTX_H 1
 
-#define EXPR_MAX_OTHER 64
-#define MAX_PARSE_ID EXPR_MAX_OTHER
-
-struct expr_parse_id {
-	const char *name;
-	double val;
-};
+#ifdef HAVE_LIBBPF_SUPPORT
+#include <bpf/hashmap.h>
+#else
+#include "hashmap.h"
+#endif
 
 struct expr_parse_ctx {
-	int num_ids;
-	struct expr_parse_id ids[MAX_PARSE_ID];
+	struct hashmap ids;
 };
 
 struct expr_scanner_ctx {
@@ -21,9 +18,12 @@ struct expr_scanner_ctx {
 };
 
 void expr__ctx_init(struct expr_parse_ctx *ctx);
-void expr__add_id(struct expr_parse_ctx *ctx, const char *id, double val);
-int expr__parse(double *final_val, struct expr_parse_ctx *ctx, const char *expr, int runtime);
-int expr__find_other(const char *expr, const char *one, const char ***other,
-		int *num_other, int runtime);
+void expr__ctx_clear(struct expr_parse_ctx *ctx);
+int expr__add_id(struct expr_parse_ctx *ctx, const char *id, double val);
+int expr__get_id(struct expr_parse_ctx *ctx, const char *id, double *val_ptr);
+int expr__parse(double *final_val, struct expr_parse_ctx *ctx,
+		const char *expr, int runtime);
+int expr__find_other(const char *expr, const char *one,
+		struct expr_parse_ctx *ids, int runtime);
 
 #endif
tools/perf/util/expr.y +2 -20
index 21e82a1e11a2..ec5a48bf5f34 100644
@@ -46,19 +46,6 @@ static void expr_error(double *final_val __maybe_unused,
 	pr_debug("%s\n", s);
 }
 
-static int lookup_id(struct expr_parse_ctx *ctx, char *id, double *val)
-{
-	int i;
-
-	for (i = 0; i < ctx->num_ids; i++) {
-		if (!strcasecmp(ctx->ids[i].name, id)) {
-			*val = ctx->ids[i].val;
-			return 0;
-		}
-	}
-	return -1;
-}
-
 %}
 %%
 
@@ -72,12 +59,7 @@ all_other: all_other other
 
 other: ID
 {
-	if (ctx->num_ids + 1 >= EXPR_MAX_OTHER) {
-		pr_err("failed: way too many variables");
-		YYABORT;
-	}
-
-	ctx->ids[ctx->num_ids++].name = $1;
+	expr__add_id(ctx, $1, 0.0);
 }
 |
 MIN | MAX | IF | ELSE | SMT_ON | NUMBER | '|' | '^' | '&' | '-' | '+' | '*' | '/' | '%' | '(' | ')' | ','
@@ -92,7 +74,7 @@ if_expr:
 	;
 
 expr:	  NUMBER
-	| ID			{ if (lookup_id(ctx, $1, &$$) < 0) {
+	| ID			{ if (expr__get_id(ctx, $1, &$$)) {
 					pr_debug("%s not found\n", $1);
 					YYABORT;
 				  }
tools/perf/util/metricgroup.c +44 -43

182 lines not on this page · raw message

tools/perf/util/stat-shadow.c +32 -17

116 lines not on this page · raw message

Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
#11 lore raw

Re: [PATCH v2 7/7] perf expr: Migrate expr ids table to a hashmap

Quoted 41 lines · Ian Rogers

Use a hashmap between a char* string and a double* value. While bpf's hashmap entries are size_t in size, we can't guarantee sizeof(size_t) >= sizeof(double). Avoid a memory allocation when gathering ids by making 0.0 a special value encoded as NULL.

Original map suggestion by Andi Kleen:
https://lore.kernel.org/lkml/[email protected]/
and seconded by Jiri Olsa:
https://lore.kernel.org/lkml/20200423112915.GH1136647@krava/

---
 tools/perf/tests/expr.c       |  40 ++++++-----
 tools/perf/tests/pmu-events.c |  25 +++----
 tools/perf/util/expr.c        | 129 +++++++++++++++++++---------------
 tools/perf/util/expr.h        |  26 +++----
 tools/perf/util/expr.y        |  22 +-----
 tools/perf/util/metricgroup.c |  87 +++++++++++------------
 tools/perf/util/stat-shadow.c |  49 ++++++++-----
 7 files changed, 197 insertions(+), 181 deletions(-)
tools/perf/tests/expr.c +2 -4
index 3f742612776a..5e606fd5a2c6 100644
@@ -19,11 +19,9 @@ static int test(struct expr_parse_ctx *ctx, const char *e, double val2)
 int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused)
 {
        const char *p;
-       const char **other;
-       double val;
-       int i, ret;
+       double val, *val_ptr;
+       int ret;
        struct expr_parse_ctx ctx;
-       int num_other;
 
        expr__ctx_init(&ctx);
        expr__add_id(&ctx, "FOO", 1);

tools/perf/tests/expr.c int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused) · line 53

Quoted 4 lines
tools/perf/tests/expr.c +1 -0
@@ -52,25 +50,29 @@ int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused)
        ret = expr__parse(&val, &ctx, p, 1);
        TEST_ASSERT_VAL("missing operand", ret == -1);
 
+       hashmap__clear(&ctx.ids);

hashmap__clear() will free up memory allocated for hashmap itself and hash entries, but not keys/values. Unless it's happening somewhere else, you'll need to do something similar to expr__ctx_clear() below?

Same below for another "lone" hashmap_clear() call.

Quoted 2 lines
tools/perf/tests/expr.c +0 -1
        TEST_ASSERT_VAL("find other",
-                       expr__find_other("FOO + BAR + BAZ + BOZO", "FOO", &other, &num_other, 1) == 0);

[...]

author reviewer
#12 lore raw

Re: [PATCH v2 7/7] perf expr: Migrate expr ids table to a hashmap

Quoted 46 lines · Andrii Nakryiko
Quoted 44 lines · Ian Rogers

Use a hashmap between a char* string and a double* value. While bpf's hashmap entries are size_t in size, we can't guarantee sizeof(size_t) >= sizeof(double). Avoid a memory allocation when gathering ids by making 0.0 a special value encoded as NULL.

Original map suggestion by Andi Kleen:
https://lore.kernel.org/lkml/[email protected]/
and seconded by Jiri Olsa:
https://lore.kernel.org/lkml/20200423112915.GH1136647@krava/

---
 tools/perf/tests/expr.c       |  40 ++++++-----
 tools/perf/tests/pmu-events.c |  25 +++----
 tools/perf/util/expr.c        | 129 +++++++++++++++++++---------------
 tools/perf/util/expr.h        |  26 +++----
 tools/perf/util/expr.y        |  22 +-----
 tools/perf/util/metricgroup.c |  87 +++++++++++------------
 tools/perf/util/stat-shadow.c |  49 ++++++++-----
 7 files changed, 197 insertions(+), 181 deletions(-)
tools/perf/tests/expr.c +2 -4
index 3f742612776a..5e606fd5a2c6 100644
@@ -19,11 +19,9 @@ static int test(struct expr_parse_ctx *ctx, const char *e, double val2)
 int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused)
 {
        const char *p;
-       const char **other;
-       double val;
-       int i, ret;
+       double val, *val_ptr;
+       int ret;
        struct expr_parse_ctx ctx;
-       int num_other;
 
        expr__ctx_init(&ctx);
        expr__add_id(&ctx, "FOO", 1);
@@ -52,25 +50,29 @@ int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused)
        ret = expr__parse(&val, &ctx, p, 1);
        TEST_ASSERT_VAL("missing operand", ret == -1);
 
Quoted 5 lines

tools/perf/tests/expr.c int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused) · line 53

Quoted 1 line
tools/perf/tests/expr.c +1 -0
+       hashmap__clear(&ctx.ids);

hashmap__clear() will free up memory allocated for hashmap itself and hash entries, but not keys/values. Unless it's happening somewhere else, you'll need to do something similar to expr__ctx_clear() below?

In this case the const char* keys come from the literals added on lines 27 and 28 and so didn't need free-ing - which is what expr__ctx_clear() does. I've made these literals strdups and switched to expr__ctx_clear() as you suggest, as this is more reflective of the real use.

Quoted 1 line

Same below for another "lone" hashmap_clear() call.

This was a memory leak, thanks!
Ian

Quoted 4 lines
Quoted 2 lines
tools/perf/tests/expr.c +0 -1
        TEST_ASSERT_VAL("find other",
-                       expr__find_other("FOO + BAR + BAZ + BOZO", "FOO", &other, &num_other, 1) == 0);

[...]

#13 lore raw

Re: [PATCH v2 7/7] perf expr: Migrate expr ids table to a hashmap

In reply to Ian Rogers

On Fri, May 15, 2020 at 09:50:07AM -0700, Ian Rogers wrote:

SNIP

Quoted 27 lines
tools/perf/util/metricgroup.c +2 -5
index b071df373f8b..37be5a368d6e 100644
@@ -85,8 +85,7 @@ static void metricgroup__rblist_init(struct rblist *metric_events)
 
 struct egroup {
 	struct list_head nd;
-	int idnum;
-	const char **ids;
+	struct expr_parse_ctx pctx;
 	const char *metric_name;
 	const char *metric_expr;
 	const char *metric_unit;
@@ -94,19 +93,21 @@ struct egroup {
 };
 
 static struct evsel *find_evsel_group(struct evlist *perf_evlist,
-				      const char **ids,
-				      int idnum,
+				      struct expr_parse_ctx *pctx,
 				      struct evsel **metric_events,
 				      bool *evlist_used)
 {
 	struct evsel *ev;
-	int i = 0, j = 0;
 	bool leader_found;

tools/perf/util/metricgroup.c struct egroup { · line 110

Quoted 10 lines
tools/perf/util/metricgroup.c +5 -1
+	const size_t idnum = hashmap__size(&pctx->ids);
+	size_t i = 0;
+	int j = 0;
+	double *val_ptr;
 
 	evlist__for_each_entry (perf_evlist, ev) {
 		if (evlist_used[j++])
 			continue;
-		if (!strcmp(ev->name, ids[i])) {
+		if (hashmap__find(&pctx->ids, ev->name, (void **)&val_ptr)) {

hum, you sure it's doing the same thing as before?

hashmap__find will succede all the time in here, while the previous code was looking for the start of the group ... the logic in here is little convoluted, so maybe I'm missing some point in here ;-)

jirka

Quoted 12 lines
tools/perf/util/metricgroup.c +2 -1
 			if (!metric_events[i])
 				metric_events[i] = ev;
 			i++;
@@ -118,7 +119,8 @@ static struct evsel *find_evsel_group(struct evlist *perf_evlist,
 			memset(metric_events, 0,
 				sizeof(struct evsel *) * idnum);
 
-			if (!strcmp(ev->name, ids[i])) {
+			if (hashmap__find(&pctx->ids, ev->name,
+					  (void **)&val_ptr)) {
 				if (!metric_events[i])
 					metric_events[i] = ev;

SNIP

author reviewer
#14 lore raw

Re: [PATCH v2 7/7] perf expr: Migrate expr ids table to a hashmap

Quoted 45 lines · Jiri Olsa

On Fri, May 15, 2020 at 09:50:07AM -0700, Ian Rogers wrote:

SNIP

Quoted 27 lines
tools/perf/util/metricgroup.c +2 -5
index b071df373f8b..37be5a368d6e 100644
@@ -85,8 +85,7 @@ static void metricgroup__rblist_init(struct rblist *metric_events)
 
 struct egroup {
      struct list_head nd;
-     int idnum;
-     const char **ids;
+     struct expr_parse_ctx pctx;
      const char *metric_name;
      const char *metric_expr;
      const char *metric_unit;
@@ -94,19 +93,21 @@ struct egroup {
 };
 
 static struct evsel *find_evsel_group(struct evlist *perf_evlist,
-                                   const char **ids,
-                                   int idnum,
+                                   struct expr_parse_ctx *pctx,
                                    struct evsel **metric_events,
                                    bool *evlist_used)
 {
      struct evsel *ev;
-     int i = 0, j = 0;
      bool leader_found;

tools/perf/util/metricgroup.c struct egroup { · line 110

Quoted 10 lines
tools/perf/util/metricgroup.c +5 -1
+     const size_t idnum = hashmap__size(&pctx->ids);
+     size_t i = 0;
+     int j = 0;
+     double *val_ptr;
 
      evlist__for_each_entry (perf_evlist, ev) {
              if (evlist_used[j++])
                      continue;
-             if (!strcmp(ev->name, ids[i])) {
+             if (hashmap__find(&pctx->ids, ev->name, (void **)&val_ptr)) {

hum, you sure it's doing the same thing as before?

Quoted 4 lines

hashmap__find will succede all the time in here, while the previous code was looking for the start of the group ... the logic in here is little convoluted, so maybe I'm missing some point in here ;-)

If we have a metric like "A + B" and another like "C / D" then by we'll generate a string (the extra_events strbuf in the code) like "{A,B}:W,{C,D}:W" from __metricgroup__add_metric. This will turn into an evlist in metricgroup__parse_groups of A,B,C,D. The code is trying to associate the events A,B with the first metric and C,D with the second. The code doesn't support sharing of events and events are marked as used and can't be part of other metrics. The evlist order is also reflective of the order of metrics, so if there were metrics "A + B + C" and "A + B", as the first metric is first in the evlist we don't run the risk of C being placed with A and B in a different group.

The old code used the order of events to match within a metric and say for metric "A+B+C" we want to match A then B, and so on. The new code acts more like a set, so "A + B + C" becomes a set containing A, B and C, we check A is in the set then B and then C. For both pieces of code they are only working because of the evlist_used "bitmap" and that the order in the evlists and metrics matches.

The current code could just use ordering to match first n1 events with the first metric, the next n2 events with the second and so on. So both the find now, and the strcmp before always return true in this branch.

In the RFC patch set I want to share events and so I do checks related to the group leader so that I know when moving from one group to another in the evlist. The find/strcmp becomes load bearing as I will re-use events as long as they match.
https://lore.kernel.org/lkml/[email protected]/

Quoted 8 lines

jirka

tools/perf/util/metricgroup.c static struct evsel *find_evsel_group(struct evlist *perf_evlist, · line 120

Quoted 6 lines
tools/perf/util/metricgroup.c +0 -0
                      if (!metric_events[i])
                              metric_events[i] = ev;
                      i++;
@@ -118,7 +119,8 @@ static struct evsel *find_evsel_group(struct evlist *perf_evlist,
                      memset(metric_events, 0,
                              sizeof(struct evsel *) * idnum);

This re-check was unnecessary in the old code and unnecessary even more so now as the hashmap_find is given exactly the same arguments. I'll remove it in v3 while addressing Andrii's memory leak fixes.

Thanks,
Ian

Quoted 8 lines
Quoted 5 lines
tools/perf/util/metricgroup.c +2 -1
-                     if (!strcmp(ev->name, ids[i])) {
+                     if (hashmap__find(&pctx->ids, ev->name,
+                                       (void **)&val_ptr)) {
                              if (!metric_events[i])
                                      metric_events[i] = ev;

SNIP

#15 lore raw

Re: [PATCH v2 7/7] perf expr: Migrate expr ids table to a hashmap

Quoted 83 lines · Ian Rogers
Quoted 45 lines · Jiri Olsa

On Fri, May 15, 2020 at 09:50:07AM -0700, Ian Rogers wrote:

SNIP

Quoted 27 lines
tools/perf/util/metricgroup.c +2 -5
index b071df373f8b..37be5a368d6e 100644
@@ -85,8 +85,7 @@ static void metricgroup__rblist_init(struct rblist *metric_events)
 
 struct egroup {
      struct list_head nd;
-     int idnum;
-     const char **ids;
+     struct expr_parse_ctx pctx;
      const char *metric_name;
      const char *metric_expr;
      const char *metric_unit;
@@ -94,19 +93,21 @@ struct egroup {
 };
 
 static struct evsel *find_evsel_group(struct evlist *perf_evlist,
-                                   const char **ids,
-                                   int idnum,
+                                   struct expr_parse_ctx *pctx,
                                    struct evsel **metric_events,
                                    bool *evlist_used)
 {
      struct evsel *ev;
-     int i = 0, j = 0;
      bool leader_found;

tools/perf/util/metricgroup.c struct egroup { · line 110

Quoted 10 lines
tools/perf/util/metricgroup.c +5 -1
+     const size_t idnum = hashmap__size(&pctx->ids);
+     size_t i = 0;
+     int j = 0;
+     double *val_ptr;
 
      evlist__for_each_entry (perf_evlist, ev) {
              if (evlist_used[j++])
                      continue;
-             if (!strcmp(ev->name, ids[i])) {
+             if (hashmap__find(&pctx->ids, ev->name, (void **)&val_ptr)) {

hum, you sure it's doing the same thing as before?

Quoted 4 lines

hashmap__find will succede all the time in here, while the previous code was looking for the start of the group ... the logic in here is little convoluted, so maybe I'm missing some point in here ;-)

If we have a metric like "A + B" and another like "C / D" then by we'll generate a string (the extra_events strbuf in the code) like "{A,B}:W,{C,D}:W" from __metricgroup__add_metric. This will turn into an evlist in metricgroup__parse_groups of A,B,C,D. The code is trying to associate the events A,B with the first metric and C,D with the second. The code doesn't support sharing of events and events are marked as used and can't be part of other metrics. The evlist order is also reflective of the order of metrics, so if there were metrics "A + B + C" and "A + B", as the first metric is first in the evlist we don't run the risk of C being placed with A and B in a different group.

The old code used the order of events to match within a metric and say for metric "A+B+C" we want to match A then B, and so on. The new code acts more like a set, so "A + B + C" becomes a set containing A, B and C, we check A is in the set then B and then C. For both pieces of code they are only working because of the evlist_used "bitmap" and that the order in the evlists and metrics matches.

The current code could just use ordering to match first n1 events with the first metric, the next n2 events with the second and so on. So both the find now, and the strcmp before always return true in this branch.

In the RFC patch set I want to share events and so I do checks related to the group leader so that I know when moving from one group to another in the evlist. The find/strcmp becomes load bearing as I will re-use events as long as they match.
https://lore.kernel.org/lkml/[email protected]/

Quoted 2 lines

jirka

Quoted 10 lines
Quoted 6 lines

tools/perf/util/metricgroup.c static struct evsel *find_evsel_group(struct evlist *perf_evlist, · line 120

Quoted 6 lines
tools/perf/util/metricgroup.c +0 -0
                      if (!metric_events[i])
                              metric_events[i] = ev;
                      i++;
@@ -118,7 +119,8 @@ static struct evsel *find_evsel_group(struct evlist *perf_evlist,
                      memset(metric_events, 0,
                              sizeof(struct evsel *) * idnum);

This re-check was unnecessary in the old code and unnecessary even more so now as the hashmap_find is given exactly the same arguments. I'll remove it in v3 while addressing Andrii's memory leak fixes.

ok, that was my other confusion, because it's useless ;-)

thanks for the explanation,
jirka

Quoted 13 lines

Thanks,
Ian

Quoted 8 lines
Quoted 5 lines
tools/perf/util/metricgroup.c +2 -1
-                     if (!strcmp(ev->name, ids[i])) {
+                     if (hashmap__find(&pctx->ids, ev->name,
+                                       (void **)&val_ptr)) {
                              if (!metric_events[i])
                                      metric_events[i] = ev;

SNIP

#16 lore raw

Re: [PATCH v2 7/7] perf expr: Migrate expr ids table to a hashmap

In reply to Ian Rogers

On Fri, May 15, 2020 at 09:50:07AM -0700, Ian Rogers wrote:

SNIP

Quoted 21 lines
tools/perf/util/expr.c +8 -0
index 8b4ce704a68d..f64ab91c432b 100644
@@ -4,25 +4,76 @@
 #include "expr.h"
 #include "expr-bison.h"
 #include "expr-flex.h"
+#include <linux/kernel.h>
 
 #ifdef PARSER_DEBUG
 extern int expr_debug;
 #endif
 
+static size_t key_hash(const void *key, void *ctx __maybe_unused)
+{
+	const char *str = (const char *)key;
+	size_t hash = 0;
+
+	while (*str != '\0') {
+		hash *= 31;

tools/perf/util/expr.c line 29

Quoted 10 lines
tools/perf/util/expr.c +10 -0
+		hash += *str;
+		str++;
+	}
+	return hash;
+}
+
+static bool key_equal(const void *key1, const void *key2,
+		    void *ctx __maybe_unused)
+{
+	return !strcmp((const char *)key1, (const char *)key2);

should that be strcasecmp ? would it affect the key_hash as well?

jirka

author reviewer
#17 lore raw

Re: [PATCH v2 7/7] perf expr: Migrate expr ids table to a hashmap

Quoted 30 lines · Jiri Olsa

On Fri, May 15, 2020 at 09:50:07AM -0700, Ian Rogers wrote:

SNIP

Quoted 25 lines
tools/perf/util/expr.c +12 -0
index 8b4ce704a68d..f64ab91c432b 100644
@@ -4,25 +4,76 @@
 #include "expr.h"
 #include "expr-bison.h"
 #include "expr-flex.h"
+#include <linux/kernel.h>
 
 #ifdef PARSER_DEBUG
 extern int expr_debug;
 #endif
 
+static size_t key_hash(const void *key, void *ctx __maybe_unused)
+{
+     const char *str = (const char *)key;
+     size_t hash = 0;
+
+     while (*str != '\0') {
+             hash *= 31;
+             hash += *str;
+             str++;
+     }
+     return hash;
Quoted 8 lines

tools/perf/util/expr.c line 29

Quoted 6 lines
tools/perf/util/expr.c +6 -0
+}
+
+static bool key_equal(const void *key1, const void *key2,
+                 void *ctx __maybe_unused)
+{
+     return !strcmp((const char *)key1, (const char *)key2);

should that be strcasecmp ? would it affect the key_hash as well?

The original code does make use of strcasecmp in one place, but in the group matching (the main useless use for this code) it doesn't. I don't think it is a regression to keep it as this, and would like a test case for when it does matter. Is that ok?

Thanks,
Ian

Quoted 2 lines

jirka

#18 lore raw
In reply to Ian Rogers

Em Fri, May 15, 2020 at 09:50:00AM -0700, Ian Rogers escreveu:

Quoted 10 lines

Perf's expr code currently builds an array of strings then removes duplicates. The array is larger than necessary and has recently been increased in size. When this was done it was commented that a hashmap would be preferable.

libbpf has a hashmap but libbpf isn't currently required to build perf. To satisfy various concerns this change copies libbpf's hashmap into tools/perf/util, it then adds a check in perf that the two are in sync.

Quoted 5 lines

Andrii's patch to hashmap from bpf-next is brought into this set to fix issues with hashmap__clear.

Two minor changes to libbpf's hashmap are made that remove an unused dependency and fix a compiler warning.

Andrii/Alexei/Daniel, what do you think about me merging these fixes in my perf-tools-next branch?

- Arnaldo

Quoted 48 lines

Two perf test changes are also brought in as they need refactoring to account for the expr API change and it is expected they will land ahead of this.
https://lore.kernel.org/lkml/[email protected]/

Tested with 'perf test' and 'make -C tools/perf build-test'.

The hashmap change was originally part of an RFC:
https://lore.kernel.org/lkml/[email protected]/

v2. moves hashmap into tools/perf/util rather than libapi, to allow hashmap's libbpf symbols to be visible when built statically for testing.

Andrii Nakryiko (1):
libbpf: Fix memory leak and possible double-free in hashmap__clear

Ian Rogers (6):
libbpf hashmap: Remove unused #include
libbpf hashmap: Fix signedness warnings
tools lib/api: Copy libbpf hashmap to tools/perf/util perf test: Provide a subtest callback to ask for the reason for
skipping a subtest
perf test: Improve pmu event metric testing
perf expr: Migrate expr ids table to a hashmap

 tools/lib/bpf/hashmap.c         |  10 +-
 tools/lib/bpf/hashmap.h         |   1 -
 tools/perf/check-headers.sh     |   4 +
 tools/perf/tests/builtin-test.c |  18 ++-
 tools/perf/tests/expr.c         |  40 +++---
 tools/perf/tests/pmu-events.c   | 169 ++++++++++++++++++++++-
 tools/perf/tests/tests.h        |   4 +
 tools/perf/util/Build           |   4 +
 tools/perf/util/expr.c          | 129 +++++++++--------
 tools/perf/util/expr.h          |  26 ++--
 tools/perf/util/expr.y          |  22 +--
 tools/perf/util/hashmap.c       | 238 ++++++++++++++++++++++++++++++++
 tools/perf/util/hashmap.h       | 177 ++++++++++++++++++++++++
 tools/perf/util/metricgroup.c   |  87 ++++++------
 tools/perf/util/stat-shadow.c   |  49 ++++---
 15 files changed, 798 insertions(+), 180 deletions(-)
 create mode 100644 tools/perf/util/hashmap.c
 create mode 100644 tools/perf/util/hashmap.h
Signature · 1 line
2.26.2.761.g0e0b3e54be-goog
Signature · 1 line
- Arnaldo
#19 lore raw
Quoted 12 lines · Arnaldo Carvalho de Melo

Em Fri, May 15, 2020 at 09:50:00AM -0700, Ian Rogers escreveu:

Quoted 10 lines

Perf's expr code currently builds an array of strings then removes duplicates. The array is larger than necessary and has recently been increased in size. When this was done it was commented that a hashmap would be preferable.

libbpf has a hashmap but libbpf isn't currently required to build perf. To satisfy various concerns this change copies libbpf's hashmap into tools/perf/util, it then adds a check in perf that the two are in sync.

Quoted 8 lines
Quoted 5 lines

Andrii's patch to hashmap from bpf-next is brought into this set to fix issues with hashmap__clear.

Two minor changes to libbpf's hashmap are made that remove an unused dependency and fix a compiler warning.

Andrii/Alexei/Daniel, what do you think about me merging these fixes in my perf-tools-next branch?

I'm ok with the idea, but it's up to maintainers to coordinate this :)

Quoted 55 lines

- Arnaldo

Quoted 45 lines

Two perf test changes are also brought in as they need refactoring to account for the expr API change and it is expected they will land ahead of this.
https://lore.kernel.org/lkml/[email protected]/

Tested with 'perf test' and 'make -C tools/perf build-test'.

The hashmap change was originally part of an RFC:
https://lore.kernel.org/lkml/[email protected]/

v2. moves hashmap into tools/perf/util rather than libapi, to allow hashmap's libbpf symbols to be visible when built statically for testing.

Andrii Nakryiko (1):
libbpf: Fix memory leak and possible double-free in hashmap__clear

Ian Rogers (6):
libbpf hashmap: Remove unused #include
libbpf hashmap: Fix signedness warnings
tools lib/api: Copy libbpf hashmap to tools/perf/util perf test: Provide a subtest callback to ask for the reason for
skipping a subtest
perf test: Improve pmu event metric testing
perf expr: Migrate expr ids table to a hashmap

 tools/lib/bpf/hashmap.c         |  10 +-
 tools/lib/bpf/hashmap.h         |   1 -
 tools/perf/check-headers.sh     |   4 +
 tools/perf/tests/builtin-test.c |  18 ++-
 tools/perf/tests/expr.c         |  40 +++---
 tools/perf/tests/pmu-events.c   | 169 ++++++++++++++++++++++-
 tools/perf/tests/tests.h        |   4 +
 tools/perf/util/Build           |   4 +
 tools/perf/util/expr.c          | 129 +++++++++--------
 tools/perf/util/expr.h          |  26 ++--
 tools/perf/util/expr.y          |  22 +--
 tools/perf/util/hashmap.c       | 238 ++++++++++++++++++++++++++++++++
 tools/perf/util/hashmap.h       | 177 ++++++++++++++++++++++++
 tools/perf/util/metricgroup.c   |  87 ++++++------
 tools/perf/util/stat-shadow.c   |  49 ++++---
 15 files changed, 798 insertions(+), 180 deletions(-)
 create mode 100644 tools/perf/util/hashmap.c
 create mode 100644 tools/perf/util/hashmap.h
Quoted 3 lines

--
2.26.2.761.g0e0b3e54be-goog

--

- Arnaldo

#20 lore raw

<[email protected]>,Stephane Eranian <[email protected]>
From: Arnaldo Carvalho de Melo <[email protected]>
Message-ID: <[email protected]>

Quoted 22 lines · Andrii Nakryiko
Quoted 4 lines · Arnaldo Carvalho de Melo

Em Fri, May 15, 2020 at 09:50:00AM -0700, Ian Rogers escreveu:

Quoted 2 lines

Perf's expr code currently builds an array of strings then removes duplicates. The array is larger than necessary and has recently

been

Quoted 1 line
Quoted 1 line

increased in size. When this was done it was commented that a

hashmap

Quoted 4 lines
Quoted 4 lines

would be preferable.

libbpf has a hashmap but libbpf isn't currently required to build perf. To satisfy various concerns this change copies libbpf's

hashmap

Quoted 1 line
Quoted 1 line

into tools/perf/util, it then adds a check in perf that the two are

in

Quoted 6 lines
Quoted 6 lines

sync.

Andrii's patch to hashmap from bpf-next is brought into this set to fix issues with hashmap__clear.

Two minor changes to libbpf's hashmap are made that remove an

Quoted 8 lines

unused

Quoted 3 lines
Quoted 1 line

dependency and fix a compiler warning.

Andrii/Alexei/Daniel, what do you think about me merging these fixes

in my

Quoted 1 line

perf-tools-next branch?

I'm ok with the idea, but it's up to maintainers to coordinate this :)

Good to know, do I'll take all patches except the ones touching libppf, will just make sure the copy is done with the patches applied.

At some point they'll land in libbpf and the warning from check_headers.sh will be resolved.

Thanks,

- Arnaldo

Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
author reviewer
#21 lore raw
Quoted 34 lines · On Fri, May 15, 2020 at 2:19 PM <[email protected]> wrote:

<[email protected]>,Stephane Eranian <[email protected]>
From: Arnaldo Carvalho de Melo <[email protected]>
Message-ID: <[email protected]>

Quoted 26 lines · Andrii Nakryiko
Quoted 4 lines · Arnaldo Carvalho de Melo

Em Fri, May 15, 2020 at 09:50:00AM -0700, Ian Rogers escreveu:

Quoted 2 lines

Perf's expr code currently builds an array of strings then removes duplicates. The array is larger than necessary and has recently

been

Quoted 1 line
Quoted 1 line

increased in size. When this was done it was commented that a

hashmap

Quoted 4 lines
Quoted 4 lines

would be preferable.

libbpf has a hashmap but libbpf isn't currently required to build perf. To satisfy various concerns this change copies libbpf's

hashmap

Quoted 1 line
Quoted 1 line

into tools/perf/util, it then adds a check in perf that the two are

in

Quoted 6 lines
Quoted 6 lines

sync.

Andrii's patch to hashmap from bpf-next is brought into this set to fix issues with hashmap__clear.

Two minor changes to libbpf's hashmap are made that remove an

unused

Quoted 3 lines
Quoted 1 line

dependency and fix a compiler warning.

Andrii/Alexei/Daniel, what do you think about me merging these fixes

Quoted 8 lines
Quoted 4 lines

in my

Quoted 1 line

perf-tools-next branch?

I'm ok with the idea, but it's up to maintainers to coordinate this :)

Good to know, do I'll take all patches except the ones touching libppf, will just make sure the copy is done with the patches applied.

At some point they'll land in libbpf and the warning from check_headers.sh will be resolved.

So tools/perf/util's hashmap will be ahead of libbpf's, as without the fixes the perf build is broken by Werror. This will cause check_headers to warn in perf builds, which would usually mean our header was older than the source one, but in this case it means the opposite, we're waiting for the libbpf patches to merge. Aside from some interim warnings everything will resolve itself and Arnaldo avoids landing patches in libbpf that can interfere with bpf-next.

It takes some getting your head around but sounds good to me :-) I think the only workable alternatives would be to explore having a single version of the code in some kind of shared libhashmap or to implement another hashmap in libapi. I'd like to get the rest of this work unblocked and so it'd be nice to land this and we can always refactor later - I like Arnaldo's plan. Can a bpf maintainer make sure the hashmap changes get pulled into bpf-next?

Thanks!
Ian

Quoted 6 lines

Thanks,

- Arnaldo

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

maintainer
#22 lore raw
In reply to [email protected]

On 5/15/20 11:18 PM, [email protected] wrote:
[...]

Quoted 9 lines
Quoted 5 lines
Quoted 1 line

Andrii/Alexei/Daniel, what do you think about me merging these fixes

in my

Quoted 1 line

perf-tools-next branch?

I'm ok with the idea, but it's up to maintainers to coordinate this :)

Good to know, do I'll take all patches except the ones touching libppf, will just make sure the copy is done with the patches applied.

At some point they'll land in libbpf and the warning from check_headers.sh will be resolved.

Works for me, I've just taken in Ian's two libbpf ones into bpf-next.

Thanks everyone,
Daniel