Skip to contentSkip to conversation

bpftool: add {i,d}tlb_misses support for bpftool profile

PATCH Superseded Yonghong Song 2 participants 3 messages Nov 19, 2020

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bpftool: add {i,d}tlb_misses support for bpftool profile

Commit 47c09d6a9f67("bpftool: Introduce "prog profile" command") introduced "bpftool prog profile" command which can be used to profile bpf program with metrics like # of instructions,

This patch added support for itlb_misses and dtlb_misses. During an internal bpf program performance evaluation, I found these two metrics are also very useful. The following is an example output:

 $ bpftool prog profile id 324 duration 3 cycles itlb_misses
           1885029 run_cnt
        5134686073 cycles
            306893 itlb_misses
 $ bpftool prog profile id 324 duration 3 cycles dtlb_misses
           1827382 run_cnt
        4943593648 cycles
           5975636 dtlb_misses
 $ bpftool prog profile id 324 duration 3 cycles llc_misses
           1836527 run_cnt
        5019612972 cycles
           4161041 llc_misses

From the above, we can see quite some dtlb misses, 3 dtlb misses perf prog run. This might be something worth further investigation.

---
 tools/bpf/bpftool/prog.c | 32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)
tools/bpf/bpftool/prog.c +30 -2
index acdb2c245f0a..e33f27b950a5 100644
@@ -1717,11 +1717,39 @@ struct profile_metric {
 		.ratio_desc = "LLC misses per million insns",
 		.ratio_mul = 1e6,
 	},
+	{
+		.name = "itlb_misses",
+		.attr = {
+			.type = PERF_TYPE_HW_CACHE,
+			.config =
+				PERF_COUNT_HW_CACHE_ITLB |
+				(PERF_COUNT_HW_CACHE_OP_READ << 8) |
+				(PERF_COUNT_HW_CACHE_RESULT_MISS << 16),
+			.exclude_user = 1
+		},
+		.ratio_metric = 2,
+		.ratio_desc = "itlb misses per million insns",
+		.ratio_mul = 1e6,
+	},
+	{
+		.name = "dtlb_misses",
+		.attr = {
+			.type = PERF_TYPE_HW_CACHE,
+			.config =
+				PERF_COUNT_HW_CACHE_DTLB |
+				(PERF_COUNT_HW_CACHE_OP_READ << 8) |
+				(PERF_COUNT_HW_CACHE_RESULT_MISS << 16),
+			.exclude_user = 1
+		},
+		.ratio_metric = 2,
+		.ratio_desc = "dtlb misses per million insns",
+		.ratio_mul = 1e6,
+	},
 };
 
 static __u64 profile_total_count;
 
-#define MAX_NUM_PROFILE_METRICS 4
+#define MAX_NUM_PROFILE_METRICS 6
 
 static int profile_parse_metrics(int argc, char **argv)
 {
@@ -2109,7 +2137,7 @@ static int do_help(int argc, char **argv)
 		"                 struct_ops | fentry | fexit | freplace | sk_lookup }\n"
 		"       ATTACH_TYPE := { msg_verdict | stream_verdict | stream_parser |\n"
 		"                        flow_dissector }\n"
-		"       METRIC := { cycles | instructions | l1d_loads | llc_misses }\n"
+		"       METRIC := { cycles | instructions | l1d_loads | llc_misses | itlb_misses | dtlb_misses }\n"
 		"       " HELP_SPEC_OPTIONS "\n"
 		"",
 		bin_name, argv[-2]);
Signature · 1 line
2.24.1
Acked-by on v1 1/1
#1 lore raw
Quoted 2 lines · Yonghong Song

[...]

Quoted 39 lines

---
tools/bpf/bpftool/prog.c | 32 ++++++++++++++++++++++++++++++--

1 file changed, 30 insertions(+), 2 deletions(-)
tools/bpf/bpftool/prog.c +24 -0
index acdb2c245f0a..e33f27b950a5 100644
@@ -1717,11 +1717,39 @@ struct profile_metric {
 		.ratio_desc = "LLC misses per million insns",
 		.ratio_mul = 1e6,
 	},
+	{
+		.name = "itlb_misses",
+		.attr = {
+			.type = PERF_TYPE_HW_CACHE,
+			.config =
+				PERF_COUNT_HW_CACHE_ITLB |
+				(PERF_COUNT_HW_CACHE_OP_READ << 8) |
+				(PERF_COUNT_HW_CACHE_RESULT_MISS << 16),
+			.exclude_user = 1
+		},
+		.ratio_metric = 2,
+		.ratio_desc = "itlb misses per million insns",
+		.ratio_mul = 1e6,
+	},
+	{
+		.name = "dtlb_misses",
+		.attr = {
+			.type = PERF_TYPE_HW_CACHE,
+			.config =
+				PERF_COUNT_HW_CACHE_DTLB |
+				(PERF_COUNT_HW_CACHE_OP_READ << 8) |
+				(PERF_COUNT_HW_CACHE_RESULT_MISS << 16),
+			.exclude_user = 1
+		},

tools/bpf/bpftool/prog.c struct profile_metric { · line 1752

Quoted 10 lines
tools/bpf/bpftool/prog.c +5 -1
+		.ratio_metric = 2,
+		.ratio_desc = "dtlb misses per million insns",
+		.ratio_mul = 1e6,
+	},
 };
 
 static __u64 profile_total_count;
 
-#define MAX_NUM_PROFILE_METRICS 4
+#define MAX_NUM_PROFILE_METRICS 6

This change is not necessary. This is the max number of enabled metrics. We don't stop the perf counters, each enabled metric adds some error to the final reading. Therefore, we don't want to enable too many metrics in the same run. If we don't have a use case for more metrics in one run, let's keep the cap of 4 metrics.

If we do want to increase this, we should also increase MAX_NUM_MATRICS in profiler.bpf.c.

Other than this,

Thanks!

#2 lore raw
Quoted 61 lines · Song Liu
Quoted 2 lines · Yonghong Song

[...]

Quoted 39 lines

---
tools/bpf/bpftool/prog.c | 32 ++++++++++++++++++++++++++++++--

1 file changed, 30 insertions(+), 2 deletions(-)
tools/bpf/bpftool/prog.c +24 -0
index acdb2c245f0a..e33f27b950a5 100644
@@ -1717,11 +1717,39 @@ struct profile_metric {
 		.ratio_desc = "LLC misses per million insns",
 		.ratio_mul = 1e6,
 	},
+	{
+		.name = "itlb_misses",
+		.attr = {
+			.type = PERF_TYPE_HW_CACHE,
+			.config =
+				PERF_COUNT_HW_CACHE_ITLB |
+				(PERF_COUNT_HW_CACHE_OP_READ << 8) |
+				(PERF_COUNT_HW_CACHE_RESULT_MISS << 16),
+			.exclude_user = 1
+		},
+		.ratio_metric = 2,
+		.ratio_desc = "itlb misses per million insns",
+		.ratio_mul = 1e6,
+	},
+	{
+		.name = "dtlb_misses",
+		.attr = {
+			.type = PERF_TYPE_HW_CACHE,
+			.config =
+				PERF_COUNT_HW_CACHE_DTLB |
+				(PERF_COUNT_HW_CACHE_OP_READ << 8) |
+				(PERF_COUNT_HW_CACHE_RESULT_MISS << 16),
+			.exclude_user = 1
+		},

tools/bpf/bpftool/prog.c struct profile_metric { · line 1752

Quoted 10 lines
tools/bpf/bpftool/prog.c +5 -1
+		.ratio_metric = 2,
+		.ratio_desc = "dtlb misses per million insns",
+		.ratio_mul = 1e6,
+	},
 };
 
 static __u64 profile_total_count;
 
-#define MAX_NUM_PROFILE_METRICS 4
+#define MAX_NUM_PROFILE_METRICS 6

This change is not necessary. This is the max number of enabled metrics. We don't stop the perf counters, each enabled metric adds some error to the final reading. Therefore, we don't want to enable too many metrics in the same run. If we don't have a use case for more metrics in one run, let's keep the cap of 4 metrics.

Quoted 2 lines

If we do want to increase this, we should also increase MAX_NUM_MATRICS in profiler.bpf.c.

Thanks for explanation. I missed that we also need to change MAX_NUM_MATRICS to really make it work. Ideally it would be best if we only need to change in one header and that header is shared by both
bpf program and non-bpf program but probably is a overkill here just
for a single macro.

Also, your concern makes sense. We do not want to enable too many metrics as this will cause counter sharing and may sacrifice accuracy. I will leave current max profile metrics 4 then.

Will respin v2.

Quoted 7 lines

Other than this,

Thanks!