bpftool: add {i,d}tlb_misses support for bpftool profile
PATCH 1/1 bpftool: add {i,d}tlb_misses support for bpftool profile
Commit 47c09d6a9f67("bpftool: Introduce "prog profile" command") introduced "bpftool prog profile" command which can be used to profile bpf program with metrics like # of instructions,
This patch added support for itlb_misses and dtlb_misses. During an internal bpf program performance evaluation, I found these two metrics are also very useful. The following is an example output:
$ bpftool prog profile id 324 duration 3 cycles itlb_misses
1885029 run_cnt
5134686073 cycles
306893 itlb_misses
$ bpftool prog profile id 324 duration 3 cycles dtlb_misses 1827382 run_cnt
4943593648 cycles
5975636 dtlb_misses
$ bpftool prog profile id 324 duration 3 cycles llc_misses 1836527 run_cnt
5019612972 cycles
4161041 llc_missesFrom the above, we can see quite some dtlb misses, 3 dtlb misses perf prog run. This might be something worth further investigation.
--- tools/bpf/bpftool/prog.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-)
tools/bpf/bpftool/prog.c +30 -2
@@ -1717,11 +1717,39 @@ struct profile_metric { .ratio_desc = "LLC misses per million insns", .ratio_mul = 1e6, }, + { + .name = "itlb_misses", + .attr = { + .type = PERF_TYPE_HW_CACHE, + .config = +PERF_COUNT_HW_CACHE_ITLB | + (PERF_COUNT_HW_CACHE_OP_READ <<8 ) | + (PERF_COUNT_HW_CACHE_RESULT_MISS << 16), + .exclude_user =1 + }, + .ratio_metric = 2, + .ratio_desc = "itlb misses per million insns", + .ratio_mul = 1e6, + }, + { + .name = "dtlb_misses", + .attr = { + .type = PERF_TYPE_HW_CACHE, + .config = +PERF_COUNT_HW_CACHE_DTLB | + (PERF_COUNT_HW_CACHE_OP_READ <<8 ) | + (PERF_COUNT_HW_CACHE_RESULT_MISS << 16), + .exclude_user =1 + }, + .ratio_metric = 2, + .ratio_desc = "dtlb misses per million insns", + .ratio_mul = 1e6, + }, };static __u64 profile_total_count; -#define MAX_NUM_PROFILE_METRICS 4 +#define MAX_NUM_PROFILE_METRICS 6static int profile_parse_metrics (int argc,char **argv) {
@@ -2109,7 +2137,7 @@ static int do_help(int argc, char **argv)" struct_ops | fentry | fexit | freplace | sk_lookup }\n" " ATTACH_TYPE := { msg_verdict | stream_verdict | stream_parser |\n" " flow_dissector }\n" -" METRIC := { cycles | instructions | l1d_loads | llc_misses }\n" +" METRIC := { cycles | instructions | l1d_loads | llc_misses | itlb_misses | dtlb_misses }\n" " " HELP_SPEC_OPTIONS "\n" "" , bin_name, argv[-2]);
Signature · 1 line
2.24.1
Quoted 2 lines · Yonghong Song
[...]
Quoted 39 lines
Cc Song Liu <[email protected]>Signed-off-by Yonghong Song <[email protected]>---
tools/bpf/bpftool/prog.c| 32 ++++++++++++++++++++++++++++++--1 file changed, 30 insertions(+), 2 deletions(-)tools/bpf/bpftool/prog.c +24
-0@@ -1717,11 +1717,39 @@ struct profile_metric { .ratio_desc = "LLC misses per million insns", .ratio_mul = 1e6, }, + { + .name = "itlb_misses", + .attr = { + .type = PERF_TYPE_HW_CACHE, + .config = +PERF_COUNT_HW_CACHE_ITLB | + (PERF_COUNT_HW_CACHE_OP_READ <<8 ) | + (PERF_COUNT_HW_CACHE_RESULT_MISS << 16), + .exclude_user =1 + }, + .ratio_metric = 2, + .ratio_desc = "itlb misses per million insns", + .ratio_mul = 1e6, + }, + { + .name = "dtlb_misses", + .attr = { + .type = PERF_TYPE_HW_CACHE, + .config = +PERF_COUNT_HW_CACHE_DTLB | + (PERF_COUNT_HW_CACHE_OP_READ <<8 ) | + (PERF_COUNT_HW_CACHE_RESULT_MISS << 16), + .exclude_user =1 + },
tools/bpf/bpftool/prog.c struct profile_metric { · line 1752
Quoted 10 lines
tools/bpf/bpftool/prog.c +5
-1+ .ratio_metric = 2, + .ratio_desc = "dtlb misses per million insns", + .ratio_mul = 1e6, + }, };static __u64 profile_total_count; -#define MAX_NUM_PROFILE_METRICS 4 +#define MAX_NUM_PROFILE_METRICS 6
This change is not necessary. This is the max number of enabled metrics. We don't stop the perf counters, each enabled metric adds some error to the final reading. Therefore, we don't want to enable too many metrics in the same run. If we don't have a use case for more metrics in one run, let's keep the cap of 4 metrics.
If we do want to increase this, we should also increase MAX_NUM_MATRICS in profiler.bpf.c.
Other than this,
Thanks!
Quoted 61 lines · Song Liu
Quoted 2 lines · Yonghong Song
[...]
Quoted 39 lines
Cc Song Liu <[email protected]>Signed-off-by Yonghong Song <[email protected]>---
tools/bpf/bpftool/prog.c| 32 ++++++++++++++++++++++++++++++--1 file changed, 30 insertions(+), 2 deletions(-)tools/bpf/bpftool/prog.c +24
-0@@ -1717,11 +1717,39 @@ struct profile_metric { .ratio_desc = "LLC misses per million insns", .ratio_mul = 1e6, }, + { + .name = "itlb_misses", + .attr = { + .type = PERF_TYPE_HW_CACHE, + .config = +PERF_COUNT_HW_CACHE_ITLB | + (PERF_COUNT_HW_CACHE_OP_READ <<8 ) | + (PERF_COUNT_HW_CACHE_RESULT_MISS << 16), + .exclude_user =1 + }, + .ratio_metric = 2, + .ratio_desc = "itlb misses per million insns", + .ratio_mul = 1e6, + }, + { + .name = "dtlb_misses", + .attr = { + .type = PERF_TYPE_HW_CACHE, + .config = +PERF_COUNT_HW_CACHE_DTLB | + (PERF_COUNT_HW_CACHE_OP_READ <<8 ) | + (PERF_COUNT_HW_CACHE_RESULT_MISS << 16), + .exclude_user =1 + },tools/bpf/bpftool/prog.c struct profile_metric { · line 1752
Quoted 10 lines
tools/bpf/bpftool/prog.c +5
-1+ .ratio_metric = 2, + .ratio_desc = "dtlb misses per million insns", + .ratio_mul = 1e6, + }, };static __u64 profile_total_count; -#define MAX_NUM_PROFILE_METRICS 4 +#define MAX_NUM_PROFILE_METRICS 6This change is not necessary. This is the max number of enabled metrics. We don't stop the perf counters, each enabled metric adds some error to the final reading. Therefore, we don't want to enable too many metrics in the same run. If we don't have a use case for more metrics in one run, let's keep the cap of 4 metrics.
Quoted 2 lines
If we do want to increase this, we should also increase
MAX_NUM_MATRICSinprofiler.bpf.c.
Thanks for explanation. I missed that we also need to change MAX_NUM_MATRICS to really make it work. Ideally it would be best if we only need to change in one header and that header is shared by both
bpf program and non-bpf program but probably is a overkill here just
for a single macro.
Also, your concern makes sense. We do not want to enable too many metrics as this will cause counter sharing and may sacrifice accuracy. I will leave current max profile metrics 4 then.
Will respin v2.
Quoted 7 lines
Other than this,
Acked-by Song Liu <[email protected]>Thanks!