perf utility question: "Sky high iTLB-load-misses"
Hello,
I was recently comparing the effect of using orphan section offloading of some data in the libqb project[*] and basically run into the same rather strange reports by perf as asked on StackOverflow:
https://stackoverflow.com/q/47107255
(except that I'm on Skylake processor).
Is iTLB-load-misses > 100% reported because of some deficiency of the platform, a bug in perf, or an expected behaviour?
Thanks and cheers.
Please, keep me in the loop, I am not subscribed to the list.
[*] comparing log_callsite_bench_section{full,less} from
https://github.com/ClusterLabs/libqb/pull/266/commits/a2b96193ec9a677e18824736c363df35ee0c68dd
Signature · 1 line
Jan (Poki)
Quoted 2 lines · Jan Pokorný
Is iTLB-load-misses > 100% reported because of some deficiency of the platform, a bug in perf, or an expected behaviour?
If you look into the kernel, the two events being measured are
[ C(RESULT_ACCESS) ] = 0x2085, /* ITLB_MISSES.STLB_HIT */ [ C(RESULT_MISS) ] = 0xe85, /* ITLB_MISSES.WALK_COMPLETED */
If you look this up in Intel Vol3b you can see that the "access" metric measures First level ITLB misses that hit in the Second-level TLB
Wheras the miss metric is a list of accesses from any level that caused a walk of the page tables.
So the events don't necessarily sound like they match up very well, so it's beleivable you will get odd results when trying to calculate percentages based on them.
Vince
Em Wed, Nov 08, 2017 at 11:55:02AM -0500, Vince Weaver escreveu:
Quoted 12 lines · Jan Pokorný
Quoted 3 lines
Is iTLB-load-misses > 100% reported because of some deficiency of the platform, a bug in perf, or an expected behaviour?
If you look into the kernel, the two events being measured are
[ C(RESULT_ACCESS) ] = 0x2085, /* ITLB_MISSES.STLB_HIT */ [ C(RESULT_MISS) ] = 0xe85, /* ITLB_MISSES.WALK_COMPLETED */If you look this up in Intel Vol3b you can see that the "access" metric measures First level ITLB misses that hit in the Second-level TLB
Quoted 6 lines
Wheras the miss metric is a list of accesses from any level that caused a walk of the page tables.
So the events don't necessarily sound like they match up very well, so it's beleivable you will get odd results when trying to calculate percentages based on them.
Andi, is this something you can help in figuring out a fix? Peter?
- Arnaldo
Quoted 13 lines · Arnaldo Carvalho de Melo
Em Wed, Nov 08, 2017 at 11:55:02AM -0500, Vince Weaver escreveu:
Quoted 12 lines · Jan Pokorný
Quoted 3 lines
Is iTLB-load-misses > 100% reported because of some deficiency of the platform, a bug in perf, or an expected behaviour?
If you look into the kernel, the two events being measured are
[ C(RESULT_ACCESS) ] = 0x2085, /* ITLB_MISSES.STLB_HIT */ [ C(RESULT_MISS) ] = 0xe85, /* ITLB_MISSES.WALK_COMPLETED */If you look this up in Intel Vol3b you can see that the "access" metric measures First level ITLB misses that hit in the Second-level TLB
Quoted 8 lines
Quoted 6 lines
Wheras the miss metric is a list of accesses from any level that caused a walk of the page tables.
So the events don't necessarily sound like they match up very well, so it's beleivable you will get odd results when trying to calculate percentages based on them.
Andi, is this something you can help in figuring out a fix? Peter?
See "perf list itlb"
itlb access could be replaced with instructions retired, but I suspect that is also not what you're looking for.
Only other alternative would be to disable the events.
-Andi
Quoted 21 lines · Andi Kleen
Quoted 20 lines · Arnaldo Carvalho de Melo
Em Wed, Nov 08, 2017 at 11:55:02AM -0500, Vince Weaver escreveu:
Quoted 18 lines · Jan Pokorný
Quoted 3 lines
Is iTLB-load-misses > 100% reported because of some deficiency of the platform, a bug in perf, or an expected behaviour?
If you look into the kernel, the two events being measured are
[ C(RESULT_ACCESS) ] = 0x2085, /* ITLB_MISSES.STLB_HIT */ [ C(RESULT_MISS) ] = 0xe85, /* ITLB_MISSES.WALK_COMPLETED */If you look this up in Intel Vol3b you can see that the "access" metric measures First level ITLB misses that hit in the Second-level TLB
Wheras the miss metric is a list of accesses from any level that caused a walk of the page tables.
So the events don't necessarily sound like they match up very well, so it's beleivable you will get odd results when trying to calculate percentages based on them.
Quoted 6 lines
Quoted 1 line
Andi, is this something you can help in figuring out a fix? Peter?
See "perf list itlb"
itlb access could be replaced with instructions retired, but I suspect that is also not what you're looking for.
Not sure I am getting the exact events right, but there are some results concentrating just on these 3:
$ perf stat -r 100 -e itlb_misses.stlb_hit \ -e frontend_retired.itlb_miss -e itlb_misses.walk_completed \ --log-fd=1 ./log_callsite_bench_sectionless 2>/dev/null
Quoted 5 lines
[...]
115,178 itlb_misses.stlb_hit ( +- 2.53% ) 46,199 frontend_retired.itlb_miss ( +- 1.93% ) 291,411 itlb_misses.walk_completed ( +- 37.72% )[...]
but the ratio is flapping for particular runs, from three subsequent ones:
Quoted 3 lines
86,124 itlb_misses.stlb_hit
31,595 frontend_retired.itlb_miss
5,509 itlb_misses.walk_completedvs.
Quoted 3 lines
97,578 itlb_misses.stlb_hit
33,673 frontend_retired.itlb_miss
15,961 itlb_misses.walk_completedvs.
Quoted 3 lines
121,994 itlb_misses.stlb_hit
43,408 frontend_retired.itlb_miss
124,473 itlb_misses.walk_completedI am by no mean knowledgeable in this field, didn't meant to make any real conclusions based on the results, and I now suspect there's a reason behind hiding iTLB-load-misses behind doubled -dd switch of the default execution mode. Was mainly curious why such obvious outliers on this Skylake machine were observed.
Quoted 1 line
Only other alternative would be to disable the events.
Perhaps depends on the target audience, whether it's rather amateurs like me or hardware-literate (I mean down to archicture manuals' level) experts that can finely reason about all these metrics :)
(Nevertheless, I think keeping the basic mode easily consumable by the former group is a noble goal.)
Signature · 1 line
Jan (Poki)
Vince mentioned that two values are compared which do not relate well. I would say the two possible approaches would then be:
(1) report absolute numbers only, no percentage.
(2) don't report it.
Quoted 4 lines · Bram Stolk
Vince mentioned that two values are compared which do not relate well. I would say the two possible approaches would then be:
(1) report absolute numbers only, no percentage.
(2) don't report it.
Well they possibly don't relate well. You would need an advanced degree in computer architecture and/or an NDA with Intel to know for sure what's really going on.
On modern procesors any event more complicated than "cycles" and "instructions" can be nearly impossible to understand, and even cycles has a lot of corner cases.
Vince