perf: Stream comparison
v7 cover perf: Stream comparison
Sometimes, a small change in a hot function reducing the cycles of this function, but the overall workload doesn't get faster. It is interesting where the cycles are moved to.
What it would like is to diff before/after streams. The stream is the branch history which is aggregated by the branch records from perf samples. For example, the callchains aggregated from the branch records. By browsing the hot stream, we can understand the hot code path.
By browsing the hot streams, we can understand the hot code path. By comparing the cycles variation of same streams between old perf data and new perf data, we can understand if the cycles are moved to other codes.
The before stream is the stream in perf.data.old. The after stream is the stream in perf.data.
Diffing before/after streams compares top N hottest streams between two perf data files.
If all entries of one stream in perf.data.old are fully matched with all entries of another stream in perf.data, we think two streams are matched, otherwise the streams are not matched.
For example,
cycles: 1, hits: 26.80% cycles: 1, hits: 27.30%
-------------------------- --------------------------
main div.c:39 main div.c:39
main div.c:44 main div.c:44
The above streams are matched and we can see for the same streams the cycles (1) are equal and the callchain hit percents are slightly changed (26.80% vs. 27.30%). That's expected.
Now let's see example.
perf record -b ... Generate perf.data.old with branch data
perf record -b ... Generate perf.data with branch data
perf diff --stream
[ Matched hot streams ]
hot chain pair 1:
cycles: 1, hits: 27.77% cycles: 1, hits: 9.24%
--------------------------- --------------------------
main div.c:39 main div.c:39
main div.c:44 main div.c:44
hot chain pair 2:
cycles: 34, hits: 20.06% cycles: 27, hits: 16.98%
--------------------------- --------------------------
__random_r random_r.c:360 __random_r random_r.c:360
__random_r random_r.c:388 __random_r random_r.c:388
__random_r random_r.c:388 __random_r random_r.c:388
__random_r random_r.c:380 __random_r random_r.c:380
__random_r random_r.c:357 __random_r random_r.c:357
__random random.c:293 __random random.c:293
__random random.c:293 __random random.c:293
__random random.c:291 __random random.c:291
__random random.c:291 __random random.c:291
__random random.c:291 __random random.c:291
__random random.c:288 __random random.c:288
rand rand.c:27 rand rand.c:27
rand rand.c:26 rand rand.c:26
rand@plt rand@plt
rand@plt rand@plt
compute_flag div.c:25 compute_flag div.c:25
compute_flag div.c:22 compute_flag div.c:22
main div.c:40 main div.c:40
main div.c:40 main div.c:40
main div.c:39 main div.c:39
hot chain pair 3:
cycles: 9, hits: 4.48% cycles: 6, hits: 4.51%
--------------------------- --------------------------
__random_r random_r.c:360 __random_r random_r.c:360
__random_r random_r.c:388 __random_r random_r.c:388
__random_r random_r.c:388 __random_r random_r.c:388
__random_r random_r.c:380 __random_r random_r.c:380[ Hot streams in old perf data only ]
hot chain 1:
cycles: 18, hits: 6.75%
--------------------------
__random_r random_r.c:360
__random_r random_r.c:388
__random_r random_r.c:388
__random_r random_r.c:380
__random_r random_r.c:357
__random random.c:293
__random random.c:293
__random random.c:291
__random random.c:291
__random random.c:291
__random random.c:288
rand rand.c:27
rand rand.c:26
rand@plt
rand@plt
compute_flag div.c:25
compute_flag div.c:22
main div.c:40hot chain 2:
cycles: 29, hits: 2.78%
--------------------------
compute_flag div.c:22
main div.c:40
main div.c:40
main div.c:39[ Hot streams in new perf data only ]
hot chain 1:
cycles: 4, hits: 4.54%
--------------------------
main div.c:42
compute_flag div.c:28hot chain 2:
cycles: 5, hits: 3.51%
--------------------------
main div.c:39
main div.c:44
main div.c:42
compute_flag div.c:28 v7:
---
Create a new struct evlist_streams which contains ev_streams and nr_evsel, so we don't need to pass nr_evsel in stream related functions.
Rename functions for better coding style.
v6:
---
Rebase to perf/core
v5:
---
1. Remove enum stream_type
2. Rebase to perf/core
v4:
---
The previous version is too big and very hard for review.
1. v4 removes the code which supports the source line mapping table and remove the source line based comparison. Now we only supports the basic functionality of stream comparison.
2. Refactor the code in a generic way.
v3:
---
v2 has 14 patches, it's hard to review. v3 is only 7 patches for basic stream comparison.
Jin Yao (7):
perf util: Create streams
perf util: Get the evsel_streams by evsel_idx perf util: Compare two streams
perf util: Link stream pair
perf util: Calculate the sum of total streams hits perf util: Report hot streams
perf diff: Support hot streams comparison
tools/perf/Documentation/perf-diff.txt | 4 + tools/perf/builtin-diff.c | 119 ++++++++- tools/perf/util/Build | 1 + tools/perf/util/callchain.c | 99 +++++++ tools/perf/util/callchain.h | 9 + tools/perf/util/stream.c | 342 +++++++++++++++++++++++++ tools/perf/util/stream.h | 41 +++ 7 files changed, 602 insertions(+), 13 deletions(-) create mode 100644 tools/perf/util/stream.c create mode 100644 tools/perf/util/stream.h
Signature · 1 line
2.17.1
PATCH v7 1/7 perf util: Create streams
We define the stream is the branch history which is aggregated by the branch records from perf samples. For example, the callchains aggregated from the branch records are considered as streams. By browsing the hot stream, we can understand the hot code path.
Now we only support the callchain for stream. For measuring the hot level for a stream, we use the callchain_node->hit, higher is hotter.
There may be many callchains sampled so we only focus on the top N hottest callchains. N is a user defined parameter or predefined default value (nr_streams_max).
This patch creates an evsel_streams array per event, and saves the top N hottest streams in a stream array.
So now we can get the per-event top N hottest streams.
---
v7:
- Create a new struct evlist_streams.
- Rename functions with appropriate prefix
v6:
- Rebase to perf/core
v5:
- Remove enum stram_type
- Rebase to perf/core
v4:
- Refactor the code
- Rename patch name from 'perf util: Create streams for managing top N hottest callchains' to 'perf util: Create streams'
v2:
- Use zfree in free_evsel_streams().
tools/perf/util/Build | 1 + tools/perf/util/stream.c | 164 +++++++++++++++++++++++++++++++++++++++ tools/perf/util/stream.h | 30 +++++++ 3 files changed, 195 insertions(+) create mode 100644 tools/perf/util/stream.c create mode 100644 tools/perf/util/stream.h
tools/perf/util/Build +1 -0
@@ -101,6 +101,7 @@ perf-y += call-path.o perf-y += rwsem.o perf-y += thread-stack.o perf-y += spark.o +perf-y += stream.o perf-$(CONFIG_AUXTRACE) += auxtrace.o perf-$(CONFIG_AUXTRACE) += intel-pt-decoder/ perf-$(CONFIG_AUXTRACE) += intel-pt.o
tools/perf/util/stream.c new file +164 -0
164 lines not on this page · raw message
tools/perf/util/stream.h new file +30 -0
@@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __PERF_STREAM_H +#define __PERF_STREAM_H + +#include "callchain.h" + +struct stream { +struct callchain_node *cnode; +}; + +struct evsel_streams { +struct stream *streams; +int nr_streams_max; +int nr_streams; +int evsel_idx; +}; + +struct evlist_streams { +struct evsel_streams *ev_streams; +int nr_evsel; +}; + +struct evlist ; + +void evlist_streams__delete (struct evlist_streams *els); + +struct evlist_streams *evlist__create_streams (struct evlist *evlist, +int nr_streams_max); + +#endif /* __PERF_STREAM_H */
Signature · 1 line
2.17.1
PATCH v7 2/7 perf util: Get the evsel_streams by evsel_idx
In reply to Jin YaoIn previous patch, we have created evsel_streams array
This patch returns the specified evsel_streams according to the evsel_idx.
---
v7:
- Rename functions:
evsel_streams__entry
v6:
- Rebase to perf/core
v5:
- Rebase to perf/core
v4:
- Rename the patch from 'perf util: Return per-event callchain streams' to 'perf util: Get the evsel_streams by evsel_idx'
tools/perf/util/stream.c | 13 +++++++++++++ tools/perf/util/stream.h | 3 +++ 2 files changed, 16 insertions(+)
tools/perf/util/stream.c +13 -0
@@ -162,3 +162,16 @@ struct evlist_streams *evlist__create_streams(struct evlist *evlist,return els; } + +struct evsel_streams *evsel_streams__entry (struct evlist_streams *els, +int evsel_idx) +{ +struct evsel_streams *es = els->ev_streams; + +for (int i =0 ; i < els->nr_evsel; i++) { +if (es[i].evsel_idx == evsel_idx) +return &es[i]; + } + +return NULL ; +}
tools/perf/util/stream.h +3 -0
@@ -27,4 +27,7 @@ void evlist_streams__delete(struct evlist_streams *els);struct evlist_streams *evlist__create_streams (struct evlist *evlist,int nr_streams_max); +struct evsel_streams *evsel_streams__entry (struct evlist_streams *els, +int evsel_idx); +#endif /* __PERF_STREAM_H */
Signature · 1 line
2.17.1
PATCH v7 3/7 perf util: Compare two streams
In reply to Jin YaoStream is the branch history which is aggregated by the branch records from perf samples. Now we support the callchain as stream.
If the callchain entries of one stream are fully matched with the callchain entries of another stream, we think two streams are matched.
For example,
cycles: 1, hits: 26.80% cycles: 1, hits: 27.30%
----------------------- -----------------------
main div.c:39 main div.c:39
main div.c:44 main div.c:44
Above two streams are matched (we don't consider the case that source code is changed).
The matching logic is, compare the chain string first. If it's not matched, fallback to dso address comparison.
---
v7:
- No change
v6:
- Rebase to perf/core
v5:
- Remove enum stream_type
- Rebase to perf/core
v4:
- Remove original source line comparison code.
tools/perf/util/callchain.c | 54 +++++++++++++++++++++++++++++++++++++ tools/perf/util/callchain.h | 4 +++ 2 files changed, 58 insertions(+)
tools/perf/util/callchain.c +54 -0
@@ -1613,3 +1613,57 @@ void callchain_param_setup(u64 sample_type) callchain_param.record_mode =CALLCHAIN_FP ; } } + +static bool chain_match (struct callchain_list *base_chain, +struct callchain_list *pair_chain) +{ +enum match_result match; + + match =match_chain_strings (base_chain->srcline, + pair_chain->srcline); +if (match !=MATCH_ERROR ) +return match ==MATCH_EQ ; + + match =match_chain_dso_addresses (base_chain->ms.map, + base_chain->ip, + pair_chain->ms.map, + pair_chain->ip); + +return match ==MATCH_EQ ; +} + +bool callchain_cnode_matched (struct callchain_node *base_cnode, +struct callchain_node *pair_cnode) +{ +struct callchain_list *base_chain, *pair_chain; +bool match = false; + + pair_chain =list_first_entry (&pair_cnode->val, + struct callchain_list, + list); + +list_for_each_entry (base_chain, &base_cnode->val, list) { +if (&pair_chain->list == &pair_cnode->val) +return false; + +if (!base_chain->srcline || !pair_chain->srcline) { + pair_chain =list_next_entry (pair_chain, list); +continue ; + } + + match =chain_match (base_chain, pair_chain); +if (!match) +return false; + + pair_chain =list_next_entry (pair_chain, list); + } + +/* +* Say chain1 is ABC, chain2 is ABCD, we consider they are +* not fully matched. +*/ +if (pair_chain && (&pair_chain->list != &pair_cnode->val)) +return false; + +return match; +}
tools/perf/util/callchain.h +4 -0
@@ -298,4 +298,8 @@ int callchain_branch_counts(struct callchain_root *root, u64 *abort_count, u64 *cycles_count);void callchain_param_setup (u64 sample_type); + +bool callchain_cnode_matched (struct callchain_node *base_cnode, +struct callchain_node *pair_cnode); +#endif /* __PERF_CALLCHAIN_H */
Signature · 1 line
2.17.1
PATCH v7 4/7 perf util: Link stream pair
In reply to Jin YaoIn previous patch, we have created an evsel_streams for one event, and top N hottest streams will be saved in a stream array in evsel_streams.
This patch compares total streams among two evsel_streams.
Once two streams are fully matched, they will be linked as a pair. From the pair, we can know which streams are matched.
---
v7:
- Rename functions with 'stream__' prefix.
v6:
- Rebase to perf/core
v5:
- Remove enum stream_type
v4:
- New patch in v4.
tools/perf/util/stream.c | 40 ++++++++++++++++++++++++++++++++++++++++ tools/perf/util/stream.h | 4 ++++ 2 files changed, 44 insertions(+)
tools/perf/util/stream.c +40 -0
@@ -175,3 +175,43 @@ struct evsel_streams *evsel_streams__entry(struct evlist_streams *els,return NULL ; } + +static struct stream *stream__callchain_match (struct stream *base_stream, +struct evsel_streams *es_pair) +{ +for (int i =0 ; i < es_pair->nr_streams; i++) { +struct stream *pair_stream = &es_pair->streams[i]; + +if (callchain_cnode_matched (base_stream->cnode, + pair_stream->cnode)) { +return pair_stream; + } + } + +return NULL ; +} + +static struct stream *stream__match (struct stream *base_stream, +struct evsel_streams *es_pair) +{ +return stream__callchain_match (base_stream, es_pair); +} + +static void stream__link (struct stream *base_stream,struct stream *pair_stream) +{ + base_stream->pair_cnode = pair_stream->cnode; + pair_stream->pair_cnode = base_stream->cnode; +} + +void evsel_streams__match (struct evsel_streams *es_base, +struct evsel_streams *es_pair) +{ +for (int i =0 ; i < es_base->nr_streams; i++) { +struct stream *base_stream = &es_base->streams[i]; +struct stream *pair_stream; + + pair_stream =stream__match (base_stream, es_pair); +if (pair_stream) +stream__link (base_stream, pair_stream); + } +}
tools/perf/util/stream.h +4 -0
@@ -6,6 +6,7 @@struct stream {struct callchain_node *cnode; +struct callchain_node *pair_cnode; };struct evsel_streams {
@@ -30,4 +31,7 @@ struct evlist_streams *evlist__create_streams(struct evlist *evlist,struct evsel_streams *evsel_streams__entry (struct evlist_streams *els,int evsel_idx); +void evsel_streams__match (struct evsel_streams *es_base, +struct evsel_streams *es_pair); +#endif /* __PERF_STREAM_H */
Signature · 1 line
2.17.1
PATCH v7 5/7 perf util: Calculate the sum of total streams hits
In reply to Jin YaoWe have used callchain_node->hit to measure the hot level of one stream. This patch calculates the sum of hits of total streams.
Thus in next patch, we can use following formula to report hot percent for one stream.
hot percent = callchain_node->hit / sum of total hits
---
v7:
- No change
v6:
- Rebase to perf/core
v5:
- Rebase to perf/core
v4:
- No functional change.
v2:
- Combine the variable decl line with its initial assignment in total_callchain_hits().
tools/perf/util/callchain.c | 32 ++++++++++++++++++++++++++++++++ tools/perf/util/callchain.h | 3 +++ tools/perf/util/stream.c | 2 ++ tools/perf/util/stream.h | 1 + 4 files changed, 38 insertions(+)
tools/perf/util/callchain.c +32 -0
@@ -1667,3 +1667,35 @@ bool callchain_cnode_matched(struct callchain_node *base_cnode,return match; } + +static u64 count_callchain_hits (struct hist_entry *he) +{ +struct rb_root *root = &he->sorted_chain; +struct rb_node *rb_node =rb_first (root); +struct callchain_node *node; +u64 chain_hits =0 ; + +while (rb_node) { + node = rb_entry(rb_node, struct callchain_node, rb_node); + chain_hits += node->hit; + rb_node =rb_next (rb_node); + } + +return chain_hits; +} + +u64 callchain_total_hits (struct hists *hists) +{ +struct rb_node *next =rb_first_cached (&hists->entries); +u64 chain_hits =0 ; + +while (next) { + struct hist_entry *he = rb_entry(next, struct hist_entry, + rb_node); + + chain_hits +=count_callchain_hits (he); + next =rb_next (&he->rb_node); + } + +return chain_hits; +}
tools/perf/util/callchain.h +3 -0
@@ -13,6 +13,7 @@ struct ip_callchain;struct map ;struct perf_sample ;struct thread ; +struct hists ;#define HELP_PAD "\t\t\t\t"
@@ -302,4 +303,6 @@ void callchain_param_setup(u64 sample_type);bool callchain_cnode_matched (struct callchain_node *base_cnode,struct callchain_node *pair_cnode); +u64 callchain_total_hits (struct hists *hists); +#endif /* __PERF_CALLCHAIN_H */
tools/perf/util/stream.c +2 -0
@@ -121,6 +121,8 @@ static void init_hot_callchain(struct hists *hists, struct evsel_streams *es)update_hot_callchain (he, es); next =rb_next (&he->rb_node); } + + es->streams_hits =callchain_total_hits (hists); } static int evlist__init_callchain_streams(struct evlist *evlist,
tools/perf/util/stream.h +1 -0
@@ -14,6 +14,7 @@ struct evsel_streams {int nr_streams_max;int nr_streams;int evsel_idx; +u64 streams_hits; };struct evlist_streams {
Signature · 1 line
2.17.1
PATCH v7 6/7 perf util: Report hot streams
In reply to Jin YaoWe show the streams separately. They are divided into different sections.
1. "Matched hot streams"
2. "Hot streams in old perf data only"
3. "Hot streams in new perf data only".
For each stream, we report the cycles and hot percent (hits%).
For example,
cycles: 2, hits: 4.08%
--------------------------
main div.c:42
compute_flag div.c:28---
v7:
- Rename functions with 'evsel_streams__' prefix
v6:
- Rebase to perf/core
v5:
- Rebase to perf/core
v4:
- Remove "Hot chains in old perf data but source line changed in new perf data"
tools/perf/util/callchain.c | 13 ++++ tools/perf/util/callchain.h | 2 + tools/perf/util/stream.c | 123 ++++++++++++++++++++++++++++++++++++ tools/perf/util/stream.h | 3 + 4 files changed, 141 insertions(+)
tools/perf/util/callchain.c +13 -0
@@ -1699,3 +1699,16 @@ u64 callchain_total_hits(struct hists *hists)return chain_hits; } + +s64 callchain_avg_cycles (struct callchain_node *cnode) +{ +struct callchain_list *chain; +s64 cycles =0 ; + +list_for_each_entry (chain, &cnode->val, list) { +if (chain->srcline && chain->branch_count) + cycles += chain->cycles_count / chain->branch_count; + } + +return cycles; +}
tools/perf/util/callchain.h +2 -0
@@ -305,4 +305,6 @@ bool callchain_cnode_matched(struct callchain_node *base_cnode,u64 callchain_total_hits (struct hists *hists); +s64 callchain_avg_cycles (struct callchain_node *cnode); +#endif /* __PERF_CALLCHAIN_H */
tools/perf/util/stream.c +123 -0
126 lines not on this page · raw message
tools/perf/util/stream.h +3 -0
@@ -35,4 +35,7 @@ struct evsel_streams *evsel_streams__entry(struct evlist_streams *els,void evsel_streams__match (struct evsel_streams *es_base,struct evsel_streams *es_pair); +void evsel_streams__report (struct evsel_streams *es_base, +struct evsel_streams *es_pair); +#endif /* __PERF_STREAM_H */
Signature · 1 line
2.17.1
PATCH v7 7/7 perf diff: Support hot streams comparison
In reply to Jin YaoThis patch enables perf-diff with "--stream" option.
"--stream": Enable hot streams comparison
Now let's see example.
perf record -b ... Generate perf.data.old with branch data
perf record -b ... Generate perf.data with branch data
perf diff --stream
[ Matched hot streams ]
hot chain pair 1:
cycles: 1, hits: 27.77% cycles: 1, hits: 9.24%
--------------------------- --------------------------
main div.c:39 main div.c:39
main div.c:44 main div.c:44
hot chain pair 2:
cycles: 34, hits: 20.06% cycles: 27, hits: 16.98%
--------------------------- --------------------------
__random_r random_r.c:360 __random_r random_r.c:360
__random_r random_r.c:388 __random_r random_r.c:388
__random_r random_r.c:388 __random_r random_r.c:388
__random_r random_r.c:380 __random_r random_r.c:380
__random_r random_r.c:357 __random_r random_r.c:357
__random random.c:293 __random random.c:293
__random random.c:293 __random random.c:293
__random random.c:291 __random random.c:291
__random random.c:291 __random random.c:291
__random random.c:291 __random random.c:291
__random random.c:288 __random random.c:288
rand rand.c:27 rand rand.c:27
rand rand.c:26 rand rand.c:26
rand@plt rand@plt
rand@plt rand@plt
compute_flag div.c:25 compute_flag div.c:25
compute_flag div.c:22 compute_flag div.c:22
main div.c:40 main div.c:40
main div.c:40 main div.c:40
main div.c:39 main div.c:39
hot chain pair 3:
cycles: 9, hits: 4.48% cycles: 6, hits: 4.51%
--------------------------- --------------------------
__random_r random_r.c:360 __random_r random_r.c:360
__random_r random_r.c:388 __random_r random_r.c:388
__random_r random_r.c:388 __random_r random_r.c:388
__random_r random_r.c:380 __random_r random_r.c:380[ Hot streams in old perf data only ]
hot chain 1:
cycles: 18, hits: 6.75%
--------------------------
__random_r random_r.c:360
__random_r random_r.c:388
__random_r random_r.c:388
__random_r random_r.c:380
__random_r random_r.c:357
__random random.c:293
__random random.c:293
__random random.c:291
__random random.c:291
__random random.c:291
__random random.c:288
rand rand.c:27
rand rand.c:26
rand@plt
rand@plt
compute_flag div.c:25
compute_flag div.c:22
main div.c:40hot chain 2:
cycles: 29, hits: 2.78%
--------------------------
compute_flag div.c:22
main div.c:40
main div.c:40
main div.c:39[ Hot streams in new perf data only ]
hot chain 1:
cycles: 4, hits: 4.54%
--------------------------
main div.c:42
compute_flag div.c:28hot chain 2:
cycles: 5, hits: 3.51%
--------------------------
main div.c:39
main div.c:44
main div.c:42
compute_flag div.c:28---
v7:
- Use new struct evlist_streams in data__file.
- Free the streams in data__free.
v6:
- Rebase to perf/core
v5:
- Remove enum stream_type
- Rebase to perf/core
v4:
- Remove the "--before" and "--after" options since they are for source line based comparison. In this patchset, we will not support source line based comparison.
tools/perf/Documentation/perf-diff.txt | 4 + tools/perf/builtin-diff.c | 119 ++++++++++++++++++++++--- 2 files changed, 110 insertions(+), 13 deletions(-)
tools/perf/Documentation/perf-diff.txt +4 -0
@@ -182,6 +182,10 @@ OPTIONS --tid=:: Only diff samples for given thread ID (comma separated list). +--stream:: + Enable hot streams comparison. Stream can be a callchain which is + aggregated by the branch records from samples. + COMPARISON ---------- The comparison is governed by the baseline file. The baseline perf.data
tools/perf/builtin-diff.c +106 -13
200 lines not on this page · raw message
Signature · 1 line
2.17.1