perf lock contention: Add -J/--inject-delay option
Subsystems (2)
Namhyung Kim
maintainerPATCH v2 1/1 perf lock contention: Add -J/--inject-delay option
This is to slow down lock acquistion (on contention locks) deliberately. A possible use case is to estimate impact on application performance by optimization of kernel locking behavior. By delaying the lock it can simulate the worse condition as a control group, and then compare with the current behavior as a optimized condition.
The syntax is 'time@function' and the time can have unit suffix like "us" and "ms". For example, I ran a simple test like below.
$ sudo perf lock con -abl -L tasklist_lock -- \
sh -c 'for i in $(seq 1000); do sleep 1 & done; wait'
contended total wait max wait avg wait address symbol92 1.18 ms 199.54 us 12.79 us ffffffff8a806080 tasklist_lock (rwlock)
The contention count was 92 and the average wait time was around 10 us. But if I add 100 usec of delay to the tasklist_lock,
$ sudo perf lock con -abl -L tasklist_lock -J 100us@tasklist_lock -- \
sh -c 'for i in $(seq 1000); do sleep 1 & done; wait'
contended total wait max wait avg wait address symbol190 15.67 ms 230.10 us 82.46 us ffffffff8a806080 tasklist_lock (rwlock)
The contention count increased and the average wait time was up closed to 100 usec. If I increase the delay even more,
$ sudo perf lock con -abl -L tasklist_lock -J 1ms@tasklist_lock -- \
sh -c 'for i in $(seq 1000); do sleep 1 & done; wait'
contended total wait max wait avg wait address symbol1002 2.80 s 3.01 ms 2.80 ms ffffffff8a806080 tasklist_lock (rwlock)
Now every sleep process had contention and the wait time was more than 1 msec. This is on my 4 CPU laptop so I guess one CPU has the lock while other 3 are waiting for it mostly.
For simplicity, it only supports global locks for now.
--- tools/perf/Documentation/perf-lock.txt | 11 +++ tools/perf/builtin-lock.c | 74 +++++++++++++++++++ tools/perf/util/bpf_lock_contention.c | 28 +++++++ .../perf/util/bpf_skel/lock_contention.bpf.c | 43 +++++++++++ tools/perf/util/lock-contention.h | 8 ++ 5 files changed, 164 insertions(+)
tools/perf/Documentation/perf-lock.txt +11 -0
@@ -216,6 +216,17 @@ CONTENTION OPTIONS --cgroup-filter=<value>:: Show lock contention only in the given cgroups (comma separated list). +-J:: +--inject-delay=<time@function>:: + Add delays to the given lock. It's added to the contention-end part so + that the (new) owner of the lock will be delayed. But by slowing down + the owner, the waiters will also be delayed as well. This is working + only with -b/--use-bpf. + + The 'time' is specified in nsec but it can have a unit suffix. Available + units are "ms" and "us". Note that it will busy-wait after it gets the + lock. Please use it at your own risk. + SEE ALSO --------
tools/perf/builtin-lock.c +74 -0
98 lines not on this page · raw message
tools/perf/util/bpf_lock_contention.c +28 -0
@@ -261,6 +261,27 @@ int lock_contention_prepare(struct lock_contention *con) skel->rodata->has_addr =1 ; } +/* resolve lock name in delays */ +if (con->nr_delays) { +struct symbol *sym; +struct map *kmap; + +for (i =0 ; i < con->nr_delays; i++) { + sym =machine__find_kernel_symbol_by_name (con->machine, + con->delays[i].sym, + &kmap); +if (sym ==NULL ) { +pr_warning ("ignore unknown symbol: %s\n" , + con->delays[i].sym); +continue ; + } + + con->delays[i].addr =map__unmap_ip (kmap, sym->start); + } + skel->rodata->lock_delay =1 ; +bpf_map__set_max_entries (skel->maps.lock_delays, con->nr_delays); + } +bpf_map__set_max_entries (skel->maps.cpu_filter, ncpus);bpf_map__set_max_entries (skel->maps.task_filter, ntasks);bpf_map__set_max_entries (skel->maps.type_filter, ntypes);
@@ -352,6 +373,13 @@ int lock_contention_prepare(struct lock_contention *con)bpf_map_update_elem (fd, &con->filters->cgrps[i], &val,BPF_ANY ); } +if (con->nr_delays) { + fd =bpf_map__fd (skel->maps.lock_delays); + +for (i =0 ; i < con->nr_delays; i++) +bpf_map_update_elem (fd, &con->delays[i].addr, &con->delays[i].time,BPF_ANY ); + } +if (con->aggr_mode ==LOCK_AGGR_CGROUP )read_all_cgroups (&con->cgroups);
tools/perf/util/bpf_skel/lock_contention.bpf.c +43 -0
73 lines not on this page · raw message
tools/perf/util/lock-contention.h +8 -0
@@ -18,6 +18,12 @@ struct lock_filter {char **slabs; }; +struct lock_delay { +char *sym; +unsigned long addr; +unsigned long time; +}; +struct lock_stat {struct hlist_node hash_entry;struct rb_node rb;/* used for sorting */
@@ -140,6 +146,7 @@ struct lock_contention {struct machine *machine;struct hlist_head *result;struct lock_filter *filters; +struct lock_delay *delays;struct lock_contention_fails fails;struct rb_root cgroups;void *btf;
@@ -149,6 +156,7 @@ struct lock_contention {int aggr_mode;int owner;int nr_filtered; +int nr_delays;bool save_callstack; };
Signature · 1 line
2.49.0.1015.ga840276032-goog
Arnaldo Carvalho de Melo
maintainerQuoted 5 lines · Namhyung Kim
This is to slow down lock acquistion (on contention locks) deliberately. A possible use case is to estimate impact on application performance by optimization of kernel locking behavior. By delaying the lock it can simulate the worse condition as a control group, and then compare with the current behavior as a optimized condition.
Thanks, tested and applied.
- Arnaldo