Skip to contentSkip to conversation

perf lock contention: Add BPF support (v1)

PATCH Applied Namhyung Kim 2 participants 5 messages Jul 29, 2022 – Aug 1, 2022 perf events

lore
Share & tools
lore

Namhyung Kim

author maintainer
#0 lore raw

cover perf lock contention: Add BPF support (v1)

Hello,

This patchset adds -b/--use-bpf option and others to use BPF to collect kernel lock contention stats. With this option it doesn't require a separate perf lock record step. Basic filtering on cpu (with -a or -C option) and on task (with -p and --tid option) is supported as usual.

  $ sudo perf lock con -a -b sleep 1
     contended   total wait     max wait     avg wait         type   caller
            42    192.67 us     13.64 us      4.59 us     spinlock   queue_work_on+0x20
            23     85.54 us     10.28 us      3.72 us     spinlock   worker_thread+0x14a
             6     13.92 us      6.51 us      2.32 us        mutex   kernfs_iop_permission+0x30
             3     11.59 us     10.04 us      3.86 us        mutex   kernfs_dop_revalidate+0x3c
             1      7.52 us      7.52 us      7.52 us     spinlock   kthread+0x115
             1      7.24 us      7.24 us      7.24 us     rwlock:W   sys_epoll_wait+0x148
             2      7.08 us      3.99 us      3.54 us     spinlock   delayed_work_timer_fn+0x1b
             1      6.41 us      6.41 us      6.41 us     spinlock   idle_balance+0xa06
             2      2.50 us      1.83 us      1.25 us        mutex   kernfs_iop_lookup+0x2f
             1      1.71 us      1.71 us      1.71 us        mutex   kernfs_iop_getattr+0x2c

It seems my system had some contentions on the workqueue spinlock and the kernfs mutex.

The code is available at perf/lock-bpf-v1 branch in

git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Thanks,
Namhyung

Namhyung Kim (3):
perf lock: Pass machine pointer to is_lock_function() perf lock: Use BPF for lock contention analysis perf lock: Implement cpu and task filters for BPF

 tools/perf/Documentation/perf-lock.txt        |  22 ++
 tools/perf/Makefile.perf                      |   2 +-
 tools/perf/builtin-lock.c                     | 226 ++++++++----------
 tools/perf/util/Build                         |   1 +
 tools/perf/util/bpf_lock_contention.c         | 181 ++++++++++++++
 .../perf/util/bpf_skel/lock_contention.bpf.c  | 170 +++++++++++++
 tools/perf/util/lock-contention.h             | 140 +++++++++++
 7 files changed, 614 insertions(+), 128 deletions(-)
 create mode 100644 tools/perf/util/bpf_lock_contention.c
 create mode 100644 tools/perf/util/bpf_skel/lock_contention.bpf.c
 create mode 100644 tools/perf/util/lock-contention.h
Signature · 1 line
2.37.1.455.g008518b4e5-goog
author maintainer
#1 lore raw

PATCH 1/3 perf lock: Pass machine pointer to is_lock_function()

This is a preparation for later change to expose the function externally so that it can be used without the implicit session data.

---
 tools/perf/builtin-lock.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
tools/perf/builtin-lock.c +7 -8
index 041801d8b6ac..10b854315b7a 100644
@@ -947,10 +947,9 @@ static int report_lock_release_event(struct evsel *evsel,
 	return 0;
 }
 
-static bool is_lock_function(u64 addr)
+static bool is_lock_function(struct machine *machine, u64 addr)
 {
 	if (!sched_text_start) {
-		struct machine *machine = &session->machines.host;
 		struct map *kmap;
 		struct symbol *sym;
 
@@ -1002,6 +1001,7 @@ static int lock_contention_caller(struct evsel *evsel, struct perf_sample *sampl
 {
 	struct thread *thread;
 	struct callchain_cursor *cursor = &callchain_cursor;
+	struct machine *machine = &session->machines.host;
 	struct symbol *sym;
 	int skip = 0;
 	int ret;
@@ -1010,8 +1010,7 @@ static int lock_contention_caller(struct evsel *evsel, struct perf_sample *sampl
 	if (show_thread_stats)
 		return -1;
 
-	thread = machine__findnew_thread(&session->machines.host,
-					 -1, sample->pid);
+	thread = machine__findnew_thread(machine, -1, sample->pid);
 	if (thread == NULL)
 		return -1;
 
@@ -1038,7 +1037,7 @@ static int lock_contention_caller(struct evsel *evsel, struct perf_sample *sampl
 			goto next;
 
 		sym = node->ms.sym;
-		if (sym && !is_lock_function(node->ip)) {
+		if (sym && !is_lock_function(machine, node->ip)) {
 			struct map *map = node->ms.map;
 			u64 offset;
 
@@ -1060,13 +1059,13 @@ static int lock_contention_caller(struct evsel *evsel, struct perf_sample *sampl
 static u64 callchain_id(struct evsel *evsel, struct perf_sample *sample)
 {
 	struct callchain_cursor *cursor = &callchain_cursor;
+	struct machine *machine = &session->machines.host;
 	struct thread *thread;
 	u64 hash = 0;
 	int skip = 0;
 	int ret;
 
-	thread = machine__findnew_thread(&session->machines.host,
-					 -1, sample->pid);
+	thread = machine__findnew_thread(machine, -1, sample->pid);
 	if (thread == NULL)
 		return -1;
 
@@ -1091,7 +1090,7 @@ static u64 callchain_id(struct evsel *evsel, struct perf_sample *sample)
 		if (++skip <= CONTENTION_STACK_SKIP)
 			goto next;
 
-		if (node->ms.sym && is_lock_function(node->ip))
+		if (node->ms.sym && is_lock_function(machine, node->ip))
 			goto next;
 
 		hash ^= hash_long((unsigned long)node->ip, 64);
Signature · 1 line
2.37.1.455.g008518b4e5-goog
author maintainer
#2 lore raw

PATCH 2/3 perf lock: Use BPF for lock contention analysis

In reply to Namhyung Kim

Add -b/--use-bpf option to use BPF to collect lock contention stats. For simplicity it now runs system-wide and requires C-c to stop. Upcoming changes will add the usual filtering.

  $ sudo perf lock con -b
  ^C
   contended   total wait     max wait     avg wait         type   caller
          42    192.67 us     13.64 us      4.59 us     spinlock   queue_work_on+0x20
          23     85.54 us     10.28 us      3.72 us     spinlock   worker_thread+0x14a
           6     13.92 us      6.51 us      2.32 us        mutex   kernfs_iop_permission+0x30
           3     11.59 us     10.04 us      3.86 us        mutex   kernfs_dop_revalidate+0x3c
           1      7.52 us      7.52 us      7.52 us     spinlock   kthread+0x115
           1      7.24 us      7.24 us      7.24 us     rwlock:W   sys_epoll_wait+0x148
           2      7.08 us      3.99 us      3.54 us     spinlock   delayed_work_timer_fn+0x1b
           1      6.41 us      6.41 us      6.41 us     spinlock   idle_balance+0xa06
           2      2.50 us      1.83 us      1.25 us        mutex   kernfs_iop_lookup+0x2f
           1      1.71 us      1.71 us      1.71 us        mutex   kernfs_iop_getattr+0x2c
---
 tools/perf/Documentation/perf-lock.txt        |   5 +
 tools/perf/Makefile.perf                      |   2 +-
 tools/perf/builtin-lock.c                     | 164 ++++++------------
 tools/perf/util/Build                         |   1 +
 tools/perf/util/bpf_lock_contention.c         | 132 ++++++++++++++
 .../perf/util/bpf_skel/lock_contention.bpf.c  | 131 ++++++++++++++
 tools/perf/util/lock-contention.h             | 133 ++++++++++++++
 7 files changed, 453 insertions(+), 115 deletions(-)
 create mode 100644 tools/perf/util/bpf_lock_contention.c
 create mode 100644 tools/perf/util/bpf_skel/lock_contention.bpf.c
 create mode 100644 tools/perf/util/lock-contention.h
tools/perf/Documentation/perf-lock.txt +5 -0
index 8f4e34f924d5..b88bb72c77d4 100644
@@ -123,6 +123,11 @@ CONTENTION OPTIONS
 --threads::
 	Show per-thread lock contention stat
 
+-b::
+--use-bpf::
+	Use BPF program to collect lock contention stats instead of
+	using the input data.
+
 
 SEE ALSO
 --------
tools/perf/Makefile.perf +1 -1
index d2083a247f73..5053b563bf9c 100644
@@ -1028,7 +1028,7 @@ SKEL_TMP_OUT := $(abspath $(SKEL_OUT)/.tmp)
 SKELETONS := $(SKEL_OUT)/bpf_prog_profiler.skel.h
 SKELETONS += $(SKEL_OUT)/bperf_leader.skel.h $(SKEL_OUT)/bperf_follower.skel.h
 SKELETONS += $(SKEL_OUT)/bperf_cgroup.skel.h $(SKEL_OUT)/func_latency.skel.h
-SKELETONS += $(SKEL_OUT)/off_cpu.skel.h
+SKELETONS += $(SKEL_OUT)/off_cpu.skel.h $(SKEL_OUT)/lock_contention.skel.h
 SKELETONS += $(SKEL_OUT)/kwork_trace.skel.h
 
 $(SKEL_TMP_OUT) $(LIBBPF_OUTPUT):
tools/perf/builtin-lock.c +50 -114

242 lines not on this page · raw message

tools/perf/util/Build +1 -0
index 66ad30cf65ec..86fcd0a614fe 100644
@@ -149,6 +149,7 @@ perf-$(CONFIG_PERF_BPF_SKEL) += bpf_counter_cgroup.o
 perf-$(CONFIG_PERF_BPF_SKEL) += bpf_ftrace.o
 perf-$(CONFIG_PERF_BPF_SKEL) += bpf_off_cpu.o
 perf-$(CONFIG_PERF_BPF_SKEL) += bpf_kwork.o
+perf-$(CONFIG_PERF_BPF_SKEL) += bpf_lock_contention.o
 perf-$(CONFIG_BPF_PROLOGUE) += bpf-prologue.o
 perf-$(CONFIG_LIBELF) += symbol-elf.o
 perf-$(CONFIG_LIBELF) += probe-file.o
tools/perf/util/bpf_lock_contention.c new file +132 -0

132 lines not on this page · raw message

tools/perf/util/bpf_skel/lock_contention.bpf.c new file +131 -0

131 lines not on this page · raw message

tools/perf/util/lock-contention.h new file +133 -0

133 lines not on this page · raw message

Signature · 1 line
2.37.1.455.g008518b4e5-goog
author maintainer
#3 lore raw

PATCH 3/3 perf lock: Implement cpu and task filters for BPF

In reply to Namhyung Kim

Add -a/--all-cpus and -C/--cpu options for cpu filtering. Also -p/--pid and --tid options are added for task filtering. The short -t option is taken for --threads already. Tracking the command line workload is possible as well.

  $ sudo perf lock contention -a -b sleep 1
---
 tools/perf/Documentation/perf-lock.txt        | 17 ++++++
 tools/perf/builtin-lock.c                     | 55 ++++++++++++++++---
 tools/perf/util/bpf_lock_contention.c         | 51 ++++++++++++++++-
 .../perf/util/bpf_skel/lock_contention.bpf.c  | 41 +++++++++++++-
 tools/perf/util/lock-contention.h             | 11 +++-
 5 files changed, 162 insertions(+), 13 deletions(-)
tools/perf/Documentation/perf-lock.txt +17 -0
index b88bb72c77d4..7949d2e6891b 100644
@@ -128,6 +128,23 @@ CONTENTION OPTIONS
 	Use BPF program to collect lock contention stats instead of
 	using the input data.
 
+-a::
+--all-cpus::
+        System-wide collection from all CPUs.
+
+-C::
+--cpu::
+	Collect samples only on the list of CPUs provided. Multiple CPUs can be
+	provided as a comma-separated list with no space: 0,1. Ranges of CPUs
+	are specified with -: 0-2.  Default is to monitor all CPUs.
+
+-p::
+--pid=::
+	Record events on existing process ID (comma separated list).
+
+--tid=::
+        Record events on existing thread ID (comma separated list).
+
 
 SEE ALSO
 --------
tools/perf/builtin-lock.c +46 -9

115 lines not on this page · raw message

tools/perf/util/bpf_lock_contention.c +50 -1

77 lines not on this page · raw message

tools/perf/util/bpf_skel/lock_contention.bpf.c +40 -1

55 lines not on this page · raw message

tools/perf/util/lock-contention.h +9 -2
index c92db4a47d8d..092c84441f9f 100644
@@ -103,11 +103,13 @@ struct thread_stat {
 #define LCB_F_PERCPU	(1U << 4)
 #define LCB_F_MUTEX	(1U << 5)
 
+struct evlist;
 struct machine;
+struct target;
 
 #ifdef HAVE_BPF_SKEL
 
-int lock_contention_prepare(void);
+int lock_contention_prepare(struct evlist *evlist, struct target *target);
 int lock_contention_start(void);
 int lock_contention_stop(void);
 int lock_contention_read(struct machine *machine, struct hlist_head *head);
@@ -115,7 +117,12 @@ int lock_contention_finish(void);
 
 #else  /* !HAVE_BPF_SKEL */
 
-static inline int lock_contention_prepare(void) { return 0; }
+static inline int lock_contention_prepare(struct evlist *evlist __maybe_unused,
+					  struct target *target __maybe_unused)
+{
+	return 0;
+}
+
 static inline int lock_contention_start(void) { return 0; }
 static inline int lock_contention_stop(void) { return 0; }
 static inline int lock_contention_finish(void) { return 0; }
Signature · 1 line
2.37.1.455.g008518b4e5-goog
maintainer Applied
#4 lore raw
In reply to Namhyung Kim

Em Fri, Jul 29, 2022 at 01:07:53PM -0700, Namhyung Kim escreveu:

Quoted 22 lines

Hello,

This patchset adds -b/--use-bpf option and others to use BPF to collect kernel lock contention stats. With this option it doesn't require a separate perf lock record step. Basic filtering on cpu (with -a or -C option) and on task (with -p and --tid option) is supported as usual.

  $ sudo perf lock con -a -b sleep 1
     contended   total wait     max wait     avg wait         type   caller
            42    192.67 us     13.64 us      4.59 us     spinlock   queue_work_on+0x20
            23     85.54 us     10.28 us      3.72 us     spinlock   worker_thread+0x14a
             6     13.92 us      6.51 us      2.32 us        mutex   kernfs_iop_permission+0x30
             3     11.59 us     10.04 us      3.86 us        mutex   kernfs_dop_revalidate+0x3c
             1      7.52 us      7.52 us      7.52 us     spinlock   kthread+0x115
             1      7.24 us      7.24 us      7.24 us     rwlock:W   sys_epoll_wait+0x148
             2      7.08 us      3.99 us      3.54 us     spinlock   delayed_work_timer_fn+0x1b
             1      6.41 us      6.41 us      6.41 us     spinlock   idle_balance+0xa06
             2      2.50 us      1.83 us      1.25 us        mutex   kernfs_iop_lookup+0x2f
             1      1.71 us      1.71 us      1.71 us        mutex   kernfs_iop_getattr+0x2c
Quoted 6 lines

It seems my system had some contentions on the workqueue spinlock and the kernfs mutex.

The code is available at perf/lock-bpf-v1 branch in

git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Thanks, applied.

- Arnaldo

Quoted 23 lines

Thanks,
Namhyung

Namhyung Kim (3):
perf lock: Pass machine pointer to is_lock_function() perf lock: Use BPF for lock contention analysis perf lock: Implement cpu and task filters for BPF

 tools/perf/Documentation/perf-lock.txt        |  22 ++
 tools/perf/Makefile.perf                      |   2 +-
 tools/perf/builtin-lock.c                     | 226 ++++++++----------
 tools/perf/util/Build                         |   1 +
 tools/perf/util/bpf_lock_contention.c         | 181 ++++++++++++++
 .../perf/util/bpf_skel/lock_contention.bpf.c  | 170 +++++++++++++
 tools/perf/util/lock-contention.h             | 140 +++++++++++
 7 files changed, 614 insertions(+), 128 deletions(-)
 create mode 100644 tools/perf/util/bpf_lock_contention.c
 create mode 100644 tools/perf/util/bpf_skel/lock_contention.bpf.c
 create mode 100644 tools/perf/util/lock-contention.h
Signature · 1 line
2.37.1.455.g008518b4e5-goog
Signature · 1 line
- Arnaldo