Skip to contentSkip to conversation

btrfs: trace: Trace events for btrfs tree locking

PATCH In review Qu Wenruo 2 participants 4 messages Apr 15, 2019

Subsystems (2)
lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

v3 cover btrfs: trace: Trace events for btrfs tree locking

This branch can be fetched from github:
https://github.com/adam900710/linux/tree/trace_locking
Which is based on v5.1-rc4 tag.

This patchset will introduce the following trace events for corresponding functions:
btrfs_tree_lock (sleepable)
btrfs_tree_unlock
btrfs_tree_read_lock (sleepable)
btrfs_tree_read_unlock
btrfs_tree_read_unlock_blocking btrfs_set_lock_blocking_read
btrfs_set_lock_blocking_write btrfs_clear_lock_blocking_read btrfs_clear_lock_blocking_write
btrfs_try_tree_read_lock
btrfs_try_tree_write_lock
btrfs_tree_read_lock_atomic

The two sleepable ones will have extra info like start_ns, end_ns, diff_ns.
Those two can be specially useful for tree lock wait time. There is tool to utilize those two events already:
https://github.com/adam900710/btrfs-profiler/blob/master/tree_lock_wait.py

The non-sleepable can be useful for possible user space based lock leakage/dead lock detector.

Changelog:
v2:
- Hide all ktime_get_ns() call out of trace events
So there will be no overhead if trace events are not enabled.

v3:
- More output for human to take a glance
- More trace events for later bcc usage
Like lock leakage and deadlock detection.

Qu Wenruo (2):
btrfs: trace: Introduce trace events for sleepable tree lock
btrfs: trace: Introduce trace events for all btrfs tree locking events

 fs/btrfs/locking.c           | 22 +++++++++
 include/trace/events/btrfs.h | 86 ++++++++++++++++++++++++++++++++++++
 2 files changed, 108 insertions(+)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH v3 1/2 btrfs: trace: Introduce trace events for sleepable tree lock

There are two tree lock events which can sleep:
- btrfs_tree_read_lock()
- btrfs_tree_lock()

Sometimes we may need to look into the concurrency picture of the fs. For that case, we need the execution time of above two functions and the owner of @eb.

Here we introduce a trace events for user space tools like bcc, to get the execution time of above two functions, and get detailed owner info where eBPF code can't.

All the overhead is hidden behind the trace events, so if events are not enabled, there is no overhead.

These trace events also output bytenr and generation, allow them to be pared with unlock events to pin down deadlock.

---
 fs/btrfs/locking.c           | 12 ++++++++++
 include/trace/events/btrfs.h | 44 ++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
fs/btrfs/locking.c +12 -0
index 82b84e4daad1..b414d64172b2 100644
@@ -90,6 +90,10 @@ void btrfs_clear_lock_blocking_write(struct extent_buffer *eb)
  */
 void btrfs_tree_read_lock(struct extent_buffer *eb)
 {
+	u64 start_ns = 0;
+
+	if (trace_btrfs_tree_read_lock_enabled())
+		start_ns = ktime_get_ns();
 again:
 	BUG_ON(!atomic_read(&eb->blocking_writers) &&
 	       current->pid == eb->lock_owner);
@@ -106,6 +110,7 @@ void btrfs_tree_read_lock(struct extent_buffer *eb)
 		BUG_ON(eb->lock_nested);
 		eb->lock_nested = 1;
 		read_unlock(&eb->lock);
+		trace_btrfs_tree_read_lock(eb, start_ns);
 		return;
 	}
 	if (atomic_read(&eb->blocking_writers)) {
@@ -116,6 +121,7 @@ void btrfs_tree_read_lock(struct extent_buffer *eb)
 	}
 	atomic_inc(&eb->read_locks);
 	atomic_inc(&eb->spinning_readers);
+	trace_btrfs_tree_read_lock(eb, start_ns);
 }
 
 /*
@@ -232,6 +238,11 @@ void btrfs_tree_read_unlock_blocking(struct extent_buffer *eb)
  */
 void btrfs_tree_lock(struct extent_buffer *eb)
 {
+	u64 start_ns = 0;
+
+	if (trace_btrfs_tree_lock_enabled())
+		start_ns = ktime_get_ns();
+
 	WARN_ON(eb->lock_owner == current->pid);
 again:
 	wait_event(eb->read_lock_wq, atomic_read(&eb->blocking_readers) == 0);
@@ -246,6 +257,7 @@ void btrfs_tree_lock(struct extent_buffer *eb)
 	atomic_inc(&eb->spinning_writers);
 	atomic_inc(&eb->write_locks);
 	eb->lock_owner = current->pid;
+	trace_btrfs_tree_lock(eb, start_ns);
 }
 
 /*
include/trace/events/btrfs.h +44 -0

50 lines not on this page · raw message

Signature · 1 line
2.21.0
author
#2 lore raw

PATCH v3 2/2 btrfs: trace: Introduce trace events for all btrfs tree locking events

In reply to Qu Wenruo

Unlike btrfs_tree_lock() and btrfs_tree_read_lock(), the remaining functions in locking.c will not sleep, thus doesn't make much sense to record their execution time.

Those events are introduced mainly for user space tool to audit and detect lock leakage or dead lock.

---
 fs/btrfs/locking.c           | 10 +++++++++
 include/trace/events/btrfs.h | 42 ++++++++++++++++++++++++++++++++++++
 2 files changed, 52 insertions(+)
fs/btrfs/locking.c +10 -0
index b414d64172b2..a09823474d9c 100644
@@ -16,6 +16,7 @@ static void btrfs_assert_tree_read_locked(struct extent_buffer *eb);
 
 void btrfs_set_lock_blocking_read(struct extent_buffer *eb)
 {
+	trace_btrfs_set_lock_blocking_read(eb);
 	/*
 	 * No lock is required.  The lock owner may change if we have a read
 	 * lock, but it won't change to or away from us.  If we have the write
@@ -32,6 +33,7 @@ void btrfs_set_lock_blocking_read(struct extent_buffer *eb)
 
 void btrfs_set_lock_blocking_write(struct extent_buffer *eb)
 {
+	trace_btrfs_set_lock_blocking_write(eb);
 	/*
 	 * No lock is required.  The lock owner may change if we have a read
 	 * lock, but it won't change to or away from us.  If we have the write
@@ -50,6 +52,7 @@ void btrfs_set_lock_blocking_write(struct extent_buffer *eb)
 
 void btrfs_clear_lock_blocking_read(struct extent_buffer *eb)
 {
+	trace_btrfs_clear_lock_blocking_read(eb);
 	/*
 	 * No lock is required.  The lock owner may change if we have a read
 	 * lock, but it won't change to or away from us.  If we have the write
@@ -67,6 +70,7 @@ void btrfs_clear_lock_blocking_read(struct extent_buffer *eb)
 
 void btrfs_clear_lock_blocking_write(struct extent_buffer *eb)
 {
+	trace_btrfs_clear_lock_blocking_write(eb);
 	/*
 	 * no lock is required.  The lock owner may change if
 	 * we have a read lock, but it won't change to or away
@@ -141,6 +145,7 @@ int btrfs_tree_read_lock_atomic(struct extent_buffer *eb)
 	}
 	atomic_inc(&eb->read_locks);
 	atomic_inc(&eb->spinning_readers);
+	trace_btrfs_tree_read_lock_atomic(eb);
 	return 1;
 }
 
@@ -162,6 +167,7 @@ int btrfs_try_tree_read_lock(struct extent_buffer *eb)
 	}
 	atomic_inc(&eb->read_locks);
 	atomic_inc(&eb->spinning_readers);
+	trace_btrfs_try_tree_read_lock(eb);
 	return 1;
 }
 
@@ -184,6 +190,7 @@ int btrfs_try_tree_write_lock(struct extent_buffer *eb)
 	atomic_inc(&eb->write_locks);
 	atomic_inc(&eb->spinning_writers);
 	eb->lock_owner = current->pid;
+	trace_btrfs_try_tree_write_lock(eb);
 	return 1;
 }
 
@@ -192,6 +199,7 @@ int btrfs_try_tree_write_lock(struct extent_buffer *eb)
  */
 void btrfs_tree_read_unlock(struct extent_buffer *eb)
 {
+	trace_btrfs_tree_read_unlock(eb);
 	/*
 	 * if we're nested, we have the write lock.  No new locking
 	 * is needed as long as we are the lock owner.
@@ -214,6 +222,7 @@ void btrfs_tree_read_unlock(struct extent_buffer *eb)
  */
 void btrfs_tree_read_unlock_blocking(struct extent_buffer *eb)
 {
+	trace_btrfs_tree_read_unlock_blocking(eb);
 	/*
 	 * if we're nested, we have the write lock.  No new locking
 	 * is needed as long as we are the lock owner.
@@ -270,6 +279,7 @@ void btrfs_tree_unlock(struct extent_buffer *eb)
 	BUG_ON(blockers > 1);
 
 	btrfs_assert_tree_locked(eb);
+	trace_btrfs_tree_unlock(eb);
 	eb->lock_owner = 0;
 	atomic_dec(&eb->write_locks);
 
include/trace/events/btrfs.h +42 -0

48 lines not on this page · raw message

Signature · 1 line
2.21.0
#3 lore raw
In reply to Qu Wenruo
Quoted 33 lines · Qu Wenruo

This branch can be fetched from github:
https://github.com/adam900710/linux/tree/trace_locking
Which is based on v5.1-rc4 tag.

This patchset will introduce the following trace events for corresponding functions:
btrfs_tree_lock (sleepable)
btrfs_tree_unlock
btrfs_tree_read_lock (sleepable)
btrfs_tree_read_unlock
btrfs_tree_read_unlock_blocking btrfs_set_lock_blocking_read
btrfs_set_lock_blocking_write btrfs_clear_lock_blocking_read btrfs_clear_lock_blocking_write
btrfs_try_tree_read_lock
btrfs_try_tree_write_lock
btrfs_tree_read_lock_atomic

The two sleepable ones will have extra info like start_ns, end_ns, diff_ns.
Those two can be specially useful for tree lock wait time. There is tool to utilize those two events already:
https://github.com/adam900710/btrfs-profiler/blob/master/tree_lock_wait.py

The non-sleepable can be useful for possible user space based lock leakage/dead lock detector.

Changelog:
v2:
- Hide all ktime_get_ns() call out of trace events
So there will be no overhead if trace events are not enabled.

Quoted 4 lines

v3:
- More output for human to take a glance
- More trace events for later bcc usage
Like lock leakage and deadlock detection.

V3 looks good to me, thanks. The enhanced output of the sleepable lock is still sane.