Skip to contentSkip to conversation

btrfs: ref_tracker for delayed_nodes

PATCH Superseded Leo Martins 2 participants 6 messages Aug 7, 2025 – Aug 12, 2025 Btrfs file system

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover btrfs: ref_tracker for delayed_nodes

The leading btrfs related crash in our fleet is a soft lockup in btrfs_kill_all_delayed_nodes caused by a btrfs_delayed_node leak. This patchset introduces ref_tracker infrastructure to detect this leak. I'm mirroring the way REF_VERIFY is setup with a Kconfig and a mount option. I've run a full fstests suite with ref_tracker enabled and experienced roughly a 7% slowdown in runtime.

Changelog:
v1: https://lore.kernel.org/linux-btrfs/fa19acf9dc38e93546183fc083c365cdb237e89b.1752098515.git.loemra.dev@gmail.com/
v2: https://lore.kernel.org/linux-btrfs/[email protected]/T/#mba44556cfc1ae54c84255667a52a65f9520582b7

v2->v3:
- wrap ref_tracker and ref_tracker_dir in btrfs helper structs
- fix long function formatting
- new Kconfig CONFIG_BTRFS_FS_REF_TRACKER + ref_tracker mount option
- add a print to expose potential leaks
- move debug fields to the end of the struct

v1->v2:
- remove typedefs, now functions always take struct ref_tracker **
- put delayed_node::count back to original position to not change delayed_node struct size
- cleanup ref_tracker_dir if btrfs_get_or_create_delayed_node fails to create a delayed_node
- remove CONFIG_BTRFS_DELAYED_NODE_REF_TRACKER and use CONFIG_BTRFS_DEBUG

Leo Martins (3):
btrfs: implement ref_tracker for delayed_nodes
btrfs: print leaked references in kill_all_delayed_nodes
btrfs: add mount option for ref_tracker

 fs/btrfs/Kconfig         |  12 +++
 fs/btrfs/delayed-inode.c | 193 ++++++++++++++++++++++++++++-----------
 fs/btrfs/delayed-inode.h |  93 +++++++++++++++++++
 fs/btrfs/fs.h            |   1 +
 fs/btrfs/super.c         |  13 +++
 5 files changed, 257 insertions(+), 55 deletions(-)
Signature · 1 line
2.47.3
#1 lore raw

PATCH v3 1/3 btrfs: implement ref_tracker for delayed_nodes

This patch adds ref_tracker infrastructure for btrfs_delayed_node.

It is a response to the largest btrfs related crash in our fleet. We're seeing softlockups in btrfs_kill_all_delayed_nodes that seem to be a result of delayed_nodes not being released properly.

A ref_tracker object is allocated on reference count increases and freed on reference count decreases. The ref_tracker object stores a stack trace of where it is allocated. The ref_tracker_dir object is embedded in btrfs_delayed_node and keeps track of all current and some old/freed ref_tracker objects. When a leak is detected we can print the stack traces for all ref_trackers that have not yet been freed.

Here is a common example of taking a reference to a delayed_node and freeing it with ref_tracker.

```C
struct btrfs_ref_tracker tracker;
struct btrfs_delayed_node *node;

node = btrfs_get_delayed_node(inode, &tracker);
// use delayed_node...
btrfs_release_delayed_node(node, &tracker);
```

There are two special cases where the delayed_node reference is "long lived", meaning that the thread that takes the reference and the thread that releases the reference are different. The inode_cache_tracker tracks the delayed_node stored in btrfs_inode. The node_list_tracker tracks the delayed_node stored in the btrfs_delayed_root node_list/prepare_list. These trackers are embedded in the btrfs_delayed_node.

btrfs_ref_tracker and btrfs_ref_tracker_dir are wrappers that either compile to the corresponding ref_tracker structs or empty structs depending on the new CONFIG_BTRFS_FS_REF_TRACKER Kconfig. There are also btrfs wrappers for the ref_tracker API.

---
 fs/btrfs/Kconfig         |  12 +++
 fs/btrfs/delayed-inode.c | 192 ++++++++++++++++++++++++++++-----------
 fs/btrfs/delayed-inode.h |  70 ++++++++++++++
 3 files changed, 219 insertions(+), 55 deletions(-)
fs/btrfs/Kconfig +12 -0
index c352f3ae0385..34a512dfc3ce 100644
@@ -126,3 +126,15 @@ config BTRFS_FS_REF_VERIFY
 	  reference problems or verifying they didn't break something.
 
 	  If unsure, say N.
+
+config BTRFS_FS_REF_TRACKER
+	bool "Btrfs with the reference tracking compiled in"
+	depends on BTRFS_FS && STACKTRACE_SUPPORT
+	select REF_TRACKER
+	default n
+	help
+	  Enable run-time reference tracking. This is meant to be used by
+	  btrfs developers for tracking down reference leaks or verifying
+	  they didn't break something.
+
+	  If unsure, say N.
fs/btrfs/delayed-inode.c +137 -55

620 lines not on this page · raw message

fs/btrfs/delayed-inode.h +70 -0

92 lines not on this page · raw message

Signature · 1 line
2.47.3
#2 lore raw

PATCH v3 2/3 btrfs: print leaked references in kill_all_delayed_nodes

In reply to Leo Martins

We are seeing softlockups in kill_all_delayed_nodes due to a delayed_node with a lingering reference count of 1. Printing at this point will reveal the guilty stack trace. If the delayed_node has no references there should be no output.

---
 fs/btrfs/delayed-inode.c | 1 +
 fs/btrfs/delayed-inode.h | 8 ++++++++
 2 files changed, 9 insertions(+)
fs/btrfs/delayed-inode.c +1 -0
index c4696c2d5b34..f665434e1963 100644
@@ -2129,6 +2129,7 @@ void btrfs_kill_all_delayed_nodes(struct btrfs_root *root)
 			__btrfs_kill_delayed_node(delayed_nodes[i]);
 			btrfs_release_delayed_node(delayed_nodes[i],
 						   &delayed_node_trackers[i]);
+			btrfs_delayed_node_ref_tracker_dir_print(delayed_nodes[i]);
 		}
 	}
 }
fs/btrfs/delayed-inode.h +8 -0
index 171b75eb1850..d62811db2a38 100644
@@ -208,6 +208,12 @@ static inline void btrfs_delayed_node_ref_tracker_dir_exit(struct btrfs_delayed_
 	ref_tracker_dir_exit(&node->ref_dir.dir);
 }
 
+static inline void btrfs_delayed_node_ref_tracker_dir_print(struct btrfs_delayed_node *node)
+{
+	ref_tracker_dir_print(&node->ref_dir.dir,
+			      BTRFS_DELAYED_NODE_REF_TRACKER_DISPLAY_LIMIT);
+}
+
 static inline int btrfs_delayed_node_ref_tracker_alloc(struct btrfs_delayed_node *node,
 						       struct btrfs_ref_tracker *tracker,
 						       gfp_t gfp)
@@ -225,6 +231,8 @@ static inline void btrfs_delayed_node_ref_tracker_dir_init(struct btrfs_delayed_
 
 static inline void btrfs_delayed_node_ref_tracker_dir_exit(struct btrfs_delayed_node *node) { }
 
+static inline void btrfs_delayed_node_ref_tracker_dir_print(struct btrfs_delayed_node *node) { }
+
 static inline int btrfs_delayed_node_ref_tracker_alloc(struct btrfs_delayed_node *node,
 						       struct btrfs_ref_tracker *tracker,
 						       gfp_t gfp)
Signature · 1 line
2.47.3
#3 lore raw

PATCH v3 3/3 btrfs: add mount option for ref_tracker

In reply to Leo Martins

This patch adds a mount option for ref_tracker, mirroring how ref_verifier is setup.

---
 fs/btrfs/delayed-inode.h | 15 +++++++++++++++
 fs/btrfs/fs.h            |  1 +
 fs/btrfs/super.c         | 13 +++++++++++++
 3 files changed, 29 insertions(+)
fs/btrfs/delayed-inode.h +15 -0
index d62811db2a38..e1c44c4499b6 100644
@@ -198,6 +198,9 @@ void btrfs_assert_delayed_root_empty(struct btrfs_fs_info *fs_info);
 #ifdef CONFIG_BTRFS_FS_REF_TRACKER
 static inline void btrfs_delayed_node_ref_tracker_dir_init(struct btrfs_delayed_node *node)
 {
+	if (!btrfs_test_opt(node->root->fs_info, REF_TRACKER))
+		return;
+
 	ref_tracker_dir_init(&node->ref_dir.dir, 
 			     BTRFS_DELAYED_NODE_REF_TRACKER_QUARANTINE_COUNT,
 			     "delayed_node");
@@ -205,11 +208,17 @@ static inline void btrfs_delayed_node_ref_tracker_dir_init(struct btrfs_delayed_
 
 static inline void btrfs_delayed_node_ref_tracker_dir_exit(struct btrfs_delayed_node *node)
 {
+	if (!btrfs_test_opt(node->root->fs_info, REF_TRACKER))
+		return;
+
 	ref_tracker_dir_exit(&node->ref_dir.dir);
 }
 
 static inline void btrfs_delayed_node_ref_tracker_dir_print(struct btrfs_delayed_node *node)
 {
+	if (!btrfs_test_opt(node->root->fs_info, REF_TRACKER))
+		return;
+
 	ref_tracker_dir_print(&node->ref_dir.dir,
 			      BTRFS_DELAYED_NODE_REF_TRACKER_DISPLAY_LIMIT);
 }
@@ -218,12 +227,18 @@ static inline int btrfs_delayed_node_ref_tracker_alloc(struct btrfs_delayed_node
 						       struct btrfs_ref_tracker *tracker,
 						       gfp_t gfp)
 {
+	if (!btrfs_test_opt(node->root->fs_info, REF_TRACKER))
+		return 0;
+
 	return ref_tracker_alloc(&node->ref_dir.dir, &tracker->tracker, gfp);
 }
 
 static inline int btrfs_delayed_node_ref_tracker_free(struct btrfs_delayed_node *node,
 						      struct btrfs_ref_tracker *tracker)
 {
+	if (!btrfs_test_opt(node->root->fs_info, REF_TRACKER))
+		return 0;
+
 	return ref_tracker_free(&node->ref_dir.dir, &tracker->tracker);
 }
 #else
fs/btrfs/fs.h +1 -0
index 8cc07cc70b12..2052b7ca86b5 100644
@@ -243,6 +243,7 @@ enum {
 	BTRFS_MOUNT_NOSPACECACHE		= (1ULL << 30),
 	BTRFS_MOUNT_IGNOREMETACSUMS		= (1ULL << 31),
 	BTRFS_MOUNT_IGNORESUPERFLAGS		= (1ULL << 32),
+	BTRFS_MOUNT_REF_TRACKER			= (1ULL << 33),
 };
 
 /*
fs/btrfs/super.c +13 -0
index 466d0450269c..924a6e5e7348 100644
@@ -133,6 +133,9 @@ enum {
 #endif
 #ifdef CONFIG_BTRFS_FS_REF_VERIFY
 	Opt_ref_verify,
+#endif
+#ifdef CONFIG_BTRFS_FS_REF_TRACKER
+	Opt_ref_tracker,
 #endif
 	Opt_err,
 };
@@ -257,6 +260,9 @@ static const struct fs_parameter_spec btrfs_fs_parameters[] = {
 #endif
 #ifdef CONFIG_BTRFS_FS_REF_VERIFY
 	fsparam_flag("ref_verify", Opt_ref_verify),
+#endif
+#ifdef CONFIG_BTRFS_FS_REF_TRACKER
+	fsparam_flag("ref_tracker", Opt_ref_tracker),
 #endif
 	{}
 };
@@ -634,6 +640,11 @@ static int btrfs_parse_param(struct fs_context *fc, struct fs_parameter *param)
 	case Opt_ref_verify:
 		btrfs_set_opt(ctx->mount_opt, REF_VERIFY);
 		break;
+#endif
+#ifdef CONFIG_BTRFS_FS_REF_TRACKER
+	case Opt_ref_tracker:
+		btrfs_set_opt(ctx->mount_opt, REF_TRACKER);
+		break;
 #endif
 	default:
 		btrfs_err(NULL, "unrecognized mount option '%s'", param->key);
@@ -1140,6 +1151,8 @@ static int btrfs_show_options(struct seq_file *seq, struct dentry *dentry)
 #endif
 	if (btrfs_test_opt(info, REF_VERIFY))
 		seq_puts(seq, ",ref_verify");
+	if (btrfs_test_opt(info, REF_TRACKER))
+		seq_puts(seq, ",ref_tracker");
 	seq_printf(seq, ",subvolid=%llu", btrfs_root_id(BTRFS_I(d_inode(dentry))->root));
 	subvol_name = btrfs_get_subvol_name_from_objectid(info,
 			btrfs_root_id(BTRFS_I(d_inode(dentry))->root));
Signature · 1 line
2.47.3
#4 lore raw
In reply to Leo Martins
Quoted 7 lines · Leo Martins

The leading btrfs related crash in our fleet is a soft lockup in btrfs_kill_all_delayed_nodes caused by a btrfs_delayed_node leak. This patchset introduces ref_tracker infrastructure to detect this leak. I'm mirroring the way REF_VERIFY is setup with a Kconfig and a mount option. I've run a full fstests suite with ref_tracker enabled and experienced roughly a 7% slowdown in runtime.

Quoted 8 lines

Changelog:
v1: https://lore.kernel.org/linux-btrfs/fa19acf9dc38e93546183fc083c365cdb237e89b.1752098515.git.loemra.dev@gmail.com/
v2: https://lore.kernel.org/linux-btrfs/[email protected]/T/#mba44556cfc1ae54c84255667a52a65f9520582b7

v2->v3:
- wrap ref_tracker and ref_tracker_dir in btrfs helper structs
- fix long function formatting
- new Kconfig CONFIG_BTRFS_FS_REF_TRACKER + ref_tracker mount option

We don't want new config options, the REF_VERIFY is there historically but is not an example to follow. I think this was mentioned already in previou iterations.

Quoted 27 lines

- add a print to expose potential leaks
- move debug fields to the end of the struct

v1->v2:
- remove typedefs, now functions always take struct ref_tracker **
- put delayed_node::count back to original position to not change delayed_node struct size
- cleanup ref_tracker_dir if btrfs_get_or_create_delayed_node fails to create a delayed_node
- remove CONFIG_BTRFS_DELAYED_NODE_REF_TRACKER and use CONFIG_BTRFS_DEBUG

Leo Martins (3):
btrfs: implement ref_tracker for delayed_nodes
btrfs: print leaked references in kill_all_delayed_nodes
btrfs: add mount option for ref_tracker

 fs/btrfs/Kconfig         |  12 +++
 fs/btrfs/delayed-inode.c | 193 ++++++++++++++++++++++++++++-----------
 fs/btrfs/delayed-inode.h |  93 +++++++++++++++++++
 fs/btrfs/fs.h            |   1 +
 fs/btrfs/super.c         |  13 +++
 5 files changed, 257 insertions(+), 55 deletions(-)
Signature · 1 line
2.47.3
#5 lore raw
In reply to Leo Martins
Quoted 5 lines · Leo Martins

The leading btrfs related crash in our fleet is a soft lockup in btrfs_kill_all_delayed_nodes caused by a btrfs_delayed_node leak. This patchset introduces ref_tracker infrastructure to detect this leak. I'm mirroring the way REF_VERIFY is setup with a Kconfig and a mount option.

To avoid copying REF_VERIFY in the future, I'd like to see moved it under debug config as well. The runtime cost if disabled is negligible, there's one extra memeber in the btrfs_ref structure but that's also acceptable.