btrfs: ref_tracker for delayed_nodes
v3 cover btrfs: ref_tracker for delayed_nodes
The leading btrfs related crash in our fleet is a soft lockup in btrfs_kill_all_delayed_nodes caused by a btrfs_delayed_node leak. This patchset introduces ref_tracker infrastructure to detect this leak. I'm mirroring the way REF_VERIFY is setup with a Kconfig and a mount option. I've run a full fstests suite with ref_tracker enabled and experienced roughly a 7% slowdown in runtime.
Changelog:
v1: https://lore.kernel.org/linux-btrfs/fa19acf9dc38e93546183fc083c365cdb237e89b.1752098515.git.loemra.dev@gmail.com/
v2: https://lore.kernel.org/linux-btrfs/[email protected]/T/#mba44556cfc1ae54c84255667a52a65f9520582b7
v2->v3:
- wrap ref_tracker and ref_tracker_dir in btrfs helper structs
- fix long function formatting
- new Kconfig CONFIG_BTRFS_FS_REF_TRACKER + ref_tracker mount option
- add a print to expose potential leaks
- move debug fields to the end of the struct
v1->v2:
- remove typedefs, now functions always take struct ref_tracker **
- put delayed_node::count back to original position to not change delayed_node struct size
- cleanup ref_tracker_dir if btrfs_get_or_create_delayed_node fails to create a delayed_node
- remove CONFIG_BTRFS_DELAYED_NODE_REF_TRACKER and use CONFIG_BTRFS_DEBUG
Leo Martins (3):
btrfs: implement ref_tracker for delayed_nodes
btrfs: print leaked references in kill_all_delayed_nodes
btrfs: add mount option for ref_tracker
fs/btrfs/Kconfig | 12 +++ fs/btrfs/delayed-inode.c | 193 ++++++++++++++++++++++++++++----------- fs/btrfs/delayed-inode.h | 93 +++++++++++++++++++ fs/btrfs/fs.h | 1 + fs/btrfs/super.c | 13 +++ 5 files changed, 257 insertions(+), 55 deletions(-)
Signature · 1 line
2.47.3
PATCH v3 1/3 btrfs: implement ref_tracker for delayed_nodes
This patch adds ref_tracker infrastructure for btrfs_delayed_node.
It is a response to the largest btrfs related crash in our fleet. We're seeing softlockups in btrfs_kill_all_delayed_nodes that seem to be a result of delayed_nodes not being released properly.
A ref_tracker object is allocated on reference count increases and freed on reference count decreases. The ref_tracker object stores a stack trace of where it is allocated. The ref_tracker_dir object is embedded in btrfs_delayed_node and keeps track of all current and some old/freed ref_tracker objects. When a leak is detected we can print the stack traces for all ref_trackers that have not yet been freed.
Here is a common example of taking a reference to a delayed_node and freeing it with ref_tracker.
```C struct btrfs_ref_tracker tracker; struct btrfs_delayed_node *node; node = btrfs_get_delayed_node(inode, &tracker); // use delayed_node... btrfs_release_delayed_node(node, &tracker); ```
There are two special cases where the delayed_node reference is "long lived", meaning that the thread that takes the reference and the thread that releases the reference are different. The inode_cache_tracker tracks the delayed_node stored in btrfs_inode. The node_list_tracker tracks the delayed_node stored in the btrfs_delayed_root node_list/prepare_list. These trackers are embedded in the btrfs_delayed_node.
btrfs_ref_tracker and btrfs_ref_tracker_dir are wrappers that either compile to the corresponding ref_tracker structs or empty structs depending on the new CONFIG_BTRFS_FS_REF_TRACKER Kconfig. There are also btrfs wrappers for the ref_tracker API.
--- fs/btrfs/Kconfig | 12 +++ fs/btrfs/delayed-inode.c | 192 ++++++++++++++++++++++++++++----------- fs/btrfs/delayed-inode.h | 70 ++++++++++++++ 3 files changed, 219 insertions(+), 55 deletions(-)
fs/btrfs/Kconfig +12 -0
@@ -126,3 +126,15 @@ config BTRFS_FS_REF_VERIFY reference problems or verifying they didn't break something. If unsure, say N. + +config BTRFS_FS_REF_TRACKER + bool "Btrfs with the reference tracking compiled in" + depends on BTRFS_FS && STACKTRACE_SUPPORT + select REF_TRACKER + default n + help + Enable run-time reference tracking. This is meant to be used by + btrfs developers for tracking down reference leaks or verifying + they didn't break something. + + If unsure, say N.
fs/btrfs/delayed-inode.c +137 -55
620 lines not on this page · raw message
fs/btrfs/delayed-inode.h +70 -0
92 lines not on this page · raw message
Signature · 1 line
2.47.3
PATCH v3 2/3 btrfs: print leaked references in kill_all_delayed_nodes
In reply to Leo MartinsWe are seeing softlockups in kill_all_delayed_nodes due to a delayed_node with a lingering reference count of 1. Printing at this point will reveal the guilty stack trace. If the delayed_node has no references there should be no output.
--- fs/btrfs/delayed-inode.c | 1 + fs/btrfs/delayed-inode.h | 8 ++++++++ 2 files changed, 9 insertions(+)
fs/btrfs/delayed-inode.c +1 -0
@@ -2129,6 +2129,7 @@ void btrfs_kill_all_delayed_nodes(struct btrfs_root *root)__btrfs_kill_delayed_node (delayed_nodes[i]);btrfs_release_delayed_node (delayed_nodes[i], &delayed_node_trackers[i]); +btrfs_delayed_node_ref_tracker_dir_print (delayed_nodes[i]); } } }
fs/btrfs/delayed-inode.h +8 -0
@@ -208,6 +208,12 @@ static inline void btrfs_delayed_node_ref_tracker_dir_exit(struct btrfs_delayed_ref_tracker_dir_exit (&node->ref_dir.dir); } +static inline void btrfs_delayed_node_ref_tracker_dir_print (struct btrfs_delayed_node *node) +{ +ref_tracker_dir_print (&node->ref_dir.dir, +BTRFS_DELAYED_NODE_REF_TRACKER_DISPLAY_LIMIT ); +} +static inline int btrfs_delayed_node_ref_tracker_alloc (struct btrfs_delayed_node *node,struct btrfs_ref_tracker *tracker,gfp_t gfp)
@@ -225,6 +231,8 @@ static inline void btrfs_delayed_node_ref_tracker_dir_init(struct btrfs_delayed_static inline void btrfs_delayed_node_ref_tracker_dir_exit (struct btrfs_delayed_node *node) { } +static inline void btrfs_delayed_node_ref_tracker_dir_print (struct btrfs_delayed_node *node) { } +static inline int btrfs_delayed_node_ref_tracker_alloc (struct btrfs_delayed_node *node,struct btrfs_ref_tracker *tracker,gfp_t gfp)
Signature · 1 line
2.47.3
PATCH v3 3/3 btrfs: add mount option for ref_tracker
In reply to Leo MartinsThis patch adds a mount option for ref_tracker, mirroring how ref_verifier is setup.
--- fs/btrfs/delayed-inode.h | 15 +++++++++++++++ fs/btrfs/fs.h | 1 + fs/btrfs/super.c | 13 +++++++++++++ 3 files changed, 29 insertions(+)
fs/btrfs/delayed-inode.h +15 -0
@@ -198,6 +198,9 @@ void btrfs_assert_delayed_root_empty(struct btrfs_fs_info *fs_info);#ifdef CONFIG_BTRFS_FS_REF_TRACKER static inline void btrfs_delayed_node_ref_tracker_dir_init (struct btrfs_delayed_node *node) { +if (!btrfs_test_opt (node->root->fs_info,REF_TRACKER )) +return ; +ref_tracker_dir_init (&node->ref_dir.dir,BTRFS_DELAYED_NODE_REF_TRACKER_QUARANTINE_COUNT ,"delayed_node" );
@@ -205,11 +208,17 @@ static inline void btrfs_delayed_node_ref_tracker_dir_init(struct btrfs_delayed_static inline void btrfs_delayed_node_ref_tracker_dir_exit (struct btrfs_delayed_node *node) { +if (!btrfs_test_opt (node->root->fs_info,REF_TRACKER )) +return ; +ref_tracker_dir_exit (&node->ref_dir.dir); }static inline void btrfs_delayed_node_ref_tracker_dir_print (struct btrfs_delayed_node *node) { +if (!btrfs_test_opt (node->root->fs_info,REF_TRACKER )) +return ; +ref_tracker_dir_print (&node->ref_dir.dir,BTRFS_DELAYED_NODE_REF_TRACKER_DISPLAY_LIMIT ); }
@@ -218,12 +227,18 @@ static inline int btrfs_delayed_node_ref_tracker_alloc(struct btrfs_delayed_node struct btrfs_ref_tracker *tracker, gfp_t gfp) { +if (!btrfs_test_opt (node->root->fs_info,REF_TRACKER )) +return 0 ; +return ref_tracker_alloc (&node->ref_dir.dir, &tracker->tracker, gfp); }static inline int btrfs_delayed_node_ref_tracker_free (struct btrfs_delayed_node *node,struct btrfs_ref_tracker *tracker) { +if (!btrfs_test_opt (node->root->fs_info,REF_TRACKER )) +return 0 ; +return ref_tracker_free (&node->ref_dir.dir, &tracker->tracker); }#else
fs/btrfs/fs.h +1 -0
@@ -243,6 +243,7 @@ enum {BTRFS_MOUNT_NOSPACECACHE = (1ULL <<30 ),BTRFS_MOUNT_IGNOREMETACSUMS = (1ULL <<31 ),BTRFS_MOUNT_IGNORESUPERFLAGS = (1ULL <<32 ), + BTRFS_MOUNT_REF_TRACKER = (1ULL << 33), };/*
fs/btrfs/super.c +13 -0
@@ -133,6 +133,9 @@ enum {#endif #ifdef CONFIG_BTRFS_FS_REF_VERIFY Opt_ref_verify, +#endif +#ifdef CONFIG_BTRFS_FS_REF_TRACKER + Opt_ref_tracker, #endif Opt_err, };
@@ -257,6 +260,9 @@ static const struct fs_parameter_spec btrfs_fs_parameters[] = {#endif #ifdef CONFIG_BTRFS_FS_REF_VERIFY fsparam_flag("ref_verify", Opt_ref_verify), +#endif +#ifdef CONFIG_BTRFS_FS_REF_TRACKER + fsparam_flag("ref_tracker", Opt_ref_tracker),#endif {} };
@@ -634,6 +640,11 @@ static int btrfs_parse_param(struct fs_context *fc, struct fs_parameter *param)case Opt_ref_verify:btrfs_set_opt (ctx->mount_opt,REF_VERIFY );break ; +#endif +#ifdef CONFIG_BTRFS_FS_REF_TRACKER +case Opt_ref_tracker: +btrfs_set_opt (ctx->mount_opt,REF_TRACKER ); +break ;#endif default :btrfs_err (NULL ,"unrecognized mount option '%s'" , param->key);
@@ -1140,6 +1151,8 @@ static int btrfs_show_options(struct seq_file *seq, struct dentry *dentry)#endif if (btrfs_test_opt (info,REF_VERIFY ))seq_puts (seq,",ref_verify" ); +if (btrfs_test_opt (info,REF_TRACKER )) +seq_puts (seq,",ref_tracker" );seq_printf (seq,",subvolid=%llu" ,btrfs_root_id (BTRFS_I (d_inode (dentry))->root)); subvol_name =btrfs_get_subvol_name_from_objectid (info,btrfs_root_id (BTRFS_I (d_inode (dentry))->root));
Signature · 1 line
2.47.3
Quoted 7 lines · Leo Martins
The leading btrfs related crash in our fleet is a soft lockup in
btrfs_kill_all_delayed_nodescaused by abtrfs_delayed_nodeleak. This patchset introducesref_trackerinfrastructure to detect this leak. I'm mirroring the wayREF_VERIFYis setup with aKconfigand a mount option. I've run a full fstests suite withref_trackerenabled and experienced roughly a 7% slowdown in runtime.
Quoted 8 lines
Changelog:
v1: https://lore.kernel.org/linux-btrfs/fa19acf9dc38e93546183fc083c365cdb237e89b.1752098515.git.loemra.dev@gmail.com/
v2: https://lore.kernel.org/linux-btrfs/[email protected]/T/#mba44556cfc1ae54c84255667a52a65f9520582b7
v2->v3:
- wrapref_trackerandref_tracker_dirin btrfs helper structs
- fix long function formatting
- newKconfigCONFIG_BTRFS_FS_REF_TRACKER+ref_trackermount option
We don't want new config options, the REF_VERIFY is there historically but is not an example to follow. I think this was mentioned already in previou iterations.
Quoted 27 lines
- add a print to expose potential leaks
- move debug fields to the end of the struct
v1->v2:
- remove typedefs, now functions always take structref_tracker**
- putdelayed_node::countback to original position to not changedelayed_nodestruct size
- cleanupref_tracker_dirifbtrfs_get_or_create_delayed_nodefails to create adelayed_node
- removeCONFIG_BTRFS_DELAYED_NODE_REF_TRACKERand useCONFIG_BTRFS_DEBUGLeo Martins (3):
btrfs: implementref_trackerfordelayed_nodes
btrfs: print leaked references inkill_all_delayed_nodes
btrfs: add mount option forref_trackerfs/btrfs/Kconfig | 12 +++ fs/btrfs/delayed-inode.c | 193 ++++++++++++++++++++++++++++----------- fs/btrfs/delayed-inode.h | 93 +++++++++++++++++++ fs/btrfs/fs.h | 1 + fs/btrfs/super.c | 13 +++ 5 files changed, 257 insertions(+), 55 deletions(-)Signature · 1 line
2.47.3
Quoted 5 lines · Leo Martins
The leading btrfs related crash in our fleet is a soft lockup in
btrfs_kill_all_delayed_nodescaused by abtrfs_delayed_nodeleak. This patchset introducesref_trackerinfrastructure to detect this leak. I'm mirroring the wayREF_VERIFYis setup with aKconfigand a mount option.
To avoid copying REF_VERIFY in the future, I'd like to see moved it under debug config as well. The runtime cost if disabled is negligible, there's one extra memeber in the btrfs_ref structure but that's also acceptable.