btrfs: workaround exhausted anonymous block device pool
cover btrfs: workaround exhausted anonymous block device pool
There is a user report about transaction abort with a rare error number, -EMFILE:
------------[ cut here ]------------ BTRFS: Transaction aborted (error -24) WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs] RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs] Call Trace: create_pending_snapshots+0x82/0xa0 [btrfs] btrfs_commit_transaction+0x275/0x8c0 [btrfs] btrfs_mksubvol+0x4b9/0x500 [btrfs] btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs] btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs] btrfs_ioctl+0x11a4/0x2da0 [btrfs] do_vfs_ioctl+0xa9/0x640 ksys_ioctl+0x67/0x90 __x64_sys_ioctl+0x1a/0x20 do_syscall_64+0x5a/0x110 entry_SYSCALL_64_after_hwframe+0x44/0xa9 ---[ end trace 33f2f83f3d5250e9 ]--- BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown BTRFS info (device sda1): forced readonly BTRFS warning (device sda1): Skipping commit of aborted transaction. BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown
It turns out that, it's caused by exhausted anonymous block device pool. For that case, we don't have any good way to enlarge that pool to match btrfs subvolume limits.
But still we can improve the situation not to abort transaction, but report error gracefully to user.
This patchset will:
- Reduce the user of anonymous block device pool
Although it can only reduce the user by data reloc tree and orphan roots, it should still save save several ids.
- Catch the unintialized btrfs_root::anon_dev for user visible roots Just an extra safe net for previous patch
- Preallocate anon_dev for snapshot/subvolume creation
So user will get a graceful error other than transaction abort This patch should be the core of the patchset, as the remaining part won't have much user visible effect.
- Free anon_dev earlier to reduce the lifespan of anon_dev
Qu Wenruo (4):
btrfs: disk-io: don't allocate anonymous block device for user
invisible roots
btrfs: detect uninitialized btrfs_root::anon_dev for user visible
subvolumes
btrfs: preallocate anon_dev for subvolume and snapshot creation
btrfs: free anon_dev earlier to prevent exhausting anonymous block
device pool
fs/btrfs/disk-io.c | 59 +++++++++++++++++++++++++++++++++++++----- fs/btrfs/disk-io.h | 2 ++ fs/btrfs/inode.c | 11 +++++++- fs/btrfs/ioctl.c | 21 ++++++++++++++- fs/btrfs/transaction.c | 3 ++- fs/btrfs/transaction.h | 2 ++ 6 files changed, 88 insertions(+), 10 deletions(-)
Signature · 1 line
2.27.0
PATCH 1/4 btrfs: disk-io: don't allocate anonymous block device for user invisible roots
[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:
------------[ cut here ]------------ BTRFS: Transaction aborted (error -24) WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs] RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs] Call Trace: create_pending_snapshots+0x82/0xa0 [btrfs] btrfs_commit_transaction+0x275/0x8c0 [btrfs] btrfs_mksubvol+0x4b9/0x500 [btrfs] btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs] btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs] btrfs_ioctl+0x11a4/0x2da0 [btrfs] do_vfs_ioctl+0xa9/0x640 ksys_ioctl+0x67/0x90 __x64_sys_ioctl+0x1a/0x20 do_syscall_64+0x5a/0x110 entry_SYSCALL_64_after_hwframe+0x44/0xa9 ---[ end trace 33f2f83f3d5250e9 ]--- BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown BTRFS info (device sda1): forced readonly BTRFS warning (device sda1): Skipping commit of aborted transaction. BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown
[CAUSE]
The root cause is we don't have unlimited resource for anonymous block device number.
The anonymous block device pool only contains 1<<20 devices, and is shared across a several fses, like ceph and overlayfs.
While btrfs has support for 1<<48 subvolumes, so it's just a problem of time to hit such limit.
[WORKAROUND]
Since it's not possible to completely solve the problem, we can only workaround it.
Firstly, we can reduce the user of anon_dev. Data reloc tree is not visible to users, thus it doesn't need anon_dev at all.
This patch will do extra check on root objectid, to rule out roots who don't need anon_dev.
Although currently it's only data reloc tree and orphan roots.
--- fs/btrfs/disk-io.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
fs/btrfs/disk-io.c +11 -3
@@ -1428,9 +1428,17 @@ static int btrfs_init_fs_root(struct btrfs_root *root)spin_lock_init (&root->ino_cache_lock);init_waitqueue_head (&root->ino_cache_wait); - ret =get_anon_bdev (&root->anon_dev); -if (ret) - goto fail; +/* +* Anonymous block device pool has limited size (1M), which is way +* smaller than btrfs subvolumes limits (1<<48). +* We shouldn't allocate any if it's not a user visible subvolume. +*/ +if (is_fstree (root->root_key.objectid) && +btrfs_root_refs (&root->root_item)) { + ret =get_anon_bdev (&root->anon_dev); +if (ret) + goto fail; + }mutex_lock (&root->objectid_mutex); ret = btrfs_find_highest_objectid(root,
Signature · 1 line
2.27.0
Re: [PATCH 1/4] btrfs: disk-io: don't allocate anonymous block device for user invisible roots
Quoted 42 lines · Qu Wenruo
[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:------------[ cut here ]------------ BTRFS: Transaction aborted (error -24) WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs] RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs] Call Trace: create_pending_snapshots+0x82/0xa0 [btrfs] btrfs_commit_transaction+0x275/0x8c0 [btrfs] btrfs_mksubvol+0x4b9/0x500 [btrfs] btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs] btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs] btrfs_ioctl+0x11a4/0x2da0 [btrfs] do_vfs_ioctl+0xa9/0x640 ksys_ioctl+0x67/0x90 __x64_sys_ioctl+0x1a/0x20 do_syscall_64+0x5a/0x110 entry_SYSCALL_64_after_hwframe+0x44/0xa9 ---[ end trace 33f2f83f3d5250e9 ]--- BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown BTRFS info (device sda1): forced readonly BTRFS warning (device sda1): Skipping commit of aborted transaction. BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown[CAUSE]
The root cause is we don't have unlimited resource for anonymous block device number.
The anonymous block device pool only contains 1<<20 devices, and is shared across a several fses, like ceph and overlayfs.While btrfs has support for 1<<48 subvolumes, so it's just a problem of time to hit such limit.
[WORKAROUND]
Since it's not possible to completely solve the problem, we can only workaround it.Firstly, we can reduce the user of
anon_dev. Data reloc tree is not visible to users, thus it doesn't needanon_devat all.
Quoted 7 lines
This patch will do extra check on root objectid, to rule out roots who don't need
anon_dev.
Although currently it's only data reloc tree and orphan roots.
Thanks,
Josef
PATCH 2/4 btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes
In reply to Qu Wenruobtrfs_root::anon_dev is an indicator for different subvolume namespaces. Thus it should always be initialized to an anonymous block device.
Add a safe net to catch such uninitialized values.
--- fs/btrfs/inode.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
fs/btrfs/inode.c +8 -1
@@ -8697,6 +8697,7 @@ static int btrfs_getattr(const struct path *path, struct kstat *stat, {u64 delalloc_bytes;struct inode *inode =d_inode (path->dentry); +struct btrfs_root *root =BTRFS_I (inode)->root;u32 blocksize = inode->i_sb->s_blocksize;u32 bi_flags =BTRFS_I (inode)->flags;
@@ -8718,7 +8719,13 @@ static int btrfs_getattr(const struct path *path, struct kstat *stat, STATX_ATTR_NODUMP);generic_fillattr (inode, stat); - stat->dev =BTRFS_I (inode)->root->anon_dev; +if (unlikely (!root->anon_dev)) { +WARN_ON (IS_ENABLED (CONFIG_BTRFS_DEBUG )); +btrfs_warn (root->fs_info, +"uninitialized btrfs_root::anon_dev detected, root=%llu" , + root->root_key.objectid); + } + stat->dev = root->anon_dev;spin_lock (&BTRFS_I (inode)->lock); delalloc_bytes =BTRFS_I (inode)->new_delalloc_bytes;
Signature · 1 line
2.27.0
Re: [PATCH 2/4] btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes
Quoted 6 lines · Qu Wenruo
btrfs_root::anon_devis an indicator for different subvolume namespaces. Thus it should always be initialized to an anonymous block device.Add a safe net to catch such uninitialized values.
Signed-off-by Qu Wenruo <[email protected]>
Can we handle stat->dev not having a device set? Or will this blow up in other ways? Thanks,
Josef
Re: [PATCH 2/4] btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes
Quoted 10 lines · Josef Bacik
Quoted 6 lines · Qu Wenruo
btrfs_root::anon_devis an indicator for different subvolume namespaces. Thus it should always be initialized to an anonymous block device.Add a safe net to catch such uninitialized values.
Signed-off-by Qu Wenruo <[email protected]>Can we handle
stat->devnot having a device set? Or will this blow up in other ways? Thanks,
We can handle it without any problem, just users get confused.
As a common practice, we use different bdev as a namespace for different subvolumes.
Without a valid bdev, some user space tools may not be able to distinguish inodes in different subvolumes.
Thanks,
Qu
Quoted 2 lines
Josef
Re: [PATCH 2/4] btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes
Quoted 14 lines · Qu Wenruo
Quoted 10 lines · Josef Bacik
Quoted 6 lines · Qu Wenruo
btrfs_root::anon_devis an indicator for different subvolume namespaces. Thus it should always be initialized to an anonymous block device.Add a safe net to catch such uninitialized values.
Signed-off-by Qu Wenruo <[email protected]>Can we handle
stat->devnot having a device set? Or will this blow up in other ways? Thanks,
Quoted 7 lines
We can handle it without any problem, just users get confused.
As a common practice, we use different bdev as a namespace for different subvolumes.
Without a valid bdev, some user space tools may not be able to distinguish inodes in different subvolumes.
Alright that's fine then. But I feel like stat is one of those things that'll flood the console, can we put this somewhere else that's going to be hit less? Thanks,
Josef
Re: [PATCH 2/4] btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes
Quoted 24 lines · Josef Bacik
Quoted 15 lines · Qu Wenruo
Quoted 11 lines · Josef Bacik
Quoted 7 lines · Qu Wenruo
btrfs_root::anon_devis an indicator for different subvolume namespaces.
Thus it should always be initialized to an anonymous block device.Add a safe net to catch such uninitialized values.
Signed-off-by Qu Wenruo <[email protected]>Can we handle
stat->devnot having a device set? Or will this blow up in other ways? Thanks,Quoted 7 lines
We can handle it without any problem, just users get confused.
As a common practice, we use different bdev as a namespace for different subvolumes.
Without a valid bdev, some user space tools may not be able to distinguish inodes in different subvolumes.
Quoted 3 lines
Alright that's fine then. But I feel like stat is one of those things that'll flood the console, can we put this somewhere else that's going to be hit less? Thanks,
Unfortunately, stat() is the only user of btrfs_root::anon_dev.
While fortunately, the logical is pretty simple, even without the safe net we can understand the lifespan pretty well.
I'm fine to drop this patch if you're concerned about the possible warning flood, as the benefit is really not that much.
Thanks,
Qu
Quoted 2 lines
Josef
Re: [PATCH 2/4] btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes
Quoted 15 lines · Qu Wenruo
Quoted 11 lines
Quoted 10 lines
Quoted 2 lines
Can we handle
stat->devnot having a device set? Or will this blow up in other ways? Thanks,We can handle it without any problem, just users get confused.
As a common practice, we use different bdev as a namespace for different subvolumes.
Without a valid bdev, some user space tools may not be able to distinguish inodes in different subvolumes.Quoted 3 lines
Alright that's fine then. But I feel like stat is one of those things that'll flood the console, can we put this somewhere else that's going to be hit less? Thanks,
Quoted 7 lines
Unfortunately,
stat()is the only user ofbtrfs_root::anon_dev.While fortunately, the logical is pretty simple, even without the safe net we can understand the lifespan pretty well.
I'm fine to drop this patch if you're concerned about the possible warning flood, as the benefit is really not that much.
It could be a developer-only warning but if there's a root with a bad anon_dev, a simple 'ls -l' would flood the log for sure.
Re: [PATCH 2/4] btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes
Quoted 18 lines · David Sterba
Quoted 17 lines · Qu Wenruo
Quoted 11 lines
Quoted 10 lines
Quoted 2 lines
Can we handle
stat->devnot having a device set? Or will this blow up in other ways? Thanks,We can handle it without any problem, just users get confused.
As a common practice, we use different bdev as a namespace for different subvolumes.
Without a valid bdev, some user space tools may not be able to distinguish inodes in different subvolumes.Quoted 3 lines
Alright that's fine then. But I feel like stat is one of those things that'll flood the console, can we put this somewhere else that's going to be hit less? Thanks,
Unfortunately,
stat()is the only user ofbtrfs_root::anon_dev.
Quoted 9 lines
Quoted 5 lines
While fortunately, the logical is pretty simple, even without the safe net we can understand the lifespan pretty well.
I'm fine to drop this patch if you're concerned about the possible warning flood, as the benefit is really not that much.
It could be a developer-only warning but if there's a root with a bad
anon_dev, a simple 'ls -l' would flood the log for sure.
We'll know in btrfs_init_fs_root() when get_anon_bdev() fails right? Can't we just complain then? That seems less spammy. Thanks,
Josef
Re: [PATCH 2/4] btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes
Quoted 27 lines · Josef Bacik
Quoted 26 lines · David Sterba
Quoted 24 lines · Qu Wenruo
Quoted 13 lines
Quoted 4 lines
Quoted 3 lines
Can we handle
stat->devnot having a device set? Or will this blow up
in other ways? Thanks,Quoted 8 lines
We can handle it without any problem, just users get confused.
As a common practice, we use different bdev as a namespace for different
subvolumes.
Without a valid bdev, some user space tools may not be able to distinguish inodes in different subvolumes.Quoted 3 lines
Alright that's fine then. But I feel like stat is one of those things that'll flood the console, can we put this somewhere else that's going to be hit less? Thanks,
Unfortunately,
stat()is the only user ofbtrfs_root::anon_dev.While fortunately, the logical is pretty simple, even without the safe net we can understand the lifespan pretty well.
I'm fine to drop this patch if you're concerned about the possible warning flood, as the benefit is really not that much.
Quoted 6 lines
Quoted 3 lines
It could be a developer-only warning but if there's a root with a bad
anon_dev, a simple 'ls -l' would flood the log for sure.We'll know in
btrfs_init_fs_root()whenget_anon_bdev()fails right? Can't we just complain then? That seems less spammy. Thanks,
That's exactly where we do the allocation, that would be something like warning about the allocation result.
That's why I'm fine to drop this patch if it's too spammy.
Thanks,
Qu
Quoted 2 lines
Josef
PATCH 3/4 btrfs: preallocate anon_dev for subvolume and snapshot creation
In reply to Qu Wenruo[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:
------------[ cut here ]------------ BTRFS: Transaction aborted (error -24) WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs] RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs] Call Trace: create_pending_snapshots+0x82/0xa0 [btrfs] btrfs_commit_transaction+0x275/0x8c0 [btrfs] btrfs_mksubvol+0x4b9/0x500 [btrfs] btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs] btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs] btrfs_ioctl+0x11a4/0x2da0 [btrfs] do_vfs_ioctl+0xa9/0x640 ksys_ioctl+0x67/0x90 __x64_sys_ioctl+0x1a/0x20 do_syscall_64+0x5a/0x110 entry_SYSCALL_64_after_hwframe+0x44/0xa9 ---[ end trace 33f2f83f3d5250e9 ]--- BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown BTRFS info (device sda1): forced readonly BTRFS warning (device sda1): Skipping commit of aborted transaction. BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown
[CAUSE]
When the global anonymous block device pool is exhausted, the following call chain will fail, and lead to transaction abort:
btrfs_ioctl_snap_create_v2()
|- btrfs_ioctl_snap_create_transid()
|- btrfs_mksubvol()
|- btrfs_commit_transaction()
|- create_pending_snapshot()
|- btrfs_get_fs_root()
|- btrfs_init_fs_root()
|- get_anon_bdev()[FIX]
Although we can't enlarge the anonymous block device pool, at least we can preallocate anon_dev for subvolume/snapshot creation. So that when the pool is exhausted, user will get an error other than aborting transaction later.
--- fs/btrfs/disk-io.c | 51 ++++++++++++++++++++++++++++++++++++------ fs/btrfs/disk-io.h | 2 ++ fs/btrfs/ioctl.c | 21 ++++++++++++++++- fs/btrfs/transaction.c | 3 ++- fs/btrfs/transaction.h | 2 ++ 5 files changed, 70 insertions(+), 9 deletions(-)
fs/btrfs/disk-io.c +44 -7
@@ -1395,7 +1395,7 @@ struct btrfs_root *btrfs_read_tree_root(struct btrfs_root *tree_root, goto out; } -static int btrfs_init_fs_root (struct btrfs_root *root) +static int btrfs_init_fs_root (struct btrfs_root *root,dev_t anon_dev) {int ret;unsigned int nofs_flag;
@@ -1435,9 +1435,13 @@ static int btrfs_init_fs_root(struct btrfs_root *root)*/ if (is_fstree (root->root_key.objectid) &&btrfs_root_refs (&root->root_item)) { - ret =get_anon_bdev (&root->anon_dev); -if (ret) - goto fail; +if (!anon_dev) { + ret =get_anon_bdev (&root->anon_dev); +if (ret) + goto fail; + }else { + root->anon_dev = anon_dev; + } }mutex_lock (&root->objectid_mutex);
@@ -1542,8 +1546,27 @@ void btrfs_free_fs_info(struct btrfs_fs_info *fs_info) } -struct btrfs_root *btrfs_get_fs_root (struct btrfs_fs_info *fs_info, -u64 objectid,bool check_ref) +/* +* Get a fs root. +* +* For essential trees like root/extent tree, we grab it from fs_info directly. +* For subvolume trees, we check the cached fs roots first. If miss then +* read it from disk and add it to cached fs roots. +* +* Caller should release the root by calling btrfs_put_root() after the usage. +* +* NOTE: Reloc and log trees can't be read by this function as they share the +* same root objectid. +* +* @objectid: Root (subvolume) id +* @anon_dev: Preallocated anonymous block device number for new roots. +* Pass 0 for automatic allocation. +* @check_ref: Whether to check root refs. If true, return -ENOENT for orphan +* roots. +*/ +static struct btrfs_root *__get_fs_root (struct btrfs_fs_info *fs_info, +u64 objectid,dev_t anon_dev, +bool check_ref) {struct btrfs_root *root;struct btrfs_path *path;
@@ -1572,6 +1595,8 @@ struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info, again: root =btrfs_lookup_fs_root (fs_info, objectid);if (root) { +/* Shouldn't get preallocated anon_dev for cached roots */ +ASSERT (!anon_dev);if (check_ref &&btrfs_root_refs (&root->root_item) ==0 ) {btrfs_put_root (root);return ERR_PTR (-ENOENT );
@@ -1591,7 +1616,7 @@ struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info, goto fail; } - ret =btrfs_init_fs_root (root); + ret =btrfs_init_fs_root (root, anon_dev);if (ret) goto fail;
@@ -1624,6 +1649,18 @@ struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info,return ERR_PTR (ret); } +struct btrfs_root *btrfs_get_fs_root (struct btrfs_fs_info *fs_info, +u64 objectid,bool check_ref) +{ +return __get_fs_root (fs_info, objectid,0 , check_ref); +} + +struct btrfs_root *btrfs_get_new_fs_root (struct btrfs_fs_info *fs_info, +u64 objectid,dev_t anon_dev) +{ +return __get_fs_root (fs_info, objectid, anon_dev, true); +} +static int btrfs_congested_fn (void *congested_data,int bdi_bits) {struct btrfs_fs_info *info = (struct btrfs_fs_info *)congested_data;
fs/btrfs/disk-io.h +2 -0
@@ -67,6 +67,8 @@ void btrfs_free_fs_roots(struct btrfs_fs_info *fs_info);struct btrfs_root *btrfs_get_fs_root (struct btrfs_fs_info *fs_info,u64 objectid,bool check_ref); +struct btrfs_root *btrfs_get_new_fs_root (struct btrfs_fs_info *fs_info, +u64 objectid,dev_t anon_dev);void btrfs_free_fs_info (struct btrfs_fs_info *fs_info);int btrfs_cleanup_fs_roots (struct btrfs_fs_info *fs_info);
fs/btrfs/ioctl.c +20 -1
@@ -566,6 +566,7 @@ static noinline int create_subvol(struct inode *dir,struct inode *inode;int ret;int err; +dev_t anon_dev =0 ;u64 objectid;u64 new_dirid =BTRFS_FIRST_FREE_OBJECTID ;u64 index =0 ;
@@ -578,6 +579,10 @@ static noinline int create_subvol(struct inode *dir,if (ret) goto fail_free; + ret =get_anon_bdev (&anon_dev); +if (ret <0 ) + goto fail_free; +/* * Don't create subvolume whose level is not zero. Or qgroup will be * screwed up since it assumes subvolume qgroup's level to be 0.
@@ -660,12 +665,15 @@ static noinline int create_subvol(struct inode *dir, goto fail; key.offset = (u64)-1 ; - new_root =btrfs_get_fs_root (fs_info, objectid, true); + new_root =btrfs_get_new_fs_root (fs_info, objectid, anon_dev);if (IS_ERR (new_root)) { +free_anon_bdev (anon_dev); ret =PTR_ERR (new_root);btrfs_abort_transaction (trans, ret); goto fail; } +/* Freeing would be done in btrfs_put_root() of @new_root */ + anon_dev =0 ;btrfs_record_root_in_trans (trans, new_root);
@@ -735,6 +743,8 @@ static noinline int create_subvol(struct inode *dir,return ret; fail_free: +if (anon_dev) +free_anon_bdev (anon_dev);kfree (root_item);return ret; }
@@ -762,6 +772,9 @@ static int create_snapshot(struct btrfs_root *root, struct inode *dir,if (!pending_snapshot)return -ENOMEM ; + ret =get_anon_bdev (&pending_snapshot->anon_dev); +if (ret <0 ) + goto free_pending; pending_snapshot->root_item =kzalloc (sizeof (struct btrfs_root_item ),GFP_KERNEL ); pending_snapshot->path =btrfs_alloc_path ();
@@ -823,10 +836,16 @@ static int create_snapshot(struct btrfs_root *root, struct inode *dir,d_instantiate (dentry, inode); ret =0 ; + pending_snapshot->anon_dev =0 ; fail: +/* Prevent doulbe freeing of anon_dev */ +if (ret && pending_snapshot->snap) + pending_snapshot->snap->anon_dev =0 ;btrfs_put_root (pending_snapshot->snap);btrfs_subvolume_release_metadata (fs_info, &pending_snapshot->block_rsv); free_pending: +if (pending_snapshot->anon_dev) +free_anon_bdev (pending_snapshot->anon_dev);kfree (pending_snapshot->root_item);btrfs_free_path (pending_snapshot->path);kfree (pending_snapshot);
fs/btrfs/transaction.c +2 -1
@@ -1630,7 +1630,8 @@ static noinline int create_pending_snapshot(struct btrfs_trans_handle *trans, } key.offset = (u64)-1 ; - pending->snap =btrfs_get_fs_root (fs_info, objectid, true); + pending->snap =btrfs_get_new_fs_root (fs_info, objectid, + pending->anon_dev);if (IS_ERR (pending->snap)) { ret =PTR_ERR (pending->snap);btrfs_abort_transaction (trans, ret);
fs/btrfs/transaction.h +2 -0
@@ -151,6 +151,8 @@ struct btrfs_pending_snapshot {struct btrfs_block_rsv block_rsv;/* extra metadata reservation for relocation */ int error; +/* Preallocated anonymous block device number */ +dev_t anon_dev;bool readonly;struct list_head list; };
Signature · 1 line
2.27.0
Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation
Quoted 116 lines · Qu Wenruo
[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:------------[ cut here ]------------ BTRFS: Transaction aborted (error -24) WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs] RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs] Call Trace: create_pending_snapshots+0x82/0xa0 [btrfs] btrfs_commit_transaction+0x275/0x8c0 [btrfs] btrfs_mksubvol+0x4b9/0x500 [btrfs] btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs] btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs] btrfs_ioctl+0x11a4/0x2da0 [btrfs] do_vfs_ioctl+0xa9/0x640 ksys_ioctl+0x67/0x90 __x64_sys_ioctl+0x1a/0x20 do_syscall_64+0x5a/0x110 entry_SYSCALL_64_after_hwframe+0x44/0xa9 ---[ end trace 33f2f83f3d5250e9 ]--- BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown BTRFS info (device sda1): forced readonly BTRFS warning (device sda1): Skipping commit of aborted transaction. BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown[CAUSE]
When the global anonymous block device pool is exhausted, the following call chain will fail, and lead to transaction abort:
btrfs_ioctl_snap_create_v2()
|-btrfs_ioctl_snap_create_transid()|- btrfs_mksubvol() |- btrfs_commit_transaction() |- create_pending_snapshot() |- btrfs_get_fs_root() |- btrfs_init_fs_root() |- get_anon_bdev()[FIX]
Although we can't enlarge the anonymous block device pool, at least we can preallocateanon_devfor subvolume/snapshot creation. So that when the pool is exhausted, user will get an error other than aborting transaction later.Reported-by Greed Rong <[email protected]>Link https://lore.kernel.org/linux-btrfs/CA+UqX+NTrZ6boGnWHhSeZmEY5J76CTqmYjO2S+=tHJX7nb9DPw@mail.gmail.com/ open hereSigned-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 51 ++++++++++++++++++++++++++++++++++++------ fs/btrfs/disk-io.h | 2 ++ fs/btrfs/ioctl.c | 21 ++++++++++++++++- fs/btrfs/transaction.c | 3 ++- fs/btrfs/transaction.h | 2 ++ 5 files changed, 70 insertions(+), 9 deletions(-)fs/btrfs/disk-io.c +30
-6@@ -1395,7 +1395,7 @@ struct btrfs_root *btrfs_read_tree_root(struct btrfs_root *tree_root, goto out; } -static int btrfs_init_fs_root (struct btrfs_root *root) +static int btrfs_init_fs_root (struct btrfs_root *root,dev_t anon_dev) {int ret;unsigned int nofs_flag;@@ -1435,9 +1435,13 @@ static int btrfs_init_fs_root(struct btrfs_root *root)*/ if (is_fstree (root->root_key.objectid) &&btrfs_root_refs (&root->root_item)) { - ret =get_anon_bdev (&root->anon_dev); -if (ret) - goto fail; +if (!anon_dev) { + ret =get_anon_bdev (&root->anon_dev); +if (ret) + goto fail; + }else { + root->anon_dev = anon_dev; + } }mutex_lock (&root->objectid_mutex);@@ -1542,8 +1546,27 @@ void btrfs_free_fs_info(struct btrfs_fs_info *fs_info) } -struct btrfs_root *btrfs_get_fs_root (struct btrfs_fs_info *fs_info, -u64 objectid,bool check_ref) +/* +* Get a fs root. +* +* For essential trees like root/extent tree, we grab it from fs_info directly. +* For subvolume trees, we check the cached fs roots first. If miss then +* read it from disk and add it to cached fs roots. +* +* Caller should release the root by calling btrfs_put_root() after the usage. +* +* NOTE: Reloc and log trees can't be read by this function as they share the +* same root objectid. +* +* @objectid: Root (subvolume) id +* @anon_dev: Preallocated anonymous block device number for new roots. +* Pass 0 for automatic allocation. +* @check_ref: Whether to check root refs. If true, return -ENOENT for orphan +* roots. +*/ +static struct btrfs_root *__get_fs_root (struct btrfs_fs_info *fs_info, +u64 objectid,dev_t anon_dev, +bool check_ref) +struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info,
fs/btrfs/disk-io.c void btrfs_free_fs_info(struct btrfs_fs_info *fs_info)
Quoted 10 lines
fs/btrfs/disk-io.c +10
-0+ u64 objectid, bool check_ref) +{ +return __get_fs_root (fs_info, objectid,0 , check_ref); +} + +struct btrfs_root *btrfs_get_new_fs_root (struct btrfs_fs_info *fs_info, +u64 objectid,dev_t anon_dev) +{ +return __get_fs_root (fs_info, objectid, anon_dev, true); +}
This does not look like a good API, we should keep btrfs_get_fs_root and add the anon_bdev initialization to the callers, there are only a few.
Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation
Quoted 116 lines · David Sterba
Quoted 113 lines · Qu Wenruo
[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:------------[ cut here ]------------ BTRFS: Transaction aborted (error -24) WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs] RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs] Call Trace: create_pending_snapshots+0x82/0xa0 [btrfs] btrfs_commit_transaction+0x275/0x8c0 [btrfs] btrfs_mksubvol+0x4b9/0x500 [btrfs] btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs] btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs] btrfs_ioctl+0x11a4/0x2da0 [btrfs] do_vfs_ioctl+0xa9/0x640 ksys_ioctl+0x67/0x90 __x64_sys_ioctl+0x1a/0x20 do_syscall_64+0x5a/0x110 entry_SYSCALL_64_after_hwframe+0x44/0xa9 ---[ end trace 33f2f83f3d5250e9 ]--- BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown BTRFS info (device sda1): forced readonly BTRFS warning (device sda1): Skipping commit of aborted transaction. BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown[CAUSE]
When the global anonymous block device pool is exhausted, the following call chain will fail, and lead to transaction abort:
btrfs_ioctl_snap_create_v2()
|-btrfs_ioctl_snap_create_transid()|- btrfs_mksubvol() |- btrfs_commit_transaction() |- create_pending_snapshot() |- btrfs_get_fs_root() |- btrfs_init_fs_root() |- get_anon_bdev()[FIX]
Although we can't enlarge the anonymous block device pool, at least we can preallocateanon_devfor subvolume/snapshot creation. So that when the pool is exhausted, user will get an error other than aborting transaction later.Reported-by Greed Rong <[email protected]>Link https://lore.kernel.org/linux-btrfs/CA+UqX+NTrZ6boGnWHhSeZmEY5J76CTqmYjO2S+=tHJX7nb9DPw@mail.gmail.com/ open hereSigned-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 51 ++++++++++++++++++++++++++++++++++++------ fs/btrfs/disk-io.h | 2 ++ fs/btrfs/ioctl.c | 21 ++++++++++++++++- fs/btrfs/transaction.c | 3 ++- fs/btrfs/transaction.h | 2 ++ 5 files changed, 70 insertions(+), 9 deletions(-)fs/btrfs/disk-io.c +29
-6@@ -1395,7 +1395,7 @@ struct btrfs_root *btrfs_read_tree_root(struct btrfs_root *tree_root, goto out; } -static int btrfs_init_fs_root (struct btrfs_root *root) +static int btrfs_init_fs_root (struct btrfs_root *root,dev_t anon_dev) {int ret;unsigned int nofs_flag;@@ -1435,9 +1435,13 @@ static int btrfs_init_fs_root(struct btrfs_root *root)*/ if (is_fstree (root->root_key.objectid) &&btrfs_root_refs (&root->root_item)) { - ret =get_anon_bdev (&root->anon_dev); -if (ret) - goto fail; +if (!anon_dev) { + ret =get_anon_bdev (&root->anon_dev); +if (ret) + goto fail; + }else { + root->anon_dev = anon_dev; + } }mutex_lock (&root->objectid_mutex);@@ -1542,8 +1546,27 @@ void btrfs_free_fs_info(struct btrfs_fs_info *fs_info) } -struct btrfs_root *btrfs_get_fs_root (struct btrfs_fs_info *fs_info, -u64 objectid,bool check_ref) +/* +* Get a fs root. +* +* For essential trees like root/extent tree, we grab it from fs_info directly. +* For subvolume trees, we check the cached fs roots first. If miss then +* read it from disk and add it to cached fs roots. +* +* Caller should release the root by calling btrfs_put_root() after the usage. +* +* NOTE: Reloc and log trees can't be read by this function as they share the +* same root objectid. +* +* @objectid: Root (subvolume) id +* @anon_dev: Preallocated anonymous block device number for new roots. +* Pass 0 for automatic allocation. +* @check_ref: Whether to check root refs. If true, return -ENOENT for orphan +* roots. +*/ +static struct btrfs_root *__get_fs_root (struct btrfs_fs_info *fs_info, +u64 objectid,dev_t anon_dev, +bool check_ref)
Quoted 15 lines
fs/btrfs/disk-io.c struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info, · line 1662
Quoted 11 lines
fs/btrfs/disk-io.c +11
-0+struct btrfs_root *btrfs_get_fs_root (struct btrfs_fs_info *fs_info, +u64 objectid,bool check_ref) +{ +return __get_fs_root (fs_info, objectid,0 , check_ref); +} + +struct btrfs_root *btrfs_get_new_fs_root (struct btrfs_fs_info *fs_info, +u64 objectid,dev_t anon_dev) +{ +return __get_fs_root (fs_info, objectid, anon_dev, true); +}This does not look like a good API, we should keep
btrfs_get_fs_rootand add theanon_bdevinitialization to the callers, there are only a few.
OK, I'd go that direction.
Thanks,
Qu
Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation
In reply to David SterbaQuoted 116 lines · David Sterba
Quoted 113 lines · Qu Wenruo
[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:------------[ cut here ]------------ BTRFS: Transaction aborted (error -24) WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs] RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs] Call Trace: create_pending_snapshots+0x82/0xa0 [btrfs] btrfs_commit_transaction+0x275/0x8c0 [btrfs] btrfs_mksubvol+0x4b9/0x500 [btrfs] btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs] btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs] btrfs_ioctl+0x11a4/0x2da0 [btrfs] do_vfs_ioctl+0xa9/0x640 ksys_ioctl+0x67/0x90 __x64_sys_ioctl+0x1a/0x20 do_syscall_64+0x5a/0x110 entry_SYSCALL_64_after_hwframe+0x44/0xa9 ---[ end trace 33f2f83f3d5250e9 ]--- BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown BTRFS info (device sda1): forced readonly BTRFS warning (device sda1): Skipping commit of aborted transaction. BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown[CAUSE]
When the global anonymous block device pool is exhausted, the following call chain will fail, and lead to transaction abort:
btrfs_ioctl_snap_create_v2()
|-btrfs_ioctl_snap_create_transid()|- btrfs_mksubvol() |- btrfs_commit_transaction() |- create_pending_snapshot() |- btrfs_get_fs_root() |- btrfs_init_fs_root() |- get_anon_bdev()[FIX]
Although we can't enlarge the anonymous block device pool, at least we can preallocateanon_devfor subvolume/snapshot creation. So that when the pool is exhausted, user will get an error other than aborting transaction later.Reported-by Greed Rong <[email protected]>Link https://lore.kernel.org/linux-btrfs/CA+UqX+NTrZ6boGnWHhSeZmEY5J76CTqmYjO2S+=tHJX7nb9DPw@mail.gmail.com/ open hereSigned-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 51 ++++++++++++++++++++++++++++++++++++------ fs/btrfs/disk-io.h | 2 ++ fs/btrfs/ioctl.c | 21 ++++++++++++++++- fs/btrfs/transaction.c | 3 ++- fs/btrfs/transaction.h | 2 ++ 5 files changed, 70 insertions(+), 9 deletions(-)fs/btrfs/disk-io.c +29
-6@@ -1395,7 +1395,7 @@ struct btrfs_root *btrfs_read_tree_root(struct btrfs_root *tree_root, goto out; } -static int btrfs_init_fs_root (struct btrfs_root *root) +static int btrfs_init_fs_root (struct btrfs_root *root,dev_t anon_dev) {int ret;unsigned int nofs_flag;@@ -1435,9 +1435,13 @@ static int btrfs_init_fs_root(struct btrfs_root *root)*/ if (is_fstree (root->root_key.objectid) &&btrfs_root_refs (&root->root_item)) { - ret =get_anon_bdev (&root->anon_dev); -if (ret) - goto fail; +if (!anon_dev) { + ret =get_anon_bdev (&root->anon_dev); +if (ret) + goto fail; + }else { + root->anon_dev = anon_dev; + } }mutex_lock (&root->objectid_mutex);@@ -1542,8 +1546,27 @@ void btrfs_free_fs_info(struct btrfs_fs_info *fs_info) } -struct btrfs_root *btrfs_get_fs_root (struct btrfs_fs_info *fs_info, -u64 objectid,bool check_ref) +/* +* Get a fs root. +* +* For essential trees like root/extent tree, we grab it from fs_info directly. +* For subvolume trees, we check the cached fs roots first. If miss then +* read it from disk and add it to cached fs roots. +* +* Caller should release the root by calling btrfs_put_root() after the usage. +* +* NOTE: Reloc and log trees can't be read by this function as they share the +* same root objectid. +* +* @objectid: Root (subvolume) id +* @anon_dev: Preallocated anonymous block device number for new roots. +* Pass 0 for automatic allocation. +* @check_ref: Whether to check root refs. If true, return -ENOENT for orphan +* roots. +*/ +static struct btrfs_root *__get_fs_root (struct btrfs_fs_info *fs_info, +u64 objectid,dev_t anon_dev, +bool check_ref)
Quoted 15 lines
fs/btrfs/disk-io.c struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info, · line 1662
Quoted 11 lines
fs/btrfs/disk-io.c +11
-0+struct btrfs_root *btrfs_get_fs_root (struct btrfs_fs_info *fs_info, +u64 objectid,bool check_ref) +{ +return __get_fs_root (fs_info, objectid,0 , check_ref); +} + +struct btrfs_root *btrfs_get_new_fs_root (struct btrfs_fs_info *fs_info, +u64 objectid,dev_t anon_dev) +{ +return __get_fs_root (fs_info, objectid, anon_dev, true); +}This does not look like a good API, we should keep
btrfs_get_fs_rootand add theanon_bdevinitialization to the callers, there are only a few.
A few = over 25?
I have switched to keep btrfs_get_fs_root(), but you won't like the summary:
Old:
fs/btrfs/disk-io.h | 2 ++ fs/btrfs/ioctl.c | 21 ++++++++++++++++- fs/btrfs/transaction.c | 3 ++- fs/btrfs/transaction.h | 2 ++ 5 files changed, 70 insertions(+), 9 deletions(-)
New:
fs/btrfs/backref.c | 4 ++-- fs/btrfs/disk-io.c | 42 ++++++++++++++++++++++++++++++++++-------- fs/btrfs/disk-io.h | 3 ++- fs/btrfs/export.c | 2 +- fs/btrfs/file.c | 2 +- fs/btrfs/inode.c | 2 +- fs/btrfs/ioctl.c | 31 +++++++++++++++++++++++++------ fs/btrfs/relocation.c | 11 ++++++----- fs/btrfs/root-tree.c | 2 +- fs/btrfs/scrub.c | 2 +- fs/btrfs/send.c | 4 ++-- fs/btrfs/super.c | 2 +- fs/btrfs/transaction.c | 3 ++- fs/btrfs/transaction.h | 2 ++ fs/btrfs/tree-log.c | 2 +- fs/btrfs/uuid-tree.c | 2 +- 16 files changed, 83 insertions(+), 33 deletions(-)
Do we really go that direction?
Thanks,
Qu
Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation
Quoted 40 lines · Qu Wenruo
Quoted 9 lines
fs/btrfs/disk-io.c struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info, · line 1662
Quoted 5 lines
fs/btrfs/disk-io.c +5
-0+struct btrfs_root *btrfs_get_new_fs_root (struct btrfs_fs_info *fs_info, +u64 objectid,dev_t anon_dev) +{ +return __get_fs_root (fs_info, objectid, anon_dev, true); +}This does not look like a good API, we should keep
btrfs_get_fs_rootand add theanon_bdevinitialization to the callers, there are only a few.A few = over 25?
I have switched to keep
btrfs_get_fs_root(), but you won't like the summary:Old:
fs/btrfs/disk-io.h | 2 ++ fs/btrfs/ioctl.c | 21 ++++++++++++++++- fs/btrfs/transaction.c | 3 ++- fs/btrfs/transaction.h | 2 ++ 5 files changed, 70 insertions(+), 9 deletions(-)New:
fs/btrfs/backref.c | 4 ++-- fs/btrfs/disk-io.c | 42 ++++++++++++++++++++++++++++++++++-------- fs/btrfs/disk-io.h | 3 ++- fs/btrfs/export.c | 2 +- fs/btrfs/file.c | 2 +- fs/btrfs/inode.c | 2 +- fs/btrfs/ioctl.c | 31 +++++++++++++++++++++++++------ fs/btrfs/relocation.c | 11 ++++++----- fs/btrfs/root-tree.c | 2 +- fs/btrfs/scrub.c | 2 +- fs/btrfs/send.c | 4 ++-- fs/btrfs/super.c | 2 +- fs/btrfs/transaction.c | 3 ++- fs/btrfs/transaction.h | 2 ++ fs/btrfs/tree-log.c | 2 +- fs/btrfs/uuid-tree.c | 2 +- 16 files changed, 83 insertions(+), 33 deletions(-)
Quoted 1 line
Do we really go that direction?
You're right, I don't like the summary and I don't like the code either.
Adding the anon_dev argument to btrfs_get_fs_root is wrong and I have never suggested that. What I meant is to put the actual id allocation to the callers where the subvolume is created, ie only 2 places.
Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation
Quoted 41 lines · David Sterba
Quoted 40 lines · Qu Wenruo
Quoted 9 lines
fs/btrfs/disk-io.c struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info, · line 1662
Quoted 5 lines
fs/btrfs/disk-io.c +5
-0+struct btrfs_root *btrfs_get_new_fs_root (struct btrfs_fs_info *fs_info, +u64 objectid,dev_t anon_dev) +{ +return __get_fs_root (fs_info, objectid, anon_dev, true); +}This does not look like a good API, we should keep
btrfs_get_fs_rootand add theanon_bdevinitialization to the callers, there are only a few.A few = over 25?
I have switched to keep
btrfs_get_fs_root(), but you won't like the summary:Old:
fs/btrfs/disk-io.h | 2 ++ fs/btrfs/ioctl.c | 21 ++++++++++++++++- fs/btrfs/transaction.c | 3 ++- fs/btrfs/transaction.h | 2 ++ 5 files changed, 70 insertions(+), 9 deletions(-)New:
fs/btrfs/backref.c | 4 ++-- fs/btrfs/disk-io.c | 42 ++++++++++++++++++++++++++++++++++-------- fs/btrfs/disk-io.h | 3 ++- fs/btrfs/export.c | 2 +- fs/btrfs/file.c | 2 +- fs/btrfs/inode.c | 2 +- fs/btrfs/ioctl.c | 31 +++++++++++++++++++++++++------ fs/btrfs/relocation.c | 11 ++++++----- fs/btrfs/root-tree.c | 2 +- fs/btrfs/scrub.c | 2 +- fs/btrfs/send.c | 4 ++-- fs/btrfs/super.c | 2 +- fs/btrfs/transaction.c | 3 ++- fs/btrfs/transaction.h | 2 ++ fs/btrfs/tree-log.c | 2 +- fs/btrfs/uuid-tree.c | 2 +- 16 files changed, 83 insertions(+), 33 deletions(-)
Quoted 8 lines
Quoted 1 line
Do we really go that direction?
You're right, I don't like the summary and I don't like the code either.
Adding the
anon_devargument tobtrfs_get_fs_rootis wrong and I have never suggested that. What I meant is to put the actual id allocation to the callers where the subvolume is created, ie only 2 places.
You mean to extract btrfs_init_fs_root() out of btrfs_get_fs_root()?
That looks a little risky and I can't find any good solution to make it more elegant than the current one.
Although I would definitely remove the "__" prefix as we shouldn't add such prefix anymore.
Thanks,
Qu
Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation
Quoted 10 lines · Qu Wenruo
Quoted 4 lines · David Sterba
On Wed, Jul 01, 2020 at 11:25:27AM +0800, Qu Wenruo wrote:
Adding theanon_devargument tobtrfs_get_fs_rootis wrong and I have never suggested that. What I meant is to put the actual id allocation to the callers where the subvolume is created, ie only 2 places.You mean to extract
btrfs_init_fs_root()out ofbtrfs_get_fs_root()?That looks a little risky and I can't find any good solution to make it more elegant than the current one.
I spent more time reading through the get-fs-root functions and the main problem is that btrfs_get_fs_root is doing several things, and it makes a lot of code simple, I certainly want to keep it that way.
The idea was to pre-insert the new root (similar to the root item insertion, btrfs_insert_root) and not letting btrfs_get_fs_root call to btrfs_init_fs_info where the anon_bdev allocation happens for all the other non-ioctl cases.
Which could be done by factoring out btrfs_init_fs_root from btrfs_get_fs_root. This would allow to extend only btrfs_init_fs_root arguments with the anon_bdev, and keep btrfs_get_fs_root intact. So this is splitting the API from the end.
What you originally proposed is a split from the begnning, ie. add a common implementation for existing and new and provide btrfs_get_fs_root and btrfs_get_new_fs_root that would hide the additional parameters.
Both ways are IMO valid but I thought it would be easier to pass the anon bdev inside ioctl callbacks. The problem that makes my proposal less appealing is that btrfs_read_tree_root gets called earlier than I'd like so factoring everything after btrfs_init_fs_root would not be so straightforward.
In conclusion, your proposal is better and I'm going to merge it.
Quoted 2 lines
Although I would definitely remove the "__" prefix as we shouldn't add such prefix anymore.
Yeah with the small naming fixups.
Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation
Quoted 36 lines · David Sterba
Quoted 10 lines · Qu Wenruo
Quoted 4 lines · David Sterba
On Wed, Jul 01, 2020 at 11:25:27AM +0800, Qu Wenruo wrote:
Adding theanon_devargument tobtrfs_get_fs_rootis wrong and I have never suggested that. What I meant is to put the actual id allocation to the callers where the subvolume is created, ie only 2 places.You mean to extract
btrfs_init_fs_root()out ofbtrfs_get_fs_root()?That looks a little risky and I can't find any good solution to make it more elegant than the current one.
I spent more time reading through the get-fs-root functions and the main problem is that
btrfs_get_fs_rootis doing several things, and it makes a lot of code simple, I certainly want to keep it that way.The idea was to pre-insert the new root (similar to the root item insertion,
btrfs_insert_root) and not lettingbtrfs_get_fs_rootcall tobtrfs_init_fs_infowhere theanon_bdevallocation happens for all the other non-ioctl cases.Which could be done by factoring out
btrfs_init_fs_rootfrombtrfs_get_fs_root. This would allow to extend onlybtrfs_init_fs_rootarguments with theanon_bdev, and keepbtrfs_get_fs_rootintact. So this is splitting the API from the end.What you originally proposed is a split from the begnning, ie. add a common implementation for existing and new and provide
btrfs_get_fs_rootandbtrfs_get_new_fs_rootthat would hide the additional parameters.Both ways are IMO valid but I thought it would be easier to pass the anon bdev inside ioctl callbacks. The problem that makes my proposal less appealing is that
btrfs_read_tree_rootgets called earlier than I'd like so factoring everything afterbtrfs_init_fs_rootwould not be so straightforward.
Quoted 6 lines
In conclusion, your proposal is better and I'm going to merge it.
Quoted 2 lines
Although I would definitely remove the "__" prefix as we shouldn't add such prefix anymore.
Yeah with the small naming fixups.
It's in for-next-20200703. I've updated the changelogs to reflect what we found during debugging the issue, the __ function renamed to btrfs_get_root_ref and some function comments added. All patches reordered and tagged for stable though the preallocation is not within the size limit.
Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation
Quoted 42 lines · David Sterba
Quoted 41 lines · David Sterba
Quoted 10 lines · Qu Wenruo
Quoted 4 lines · David Sterba
On Wed, Jul 01, 2020 at 11:25:27AM +0800, Qu Wenruo wrote:
Adding theanon_devargument tobtrfs_get_fs_rootis wrong and I have never suggested that. What I meant is to put the actual id allocation to the callers where the subvolume is created, ie only 2 places.You mean to extract
btrfs_init_fs_root()out ofbtrfs_get_fs_root()?That looks a little risky and I can't find any good solution to make it more elegant than the current one.
I spent more time reading through the get-fs-root functions and the main problem is that
btrfs_get_fs_rootis doing several things, and it makes a lot of code simple, I certainly want to keep it that way.The idea was to pre-insert the new root (similar to the root item insertion,
btrfs_insert_root) and not lettingbtrfs_get_fs_rootcall tobtrfs_init_fs_infowhere theanon_bdevallocation happens for all the other non-ioctl cases.Which could be done by factoring out
btrfs_init_fs_rootfrombtrfs_get_fs_root. This would allow to extend onlybtrfs_init_fs_rootarguments with theanon_bdev, and keepbtrfs_get_fs_rootintact. So this is splitting the API from the end.What you originally proposed is a split from the begnning, ie. add a common implementation for existing and new and provide
btrfs_get_fs_rootandbtrfs_get_new_fs_rootthat would hide the additional parameters.Both ways are IMO valid but I thought it would be easier to pass the anon bdev inside ioctl callbacks. The problem that makes my proposal less appealing is that
btrfs_read_tree_rootgets called earlier than I'd like so factoring everything afterbtrfs_init_fs_rootwould not be so straightforward.In conclusion, your proposal is better and I'm going to merge it.
Quoted 2 lines
Although I would definitely remove the "__" prefix as we shouldn't add such prefix anymore.
Quoted 8 lines
Quoted 1 line
Yeah with the small naming fixups.
It's in for-next-20200703. I've updated the changelogs to reflect what we found during debugging the issue, the __ function renamed to
btrfs_get_root_refand some function comments added. All patches reordered and tagged for stable though the preallocation is not within the size limit.
Thanks for the merge and dropping the unneeded check patch.
All the modification looks good to me.
Just a small nitpick for commit a561defc34aa ("btrfs: don't allocate anonymous block device for user invisible roots"), there is an unnecessary new line after "[CAUSE]".
Thanks for your daily work of maintaining btrfs,
Qu
Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation
Quoted 14 lines · Qu Wenruo
Quoted 5 lines
Quoted 5 lines
In conclusion, your proposal is better and I'm going to merge it.
Quoted 2 lines
Although I would definitely remove the "__" prefix as we shouldn't add such prefix anymore.
Quoted 8 lines
Quoted 1 line
Yeah with the small naming fixups.
It's in for-next-20200703. I've updated the changelogs to reflect what we found during debugging the issue, the __ function renamed to
btrfs_get_root_refand some function comments added. All patches reordered and tagged for stable though the preallocation is not within the size limit.
Quoted 7 lines
Thanks for the merge and dropping the unneeded check patch.
All the modification looks good to me.
Just a small nitpick for commit
a561defc34aa("btrfs: don't allocate anonymous block device for user invisible roots"), there is an unnecessary new line after "[CAUSE]".
Fixed, thanks for checking. I'm not entirely satisfied with the name of btrfs_get_root_ref, as it could be confused with the on-disk btrfs_root_ref. The get-ref functions could use some cleanup as btrfs_get_fs_root is sometimes used for non-filesystem roots. Adding a generic get_any_root that would accept any tree and btrfs_get_fs_root would be only for subvolume roots or perhaps related trees like data reloc. But this is not essential for the anon bdev fixes so that's for later.
Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation
Quoted 19 lines · David Sterba
Quoted 18 lines · Qu Wenruo
Quoted 5 lines
Quoted 5 lines
In conclusion, your proposal is better and I'm going to merge it.
Quoted 2 lines
Although I would definitely remove the "__" prefix as we shouldn't add such prefix anymore.
Quoted 8 lines
Quoted 1 line
Yeah with the small naming fixups.
It's in for-next-20200703. I've updated the changelogs to reflect what we found during debugging the issue, the __ function renamed to
btrfs_get_root_refand some function comments added. All patches reordered and tagged for stable though the preallocation is not within the size limit.Thanks for the merge and dropping the unneeded check patch.
All the modification looks good to me.
Quoted 7 lines
Quoted 3 lines
Just a small nitpick for commit
a561defc34aa("btrfs: don't allocate anonymous block device for user invisible roots"), there is an unnecessary new line after "[CAUSE]".Fixed, thanks for checking. I'm not entirely satisfied with the name of
btrfs_get_root_ref, as it could be confused with the on-diskbtrfs_root_ref.
My original plan is even simpler, just get_fs_root(), remove the btrfs_ prefix...
Quoted 6 lines
The get-ref functions could use some cleanup as
btrfs_get_fs_rootis sometimes used for non-filesystem roots. Adding a genericget_any_rootthat would accept any tree andbtrfs_get_fs_rootwould be only for subvolume roots or perhaps related trees like data reloc. But this is not essential for the anon bdev fixes so that's for later.
No problem.
Thanks,
Qu
PATCH 4/4 btrfs: free anon_dev earlier to prevent exhausting anonymous block device pool
In reply to Qu Wenruo[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:
------------[ cut here ]------------ BTRFS: Transaction aborted (error -24) WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs] RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs] Call Trace: create_pending_snapshots+0x82/0xa0 [btrfs] btrfs_commit_transaction+0x275/0x8c0 [btrfs] btrfs_mksubvol+0x4b9/0x500 [btrfs] btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs] btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs] btrfs_ioctl+0x11a4/0x2da0 [btrfs] do_vfs_ioctl+0xa9/0x640 ksys_ioctl+0x67/0x90 __x64_sys_ioctl+0x1a/0x20 do_syscall_64+0x5a/0x110 entry_SYSCALL_64_after_hwframe+0x44/0xa9 ---[ end trace 33f2f83f3d5250e9 ]--- BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown BTRFS info (device sda1): forced readonly BTRFS warning (device sda1): Skipping commit of aborted transaction. BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown
[CAUSE]
The root cause is we don't have unlimited resource for anonymous block device number.
The anonymous block device pool only contains 1<<20 devices, and is shared across a several fses, like ceph and overlayfs.
While btrfs has support for 1<<48 subvolumes, so it's just a problem of time to hit such limit.
[WORKAROUND]
Here we can free btrfs_root::anon_dev as long as the subvolume is no longer visible to users.
By freeing it earlier we reclaim the anon_dev quicker, hopefully to reduce the chance of exhausting the pool.
--- fs/btrfs/inode.c | 2 ++ 1 file changed, 2 insertions(+)
fs/btrfs/inode.c +2 -0
@@ -4045,6 +4045,8 @@ int btrfs_delete_subvolume(struct inode *dir, struct dentry *dentry) } } +free_anon_bdev (dest->anon_dev); + dest->anon_dev =0 ; out_end_trans: trans->block_rsv =NULL ; trans->bytes_reserved =0 ;
Signature · 1 line
2.27.0
Re: [PATCH 4/4] btrfs: free anon_dev earlier to prevent exhausting anonymous block device pool
Quoted 35 lines · Qu Wenruo
[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:------------[ cut here ]------------ BTRFS: Transaction aborted (error -24) WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs] RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs] Call Trace: create_pending_snapshots+0x82/0xa0 [btrfs] btrfs_commit_transaction+0x275/0x8c0 [btrfs] btrfs_mksubvol+0x4b9/0x500 [btrfs] btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs] btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs] btrfs_ioctl+0x11a4/0x2da0 [btrfs] do_vfs_ioctl+0xa9/0x640 ksys_ioctl+0x67/0x90 __x64_sys_ioctl+0x1a/0x20 do_syscall_64+0x5a/0x110 entry_SYSCALL_64_after_hwframe+0x44/0xa9 ---[ end trace 33f2f83f3d5250e9 ]--- BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown BTRFS info (device sda1): forced readonly BTRFS warning (device sda1): Skipping commit of aborted transaction. BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown[CAUSE]
The root cause is we don't have unlimited resource for anonymous block device number.
The anonymous block device pool only contains 1<<20 devices, and is shared across a several fses, like ceph and overlayfs.While btrfs has support for 1<<48 subvolumes, so it's just a problem of time to hit such limit.
Quoted 7 lines
[WORKAROUND]
Here we can freebtrfs_root::anon_devas long as the subvolume is no longer visible to users.By freeing it earlier we reclaim the
anon_devquicker, hopefully to reduce the chance of exhausting the pool.
Why isn't this happening as part of the root teardown once all the references to it are gone? Thanks,
Josef
Re: [PATCH 4/4] btrfs: free anon_dev earlier to prevent exhausting anonymous block device pool
Quoted 5 lines · Josef Bacik
Quoted 2 lines
By freeing it earlier we reclaim the
anon_devquicker, hopefully to reduce the chance of exhausting the pool.Why isn't this happening as part of the root teardown once all the references to it are gone? Thanks,
This is where it happens now and is correct. The problem is that deleted subvolumes keep the id allocated until they are cleaned up, ie. all the dead roots consume the id though we don't need it anymore. Creating and deleting snapshots at large will produce a long list of dead subvolumes. THis patch will return the ids at the earlies possible moment so they can get reused.
Josef Bacik
Reviewed-byRe: [PATCH 4/4] btrfs: free anon_dev earlier to prevent exhausting anonymous block device pool
Quoted 7 lines · David Sterba
Quoted 5 lines · Josef Bacik
Quoted 2 lines
By freeing it earlier we reclaim the
anon_devquicker, hopefully to reduce the chance of exhausting the pool.Why isn't this happening as part of the root teardown once all the references to it are gone? Thanks,
Quoted 7 lines
This is where it happens now and is correct. The problem is that deleted subvolumes keep the id allocated until they are cleaned up, ie. all the dead roots consume the id though we don't need it anymore. Creating and deleting snapshots at large will produce a long list of dead subvolumes. THis patch will return the ids at the earlies possible moment so they can get reused.
Oh ok I misread, we're doing it earlier on purpose. Alright that's fine, you can add
Thanks,
Josef
Quoted 8 lines · Qu Wenruo
Qu Wenruo (4):
btrfs: disk-io: don't allocate anonymous block device for user
invisible roots
btrfs: detect uninitializedbtrfs_root::anon_devfor user visible
subvolumes
btrfs: preallocateanon_devfor subvolume and snapshot creation
btrfs: freeanon_devearlier to prevent exhausting anonymous block
device pool
I haven't seen an update to this patchset and would really like to get it fixed. Let me know if I missed an update or you need help, there weren't any big issues left.