Skip to contentSkip to conversation

btrfs: workaround exhausted anonymous block device pool

PATCH In review Qu Wenruo 4 participants 27 messages Jun 16, 2020 – Jul 3, 2020 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

cover btrfs: workaround exhausted anonymous block device pool

There is a user report about transaction abort with a rare error number, -EMFILE:

  ------------[ cut here ]------------
  BTRFS: Transaction aborted (error -24)
  WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs]
  RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs]
  Call Trace:
   create_pending_snapshots+0x82/0xa0 [btrfs]
   btrfs_commit_transaction+0x275/0x8c0 [btrfs]
   btrfs_mksubvol+0x4b9/0x500 [btrfs]
   btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs]
   btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs]
   btrfs_ioctl+0x11a4/0x2da0 [btrfs]
   do_vfs_ioctl+0xa9/0x640
   ksys_ioctl+0x67/0x90
   __x64_sys_ioctl+0x1a/0x20
   do_syscall_64+0x5a/0x110
   entry_SYSCALL_64_after_hwframe+0x44/0xa9
  ---[ end trace 33f2f83f3d5250e9 ]---
  BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown
  BTRFS info (device sda1): forced readonly
  BTRFS warning (device sda1): Skipping commit of aborted transaction.
  BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown

It turns out that, it's caused by exhausted anonymous block device pool. For that case, we don't have any good way to enlarge that pool to match btrfs subvolume limits.

But still we can improve the situation not to abort transaction, but report error gracefully to user.

This patchset will:
- Reduce the user of anonymous block device pool
Although it can only reduce the user by data reloc tree and orphan roots, it should still save save several ids.

- Catch the unintialized btrfs_root::anon_dev for user visible roots Just an extra safe net for previous patch

- Preallocate anon_dev for snapshot/subvolume creation
So user will get a graceful error other than transaction abort This patch should be the core of the patchset, as the remaining part won't have much user visible effect.

- Free anon_dev earlier to reduce the lifespan of anon_dev

Qu Wenruo (4):
btrfs: disk-io: don't allocate anonymous block device for user
invisible roots
btrfs: detect uninitialized btrfs_root::anon_dev for user visible
subvolumes
btrfs: preallocate anon_dev for subvolume and snapshot creation
btrfs: free anon_dev earlier to prevent exhausting anonymous block
device pool

 fs/btrfs/disk-io.c     | 59 +++++++++++++++++++++++++++++++++++++-----
 fs/btrfs/disk-io.h     |  2 ++
 fs/btrfs/inode.c       | 11 +++++++-
 fs/btrfs/ioctl.c       | 21 ++++++++++++++-
 fs/btrfs/transaction.c |  3 ++-
 fs/btrfs/transaction.h |  2 ++
 6 files changed, 88 insertions(+), 10 deletions(-)
Signature · 1 line
2.27.0
author
#1 lore raw

PATCH 1/4 btrfs: disk-io: don't allocate anonymous block device for user invisible roots

[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:

  ------------[ cut here ]------------
  BTRFS: Transaction aborted (error -24)
  WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs]
  RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs]
  Call Trace:
   create_pending_snapshots+0x82/0xa0 [btrfs]
   btrfs_commit_transaction+0x275/0x8c0 [btrfs]
   btrfs_mksubvol+0x4b9/0x500 [btrfs]
   btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs]
   btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs]
   btrfs_ioctl+0x11a4/0x2da0 [btrfs]
   do_vfs_ioctl+0xa9/0x640
   ksys_ioctl+0x67/0x90
   __x64_sys_ioctl+0x1a/0x20
   do_syscall_64+0x5a/0x110
   entry_SYSCALL_64_after_hwframe+0x44/0xa9
  ---[ end trace 33f2f83f3d5250e9 ]---
  BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown
  BTRFS info (device sda1): forced readonly
  BTRFS warning (device sda1): Skipping commit of aborted transaction.
  BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown

[CAUSE]
The root cause is we don't have unlimited resource for anonymous block device number.
The anonymous block device pool only contains 1<<20 devices, and is shared across a several fses, like ceph and overlayfs.

While btrfs has support for 1<<48 subvolumes, so it's just a problem of time to hit such limit.

[WORKAROUND]
Since it's not possible to completely solve the problem, we can only workaround it.

Firstly, we can reduce the user of anon_dev. Data reloc tree is not visible to users, thus it doesn't need anon_dev at all.

This patch will do extra check on root objectid, to rule out roots who don't need anon_dev.
Although currently it's only data reloc tree and orphan roots.

---
 fs/btrfs/disk-io.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
fs/btrfs/disk-io.c +11 -3
index c70d47b8090a..cfc0ff288238 100644
@@ -1428,9 +1428,17 @@ static int btrfs_init_fs_root(struct btrfs_root *root)
 	spin_lock_init(&root->ino_cache_lock);
 	init_waitqueue_head(&root->ino_cache_wait);
 
-	ret = get_anon_bdev(&root->anon_dev);
-	if (ret)
-		goto fail;
+	/*
+	 * Anonymous block device pool has limited size (1M), which is way
+	 * smaller than btrfs subvolumes limits (1<<48).
+	 * We shouldn't allocate any if it's not a user visible subvolume.
+	 */
+	if (is_fstree(root->root_key.objectid) &&
+	    btrfs_root_refs(&root->root_item)) {
+		ret = get_anon_bdev(&root->anon_dev);
+		if (ret)
+			goto fail;
+	}
 
 	mutex_lock(&root->objectid_mutex);
 	ret = btrfs_find_highest_objectid(root,
Signature · 1 line
2.27.0
Reviewed-by on v1 1/4
#2 lore raw

Re: [PATCH 1/4] btrfs: disk-io: don't allocate anonymous block device for user invisible roots

Quoted 42 lines · Qu Wenruo

[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:

   ------------[ cut here ]------------
   BTRFS: Transaction aborted (error -24)
   WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs]
   RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs]
   Call Trace:
    create_pending_snapshots+0x82/0xa0 [btrfs]
    btrfs_commit_transaction+0x275/0x8c0 [btrfs]
    btrfs_mksubvol+0x4b9/0x500 [btrfs]
    btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs]
    btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs]
    btrfs_ioctl+0x11a4/0x2da0 [btrfs]
    do_vfs_ioctl+0xa9/0x640
    ksys_ioctl+0x67/0x90
    __x64_sys_ioctl+0x1a/0x20
    do_syscall_64+0x5a/0x110
    entry_SYSCALL_64_after_hwframe+0x44/0xa9
   ---[ end trace 33f2f83f3d5250e9 ]---
   BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown
   BTRFS info (device sda1): forced readonly
   BTRFS warning (device sda1): Skipping commit of aborted transaction.
   BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown

[CAUSE]
The root cause is we don't have unlimited resource for anonymous block device number.
The anonymous block device pool only contains 1<<20 devices, and is shared across a several fses, like ceph and overlayfs.

While btrfs has support for 1<<48 subvolumes, so it's just a problem of time to hit such limit.

[WORKAROUND]
Since it's not possible to completely solve the problem, we can only workaround it.

Firstly, we can reduce the user of anon_dev. Data reloc tree is not visible to users, thus it doesn't need anon_dev at all.

Quoted 7 lines

This patch will do extra check on root objectid, to rule out roots who don't need anon_dev.
Although currently it's only data reloc tree and orphan roots.

Thanks,

Josef

author
#3 lore raw

PATCH 2/4 btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes

In reply to Qu Wenruo

btrfs_root::anon_dev is an indicator for different subvolume namespaces. Thus it should always be initialized to an anonymous block device.

Add a safe net to catch such uninitialized values.

---
 fs/btrfs/inode.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
fs/btrfs/inode.c +8 -1
index 00647e8cf2df..195aac71fe32 100644
@@ -8697,6 +8697,7 @@ static int btrfs_getattr(const struct path *path, struct kstat *stat,
 {
 	u64 delalloc_bytes;
 	struct inode *inode = d_inode(path->dentry);
+	struct btrfs_root *root = BTRFS_I(inode)->root;
 	u32 blocksize = inode->i_sb->s_blocksize;
 	u32 bi_flags = BTRFS_I(inode)->flags;
 
@@ -8718,7 +8719,13 @@ static int btrfs_getattr(const struct path *path, struct kstat *stat,
 				  STATX_ATTR_NODUMP);
 
 	generic_fillattr(inode, stat);
-	stat->dev = BTRFS_I(inode)->root->anon_dev;
+	if (unlikely(!root->anon_dev)) {
+		WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
+		btrfs_warn(root->fs_info,
+		"uninitialized btrfs_root::anon_dev detected, root=%llu",
+			   root->root_key.objectid);
+	}
+	stat->dev = root->anon_dev;
 
 	spin_lock(&BTRFS_I(inode)->lock);
 	delalloc_bytes = BTRFS_I(inode)->new_delalloc_bytes;
Signature · 1 line
2.27.0
#4 lore raw

Re: [PATCH 2/4] btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes

Quoted 6 lines · Qu Wenruo

btrfs_root::anon_dev is an indicator for different subvolume namespaces. Thus it should always be initialized to an anonymous block device.

Add a safe net to catch such uninitialized values.

Can we handle stat->dev not having a device set? Or will this blow up in other ways? Thanks,

Josef

#5 lore raw

Re: [PATCH 2/4] btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes

Quoted 10 lines · Josef Bacik
Quoted 6 lines · Qu Wenruo

btrfs_root::anon_dev is an indicator for different subvolume namespaces. Thus it should always be initialized to an anonymous block device.

Add a safe net to catch such uninitialized values.

Can we handle stat->dev not having a device set?  Or will this blow up in other ways?  Thanks,

We can handle it without any problem, just users get confused.

As a common practice, we use different bdev as a namespace for different subvolumes.
Without a valid bdev, some user space tools may not be able to distinguish inodes in different subvolumes.

Thanks,
Qu

Quoted 2 lines

Josef

#6 lore raw

Re: [PATCH 2/4] btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes

Quoted 14 lines · Qu Wenruo
Quoted 10 lines · Josef Bacik
Quoted 6 lines · Qu Wenruo

btrfs_root::anon_dev is an indicator for different subvolume namespaces. Thus it should always be initialized to an anonymous block device.

Add a safe net to catch such uninitialized values.

Can we handle stat->dev not having a device set?  Or will this blow up in other ways?  Thanks,

Quoted 7 lines

We can handle it without any problem, just users get confused.

As a common practice, we use different bdev as a namespace for different subvolumes.
Without a valid bdev, some user space tools may not be able to distinguish inodes in different subvolumes.

Alright that's fine then. But I feel like stat is one of those things that'll flood the console, can we put this somewhere else that's going to be hit less? Thanks,

Josef

author
#7 lore raw

Re: [PATCH 2/4] btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes

Quoted 24 lines · Josef Bacik
Quoted 15 lines · Qu Wenruo
Quoted 11 lines · Josef Bacik
Quoted 7 lines · Qu Wenruo

btrfs_root::anon_dev is an indicator for different subvolume namespaces.
Thus it should always be initialized to an anonymous block device.

Add a safe net to catch such uninitialized values.

Can we handle stat->dev not having a device set?  Or will this blow up in other ways?  Thanks,

Quoted 7 lines

We can handle it without any problem, just users get confused.

As a common practice, we use different bdev as a namespace for different subvolumes.
Without a valid bdev, some user space tools may not be able to distinguish inodes in different subvolumes.

Quoted 3 lines

Alright that's fine then.  But I feel like stat is one of those things that'll flood the console, can we put this somewhere else that's going to be hit less? Thanks,

Unfortunately, stat() is the only user of btrfs_root::anon_dev.

While fortunately, the logical is pretty simple, even without the safe net we can understand the lifespan pretty well.

I'm fine to drop this patch if you're concerned about the possible warning flood, as the benefit is really not that much.

Thanks,
Qu

Quoted 2 lines

Josef

#8 lore raw

Re: [PATCH 2/4] btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes

Quoted 15 lines · Qu Wenruo
Quoted 11 lines
Quoted 10 lines
Quoted 2 lines

Can we handle stat->dev not having a device set?  Or will this blow up in other ways?  Thanks,

We can handle it without any problem, just users get confused.

As a common practice, we use different bdev as a namespace for different subvolumes.
Without a valid bdev, some user space tools may not be able to distinguish inodes in different subvolumes.

Quoted 3 lines

Alright that's fine then.  But I feel like stat is one of those things that'll flood the console, can we put this somewhere else that's going to be hit less? Thanks,

Quoted 7 lines

Unfortunately, stat() is the only user of btrfs_root::anon_dev.

While fortunately, the logical is pretty simple, even without the safe net we can understand the lifespan pretty well.

I'm fine to drop this patch if you're concerned about the possible warning flood, as the benefit is really not that much.

It could be a developer-only warning but if there's a root with a bad anon_dev, a simple 'ls -l' would flood the log for sure.

#9 lore raw

Re: [PATCH 2/4] btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes

Quoted 18 lines · David Sterba
Quoted 17 lines · Qu Wenruo
Quoted 11 lines
Quoted 10 lines
Quoted 2 lines

Can we handle stat->dev not having a device set?  Or will this blow up in other ways?  Thanks,

We can handle it without any problem, just users get confused.

As a common practice, we use different bdev as a namespace for different subvolumes.
Without a valid bdev, some user space tools may not be able to distinguish inodes in different subvolumes.

Quoted 3 lines

Alright that's fine then.  But I feel like stat is one of those things that'll flood the console, can we put this somewhere else that's going to be hit less? Thanks,

Unfortunately, stat() is the only user of btrfs_root::anon_dev.

Quoted 9 lines
Quoted 5 lines

While fortunately, the logical is pretty simple, even without the safe net we can understand the lifespan pretty well.

I'm fine to drop this patch if you're concerned about the possible warning flood, as the benefit is really not that much.

It could be a developer-only warning but if there's a root with a bad anon_dev, a simple 'ls -l' would flood the log for sure.

We'll know in btrfs_init_fs_root() when get_anon_bdev() fails right? Can't we just complain then? That seems less spammy. Thanks,

Josef

#10 lore raw

Re: [PATCH 2/4] btrfs: detect uninitialized btrfs_root::anon_dev for user visible subvolumes

Quoted 27 lines · Josef Bacik
Quoted 26 lines · David Sterba
Quoted 24 lines · Qu Wenruo
Quoted 13 lines
Quoted 4 lines
Quoted 3 lines

Can we handle stat->dev not having a device set?  Or will this blow up
in other ways?  Thanks,

Quoted 8 lines

We can handle it without any problem, just users get confused.

As a common practice, we use different bdev as a namespace for different
subvolumes.
Without a valid bdev, some user space tools may not be able to distinguish inodes in different subvolumes.

Quoted 3 lines

Alright that's fine then.  But I feel like stat is one of those things that'll flood the console, can we put this somewhere else that's going to be hit less? Thanks,

Unfortunately, stat() is the only user of btrfs_root::anon_dev.

While fortunately, the logical is pretty simple, even without the safe net we can understand the lifespan pretty well.

I'm fine to drop this patch if you're concerned about the possible warning flood, as the benefit is really not that much.

Quoted 6 lines
Quoted 3 lines

It could be a developer-only warning but if there's a root with a bad anon_dev, a simple 'ls -l' would flood the log for sure.

We'll know in btrfs_init_fs_root() when get_anon_bdev() fails right? Can't we just complain then?  That seems less spammy.  Thanks,

That's exactly where we do the allocation, that would be something like warning about the allocation result.

That's why I'm fine to drop this patch if it's too spammy.

Thanks,
Qu

Quoted 2 lines

Josef

author
#11 lore raw

PATCH 3/4 btrfs: preallocate anon_dev for subvolume and snapshot creation

In reply to Qu Wenruo

[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:

  ------------[ cut here ]------------
  BTRFS: Transaction aborted (error -24)
  WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs]
  RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs]
  Call Trace:
   create_pending_snapshots+0x82/0xa0 [btrfs]
   btrfs_commit_transaction+0x275/0x8c0 [btrfs]
   btrfs_mksubvol+0x4b9/0x500 [btrfs]
   btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs]
   btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs]
   btrfs_ioctl+0x11a4/0x2da0 [btrfs]
   do_vfs_ioctl+0xa9/0x640
   ksys_ioctl+0x67/0x90
   __x64_sys_ioctl+0x1a/0x20
   do_syscall_64+0x5a/0x110
   entry_SYSCALL_64_after_hwframe+0x44/0xa9
  ---[ end trace 33f2f83f3d5250e9 ]---
  BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown
  BTRFS info (device sda1): forced readonly
  BTRFS warning (device sda1): Skipping commit of aborted transaction.
  BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown

[CAUSE]
When the global anonymous block device pool is exhausted, the following call chain will fail, and lead to transaction abort:

btrfs_ioctl_snap_create_v2()
|- btrfs_ioctl_snap_create_transid()

    |- btrfs_mksubvol()
       |- btrfs_commit_transaction()
          |- create_pending_snapshot()
             |- btrfs_get_fs_root()
                |- btrfs_init_fs_root()
                   |- get_anon_bdev()

[FIX]
Although we can't enlarge the anonymous block device pool, at least we can preallocate anon_dev for subvolume/snapshot creation. So that when the pool is exhausted, user will get an error other than aborting transaction later.

---
 fs/btrfs/disk-io.c     | 51 ++++++++++++++++++++++++++++++++++++------
 fs/btrfs/disk-io.h     |  2 ++
 fs/btrfs/ioctl.c       | 21 ++++++++++++++++-
 fs/btrfs/transaction.c |  3 ++-
 fs/btrfs/transaction.h |  2 ++
 5 files changed, 70 insertions(+), 9 deletions(-)
fs/btrfs/disk-io.c +44 -7
index cfc0ff288238..14fd69b71cb8 100644
@@ -1395,7 +1395,7 @@ struct btrfs_root *btrfs_read_tree_root(struct btrfs_root *tree_root,
 	goto out;
 }
 
-static int btrfs_init_fs_root(struct btrfs_root *root)
+static int btrfs_init_fs_root(struct btrfs_root *root, dev_t anon_dev)
 {
 	int ret;
 	unsigned int nofs_flag;
@@ -1435,9 +1435,13 @@ static int btrfs_init_fs_root(struct btrfs_root *root)
 	 */
 	if (is_fstree(root->root_key.objectid) &&
 	    btrfs_root_refs(&root->root_item)) {
-		ret = get_anon_bdev(&root->anon_dev);
-		if (ret)
-			goto fail;
+		if (!anon_dev) {
+			ret = get_anon_bdev(&root->anon_dev);
+			if (ret)
+				goto fail;
+		} else {
+			root->anon_dev = anon_dev;
+		}
 	}
 
 	mutex_lock(&root->objectid_mutex);
@@ -1542,8 +1546,27 @@ void btrfs_free_fs_info(struct btrfs_fs_info *fs_info)
 }
 
 
-struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info,
-				     u64 objectid, bool check_ref)
+/*
+ * Get a fs root.
+ *
+ * For essential trees like root/extent tree, we grab it from fs_info directly.
+ * For subvolume trees, we check the cached fs roots first. If miss then
+ * read it from disk and add it to cached fs roots.
+ *
+ * Caller should release the root by calling btrfs_put_root() after the usage.
+ *
+ * NOTE: Reloc and log trees can't be read by this function as they share the
+ *	 same root objectid.
+ *
+ * @objectid:	Root (subvolume) id
+ * @anon_dev:	Preallocated anonymous block device number for new roots.
+ * 		Pass 0 for automatic allocation.
+ * @check_ref:	Whether to check root refs. If true, return -ENOENT for orphan
+ * 		roots.
+ */
+static struct btrfs_root *__get_fs_root(struct btrfs_fs_info *fs_info,
+					u64 objectid, dev_t anon_dev,
+					bool check_ref)
 {
 	struct btrfs_root *root;
 	struct btrfs_path *path;
@@ -1572,6 +1595,8 @@ struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info,
 again:
 	root = btrfs_lookup_fs_root(fs_info, objectid);
 	if (root) {
+		/* Shouldn't get preallocated anon_dev for cached roots */
+		ASSERT(!anon_dev);
 		if (check_ref && btrfs_root_refs(&root->root_item) == 0) {
 			btrfs_put_root(root);
 			return ERR_PTR(-ENOENT);
@@ -1591,7 +1616,7 @@ struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info,
 		goto fail;
 	}
 
-	ret = btrfs_init_fs_root(root);
+	ret = btrfs_init_fs_root(root, anon_dev);
 	if (ret)
 		goto fail;
 
@@ -1624,6 +1649,18 @@ struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info,
 	return ERR_PTR(ret);
 }
 
+struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info,
+				     u64 objectid, bool check_ref)
+{
+	return __get_fs_root(fs_info, objectid, 0, check_ref);
+}
+
+struct btrfs_root *btrfs_get_new_fs_root(struct btrfs_fs_info *fs_info,
+					 u64 objectid, dev_t anon_dev)
+{
+	return __get_fs_root(fs_info, objectid, anon_dev, true);
+}
+
 static int btrfs_congested_fn(void *congested_data, int bdi_bits)
 {
 	struct btrfs_fs_info *info = (struct btrfs_fs_info *)congested_data;
fs/btrfs/disk-io.h +2 -0
index bf43245406c4..00dc39d47ed3 100644
@@ -67,6 +67,8 @@ void btrfs_free_fs_roots(struct btrfs_fs_info *fs_info);
 
 struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info,
 				     u64 objectid, bool check_ref);
+struct btrfs_root *btrfs_get_new_fs_root(struct btrfs_fs_info *fs_info,
+					 u64 objectid, dev_t anon_dev);
 
 void btrfs_free_fs_info(struct btrfs_fs_info *fs_info);
 int btrfs_cleanup_fs_roots(struct btrfs_fs_info *fs_info);
fs/btrfs/ioctl.c +20 -1
index 168deb8ef68a..644bfe53ff61 100644
@@ -566,6 +566,7 @@ static noinline int create_subvol(struct inode *dir,
 	struct inode *inode;
 	int ret;
 	int err;
+	dev_t anon_dev = 0;
 	u64 objectid;
 	u64 new_dirid = BTRFS_FIRST_FREE_OBJECTID;
 	u64 index = 0;
@@ -578,6 +579,10 @@ static noinline int create_subvol(struct inode *dir,
 	if (ret)
 		goto fail_free;
 
+	ret = get_anon_bdev(&anon_dev);
+	if (ret < 0)
+		goto fail_free;
+
 	/*
 	 * Don't create subvolume whose level is not zero. Or qgroup will be
 	 * screwed up since it assumes subvolume qgroup's level to be 0.
@@ -660,12 +665,15 @@ static noinline int create_subvol(struct inode *dir,
 		goto fail;
 
 	key.offset = (u64)-1;
-	new_root = btrfs_get_fs_root(fs_info, objectid, true);
+	new_root = btrfs_get_new_fs_root(fs_info, objectid, anon_dev);
 	if (IS_ERR(new_root)) {
+		free_anon_bdev(anon_dev);
 		ret = PTR_ERR(new_root);
 		btrfs_abort_transaction(trans, ret);
 		goto fail;
 	}
+	/* Freeing would be done in btrfs_put_root() of @new_root */
+	anon_dev = 0;
 
 	btrfs_record_root_in_trans(trans, new_root);
 
@@ -735,6 +743,8 @@ static noinline int create_subvol(struct inode *dir,
 	return ret;
 
 fail_free:
+	if (anon_dev)
+		free_anon_bdev(anon_dev);
 	kfree(root_item);
 	return ret;
 }
@@ -762,6 +772,9 @@ static int create_snapshot(struct btrfs_root *root, struct inode *dir,
 	if (!pending_snapshot)
 		return -ENOMEM;
 
+	ret = get_anon_bdev(&pending_snapshot->anon_dev);
+	if (ret < 0)
+		goto free_pending;
 	pending_snapshot->root_item = kzalloc(sizeof(struct btrfs_root_item),
 			GFP_KERNEL);
 	pending_snapshot->path = btrfs_alloc_path();
@@ -823,10 +836,16 @@ static int create_snapshot(struct btrfs_root *root, struct inode *dir,
 
 	d_instantiate(dentry, inode);
 	ret = 0;
+	pending_snapshot->anon_dev = 0;
 fail:
+	/* Prevent doulbe freeing of anon_dev */
+	if (ret && pending_snapshot->snap)
+		pending_snapshot->snap->anon_dev = 0;
 	btrfs_put_root(pending_snapshot->snap);
 	btrfs_subvolume_release_metadata(fs_info, &pending_snapshot->block_rsv);
 free_pending:
+	if (pending_snapshot->anon_dev)
+		free_anon_bdev(pending_snapshot->anon_dev);
 	kfree(pending_snapshot->root_item);
 	btrfs_free_path(pending_snapshot->path);
 	kfree(pending_snapshot);
fs/btrfs/transaction.c +2 -1
index b359d4b17658..48f3db64cd40 100644
@@ -1630,7 +1630,8 @@ static noinline int create_pending_snapshot(struct btrfs_trans_handle *trans,
 	}
 
 	key.offset = (u64)-1;
-	pending->snap = btrfs_get_fs_root(fs_info, objectid, true);
+	pending->snap = btrfs_get_new_fs_root(fs_info, objectid,
+					      pending->anon_dev);
 	if (IS_ERR(pending->snap)) {
 		ret = PTR_ERR(pending->snap);
 		btrfs_abort_transaction(trans, ret);
fs/btrfs/transaction.h +2 -0
index bf102e64bfb2..a122a712f5cc 100644
@@ -151,6 +151,8 @@ struct btrfs_pending_snapshot {
 	struct btrfs_block_rsv block_rsv;
 	/* extra metadata reservation for relocation */
 	int error;
+	/* Preallocated anonymous block device number */
+	dev_t anon_dev;
 	bool readonly;
 	struct list_head list;
 };
Signature · 1 line
2.27.0
#12 lore raw

Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation

Quoted 116 lines · Qu Wenruo

[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:

  ------------[ cut here ]------------
  BTRFS: Transaction aborted (error -24)
  WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs]
  RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs]
  Call Trace:
   create_pending_snapshots+0x82/0xa0 [btrfs]
   btrfs_commit_transaction+0x275/0x8c0 [btrfs]
   btrfs_mksubvol+0x4b9/0x500 [btrfs]
   btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs]
   btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs]
   btrfs_ioctl+0x11a4/0x2da0 [btrfs]
   do_vfs_ioctl+0xa9/0x640
   ksys_ioctl+0x67/0x90
   __x64_sys_ioctl+0x1a/0x20
   do_syscall_64+0x5a/0x110
   entry_SYSCALL_64_after_hwframe+0x44/0xa9
  ---[ end trace 33f2f83f3d5250e9 ]---
  BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown
  BTRFS info (device sda1): forced readonly
  BTRFS warning (device sda1): Skipping commit of aborted transaction.
  BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown

[CAUSE]
When the global anonymous block device pool is exhausted, the following call chain will fail, and lead to transaction abort:

btrfs_ioctl_snap_create_v2()
|- btrfs_ioctl_snap_create_transid()

    |- btrfs_mksubvol()
       |- btrfs_commit_transaction()
          |- create_pending_snapshot()
             |- btrfs_get_fs_root()
                |- btrfs_init_fs_root()
                   |- get_anon_bdev()

[FIX]
Although we can't enlarge the anonymous block device pool, at least we can preallocate anon_dev for subvolume/snapshot creation. So that when the pool is exhausted, user will get an error other than aborting transaction later.

---
 fs/btrfs/disk-io.c     | 51 ++++++++++++++++++++++++++++++++++++------
 fs/btrfs/disk-io.h     |  2 ++
 fs/btrfs/ioctl.c       | 21 ++++++++++++++++-
 fs/btrfs/transaction.c |  3 ++-
 fs/btrfs/transaction.h |  2 ++
 5 files changed, 70 insertions(+), 9 deletions(-)
fs/btrfs/disk-io.c +30 -6
index cfc0ff288238..14fd69b71cb8 100644
@@ -1395,7 +1395,7 @@ struct btrfs_root *btrfs_read_tree_root(struct btrfs_root *tree_root,
 	goto out;
 }
 
-static int btrfs_init_fs_root(struct btrfs_root *root)
+static int btrfs_init_fs_root(struct btrfs_root *root, dev_t anon_dev)
 {
 	int ret;
 	unsigned int nofs_flag;
@@ -1435,9 +1435,13 @@ static int btrfs_init_fs_root(struct btrfs_root *root)
 	 */
 	if (is_fstree(root->root_key.objectid) &&
 	    btrfs_root_refs(&root->root_item)) {
-		ret = get_anon_bdev(&root->anon_dev);
-		if (ret)
-			goto fail;
+		if (!anon_dev) {
+			ret = get_anon_bdev(&root->anon_dev);
+			if (ret)
+				goto fail;
+		} else {
+			root->anon_dev = anon_dev;
+		}
 	}
 
 	mutex_lock(&root->objectid_mutex);
@@ -1542,8 +1546,27 @@ void btrfs_free_fs_info(struct btrfs_fs_info *fs_info)
 }
 
 
-struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info,
-				     u64 objectid, bool check_ref)
+/*
+ * Get a fs root.
+ *
+ * For essential trees like root/extent tree, we grab it from fs_info directly.
+ * For subvolume trees, we check the cached fs roots first. If miss then
+ * read it from disk and add it to cached fs roots.
+ *
+ * Caller should release the root by calling btrfs_put_root() after the usage.
+ *
+ * NOTE: Reloc and log trees can't be read by this function as they share the
+ *	 same root objectid.
+ *
+ * @objectid:	Root (subvolume) id
+ * @anon_dev:	Preallocated anonymous block device number for new roots.
+ * 		Pass 0 for automatic allocation.
+ * @check_ref:	Whether to check root refs. If true, return -ENOENT for orphan
+ * 		roots.
+ */
+static struct btrfs_root *__get_fs_root(struct btrfs_fs_info *fs_info,
+					u64 objectid, dev_t anon_dev,
+					bool check_ref)
 
 
+struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info,

fs/btrfs/disk-io.c void btrfs_free_fs_info(struct btrfs_fs_info *fs_info)

Quoted 10 lines
fs/btrfs/disk-io.c +10 -0
+				     u64 objectid, bool check_ref)
+{
+	return __get_fs_root(fs_info, objectid, 0, check_ref);
+}
+
+struct btrfs_root *btrfs_get_new_fs_root(struct btrfs_fs_info *fs_info,
+					 u64 objectid, dev_t anon_dev)
+{
+	return __get_fs_root(fs_info, objectid, anon_dev, true);
+}

This does not look like a good API, we should keep btrfs_get_fs_root and add the anon_bdev initialization to the callers, there are only a few.

author
#13 lore raw

Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation

Quoted 116 lines · David Sterba
Quoted 113 lines · Qu Wenruo

[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:

  ------------[ cut here ]------------
  BTRFS: Transaction aborted (error -24)
  WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs]
  RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs]
  Call Trace:
   create_pending_snapshots+0x82/0xa0 [btrfs]
   btrfs_commit_transaction+0x275/0x8c0 [btrfs]
   btrfs_mksubvol+0x4b9/0x500 [btrfs]
   btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs]
   btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs]
   btrfs_ioctl+0x11a4/0x2da0 [btrfs]
   do_vfs_ioctl+0xa9/0x640
   ksys_ioctl+0x67/0x90
   __x64_sys_ioctl+0x1a/0x20
   do_syscall_64+0x5a/0x110
   entry_SYSCALL_64_after_hwframe+0x44/0xa9
  ---[ end trace 33f2f83f3d5250e9 ]---
  BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown
  BTRFS info (device sda1): forced readonly
  BTRFS warning (device sda1): Skipping commit of aborted transaction.
  BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown

[CAUSE]
When the global anonymous block device pool is exhausted, the following call chain will fail, and lead to transaction abort:

btrfs_ioctl_snap_create_v2()
|- btrfs_ioctl_snap_create_transid()

    |- btrfs_mksubvol()
       |- btrfs_commit_transaction()
          |- create_pending_snapshot()
             |- btrfs_get_fs_root()
                |- btrfs_init_fs_root()
                   |- get_anon_bdev()

[FIX]
Although we can't enlarge the anonymous block device pool, at least we can preallocate anon_dev for subvolume/snapshot creation. So that when the pool is exhausted, user will get an error other than aborting transaction later.

---
 fs/btrfs/disk-io.c     | 51 ++++++++++++++++++++++++++++++++++++------
 fs/btrfs/disk-io.h     |  2 ++
 fs/btrfs/ioctl.c       | 21 ++++++++++++++++-
 fs/btrfs/transaction.c |  3 ++-
 fs/btrfs/transaction.h |  2 ++
 5 files changed, 70 insertions(+), 9 deletions(-)
fs/btrfs/disk-io.c +29 -6
index cfc0ff288238..14fd69b71cb8 100644
@@ -1395,7 +1395,7 @@ struct btrfs_root *btrfs_read_tree_root(struct btrfs_root *tree_root,
 	goto out;
 }
 
-static int btrfs_init_fs_root(struct btrfs_root *root)
+static int btrfs_init_fs_root(struct btrfs_root *root, dev_t anon_dev)
 {
 	int ret;
 	unsigned int nofs_flag;
@@ -1435,9 +1435,13 @@ static int btrfs_init_fs_root(struct btrfs_root *root)
 	 */
 	if (is_fstree(root->root_key.objectid) &&
 	    btrfs_root_refs(&root->root_item)) {
-		ret = get_anon_bdev(&root->anon_dev);
-		if (ret)
-			goto fail;
+		if (!anon_dev) {
+			ret = get_anon_bdev(&root->anon_dev);
+			if (ret)
+				goto fail;
+		} else {
+			root->anon_dev = anon_dev;
+		}
 	}
 
 	mutex_lock(&root->objectid_mutex);
@@ -1542,8 +1546,27 @@ void btrfs_free_fs_info(struct btrfs_fs_info *fs_info)
 }
 
 
-struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info,
-				     u64 objectid, bool check_ref)
+/*
+ * Get a fs root.
+ *
+ * For essential trees like root/extent tree, we grab it from fs_info directly.
+ * For subvolume trees, we check the cached fs roots first. If miss then
+ * read it from disk and add it to cached fs roots.
+ *
+ * Caller should release the root by calling btrfs_put_root() after the usage.
+ *
+ * NOTE: Reloc and log trees can't be read by this function as they share the
+ *	 same root objectid.
+ *
+ * @objectid:	Root (subvolume) id
+ * @anon_dev:	Preallocated anonymous block device number for new roots.
+ * 		Pass 0 for automatic allocation.
+ * @check_ref:	Whether to check root refs. If true, return -ENOENT for orphan
+ * 		roots.
+ */
+static struct btrfs_root *__get_fs_root(struct btrfs_fs_info *fs_info,
+					u64 objectid, dev_t anon_dev,
+					bool check_ref)
Quoted 15 lines

fs/btrfs/disk-io.c struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info, · line 1662

Quoted 11 lines
fs/btrfs/disk-io.c +11 -0
+struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info,
+				     u64 objectid, bool check_ref)
+{
+	return __get_fs_root(fs_info, objectid, 0, check_ref);
+}
+
+struct btrfs_root *btrfs_get_new_fs_root(struct btrfs_fs_info *fs_info,
+					 u64 objectid, dev_t anon_dev)
+{
+	return __get_fs_root(fs_info, objectid, anon_dev, true);
+}

This does not look like a good API, we should keep btrfs_get_fs_root and add the anon_bdev initialization to the callers, there are only a few.

OK, I'd go that direction.

Thanks,
Qu

#14 lore raw

Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation

In reply to David Sterba
Quoted 116 lines · David Sterba
Quoted 113 lines · Qu Wenruo

[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:

  ------------[ cut here ]------------
  BTRFS: Transaction aborted (error -24)
  WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs]
  RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs]
  Call Trace:
   create_pending_snapshots+0x82/0xa0 [btrfs]
   btrfs_commit_transaction+0x275/0x8c0 [btrfs]
   btrfs_mksubvol+0x4b9/0x500 [btrfs]
   btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs]
   btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs]
   btrfs_ioctl+0x11a4/0x2da0 [btrfs]
   do_vfs_ioctl+0xa9/0x640
   ksys_ioctl+0x67/0x90
   __x64_sys_ioctl+0x1a/0x20
   do_syscall_64+0x5a/0x110
   entry_SYSCALL_64_after_hwframe+0x44/0xa9
  ---[ end trace 33f2f83f3d5250e9 ]---
  BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown
  BTRFS info (device sda1): forced readonly
  BTRFS warning (device sda1): Skipping commit of aborted transaction.
  BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown

[CAUSE]
When the global anonymous block device pool is exhausted, the following call chain will fail, and lead to transaction abort:

btrfs_ioctl_snap_create_v2()
|- btrfs_ioctl_snap_create_transid()

    |- btrfs_mksubvol()
       |- btrfs_commit_transaction()
          |- create_pending_snapshot()
             |- btrfs_get_fs_root()
                |- btrfs_init_fs_root()
                   |- get_anon_bdev()

[FIX]
Although we can't enlarge the anonymous block device pool, at least we can preallocate anon_dev for subvolume/snapshot creation. So that when the pool is exhausted, user will get an error other than aborting transaction later.

---
 fs/btrfs/disk-io.c     | 51 ++++++++++++++++++++++++++++++++++++------
 fs/btrfs/disk-io.h     |  2 ++
 fs/btrfs/ioctl.c       | 21 ++++++++++++++++-
 fs/btrfs/transaction.c |  3 ++-
 fs/btrfs/transaction.h |  2 ++
 5 files changed, 70 insertions(+), 9 deletions(-)
fs/btrfs/disk-io.c +29 -6
index cfc0ff288238..14fd69b71cb8 100644
@@ -1395,7 +1395,7 @@ struct btrfs_root *btrfs_read_tree_root(struct btrfs_root *tree_root,
 	goto out;
 }
 
-static int btrfs_init_fs_root(struct btrfs_root *root)
+static int btrfs_init_fs_root(struct btrfs_root *root, dev_t anon_dev)
 {
 	int ret;
 	unsigned int nofs_flag;
@@ -1435,9 +1435,13 @@ static int btrfs_init_fs_root(struct btrfs_root *root)
 	 */
 	if (is_fstree(root->root_key.objectid) &&
 	    btrfs_root_refs(&root->root_item)) {
-		ret = get_anon_bdev(&root->anon_dev);
-		if (ret)
-			goto fail;
+		if (!anon_dev) {
+			ret = get_anon_bdev(&root->anon_dev);
+			if (ret)
+				goto fail;
+		} else {
+			root->anon_dev = anon_dev;
+		}
 	}
 
 	mutex_lock(&root->objectid_mutex);
@@ -1542,8 +1546,27 @@ void btrfs_free_fs_info(struct btrfs_fs_info *fs_info)
 }
 
 
-struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info,
-				     u64 objectid, bool check_ref)
+/*
+ * Get a fs root.
+ *
+ * For essential trees like root/extent tree, we grab it from fs_info directly.
+ * For subvolume trees, we check the cached fs roots first. If miss then
+ * read it from disk and add it to cached fs roots.
+ *
+ * Caller should release the root by calling btrfs_put_root() after the usage.
+ *
+ * NOTE: Reloc and log trees can't be read by this function as they share the
+ *	 same root objectid.
+ *
+ * @objectid:	Root (subvolume) id
+ * @anon_dev:	Preallocated anonymous block device number for new roots.
+ * 		Pass 0 for automatic allocation.
+ * @check_ref:	Whether to check root refs. If true, return -ENOENT for orphan
+ * 		roots.
+ */
+static struct btrfs_root *__get_fs_root(struct btrfs_fs_info *fs_info,
+					u64 objectid, dev_t anon_dev,
+					bool check_ref)
Quoted 15 lines

fs/btrfs/disk-io.c struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info, · line 1662

Quoted 11 lines
fs/btrfs/disk-io.c +11 -0
+struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info,
+				     u64 objectid, bool check_ref)
+{
+	return __get_fs_root(fs_info, objectid, 0, check_ref);
+}
+
+struct btrfs_root *btrfs_get_new_fs_root(struct btrfs_fs_info *fs_info,
+					 u64 objectid, dev_t anon_dev)
+{
+	return __get_fs_root(fs_info, objectid, anon_dev, true);
+}

This does not look like a good API, we should keep btrfs_get_fs_root and add the anon_bdev initialization to the callers, there are only a few.

A few = over 25?

I have switched to keep btrfs_get_fs_root(), but you won't like the summary:

Old:

 fs/btrfs/disk-io.h     |  2 ++
 fs/btrfs/ioctl.c       | 21 ++++++++++++++++-
 fs/btrfs/transaction.c |  3 ++-
 fs/btrfs/transaction.h |  2 ++
 5 files changed, 70 insertions(+), 9 deletions(-)

New:

 fs/btrfs/backref.c     |  4 ++--
 fs/btrfs/disk-io.c     | 42 ++++++++++++++++++++++++++++++++++--------
 fs/btrfs/disk-io.h     |  3 ++-
 fs/btrfs/export.c      |  2 +-
 fs/btrfs/file.c        |  2 +-
 fs/btrfs/inode.c       |  2 +-
 fs/btrfs/ioctl.c       | 31 +++++++++++++++++++++++++------
 fs/btrfs/relocation.c  | 11 ++++++-----
 fs/btrfs/root-tree.c   |  2 +-
 fs/btrfs/scrub.c       |  2 +-
 fs/btrfs/send.c        |  4 ++--
 fs/btrfs/super.c       |  2 +-
 fs/btrfs/transaction.c |  3 ++-
 fs/btrfs/transaction.h |  2 ++
 fs/btrfs/tree-log.c    |  2 +-
 fs/btrfs/uuid-tree.c   |  2 +-
 16 files changed, 83 insertions(+), 33 deletions(-)

Do we really go that direction?

Thanks,
Qu

#15 lore raw

Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation

Quoted 40 lines · Qu Wenruo
Quoted 9 lines

fs/btrfs/disk-io.c struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info, · line 1662

Quoted 5 lines
fs/btrfs/disk-io.c +5 -0
+struct btrfs_root *btrfs_get_new_fs_root(struct btrfs_fs_info *fs_info,
+					 u64 objectid, dev_t anon_dev)
+{
+	return __get_fs_root(fs_info, objectid, anon_dev, true);
+}

This does not look like a good API, we should keep btrfs_get_fs_root and add the anon_bdev initialization to the callers, there are only a few.

A few = over 25?

I have switched to keep btrfs_get_fs_root(), but you won't like the summary:

Old:

 fs/btrfs/disk-io.h     |  2 ++
 fs/btrfs/ioctl.c       | 21 ++++++++++++++++-
 fs/btrfs/transaction.c |  3 ++-
 fs/btrfs/transaction.h |  2 ++
 5 files changed, 70 insertions(+), 9 deletions(-)

New:

 fs/btrfs/backref.c     |  4 ++--
 fs/btrfs/disk-io.c     | 42 ++++++++++++++++++++++++++++++++++--------
 fs/btrfs/disk-io.h     |  3 ++-
 fs/btrfs/export.c      |  2 +-
 fs/btrfs/file.c        |  2 +-
 fs/btrfs/inode.c       |  2 +-
 fs/btrfs/ioctl.c       | 31 +++++++++++++++++++++++++------
 fs/btrfs/relocation.c  | 11 ++++++-----
 fs/btrfs/root-tree.c   |  2 +-
 fs/btrfs/scrub.c       |  2 +-
 fs/btrfs/send.c        |  4 ++--
 fs/btrfs/super.c       |  2 +-
 fs/btrfs/transaction.c |  3 ++-
 fs/btrfs/transaction.h |  2 ++
 fs/btrfs/tree-log.c    |  2 +-
 fs/btrfs/uuid-tree.c   |  2 +-
 16 files changed, 83 insertions(+), 33 deletions(-)
Quoted 1 line

Do we really go that direction?

You're right, I don't like the summary and I don't like the code either.

Adding the anon_dev argument to btrfs_get_fs_root is wrong and I have never suggested that. What I meant is to put the actual id allocation to the callers where the subvolume is created, ie only 2 places.

#16 lore raw

Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation

Quoted 41 lines · David Sterba
Quoted 40 lines · Qu Wenruo
Quoted 9 lines

fs/btrfs/disk-io.c struct btrfs_root *btrfs_get_fs_root(struct btrfs_fs_info *fs_info, · line 1662

Quoted 5 lines
fs/btrfs/disk-io.c +5 -0
+struct btrfs_root *btrfs_get_new_fs_root(struct btrfs_fs_info *fs_info,
+					 u64 objectid, dev_t anon_dev)
+{
+	return __get_fs_root(fs_info, objectid, anon_dev, true);
+}

This does not look like a good API, we should keep btrfs_get_fs_root and add the anon_bdev initialization to the callers, there are only a few.

A few = over 25?

I have switched to keep btrfs_get_fs_root(), but you won't like the summary:

Old:

 fs/btrfs/disk-io.h     |  2 ++
 fs/btrfs/ioctl.c       | 21 ++++++++++++++++-
 fs/btrfs/transaction.c |  3 ++-
 fs/btrfs/transaction.h |  2 ++
 5 files changed, 70 insertions(+), 9 deletions(-)

New:

 fs/btrfs/backref.c     |  4 ++--
 fs/btrfs/disk-io.c     | 42 ++++++++++++++++++++++++++++++++++--------
 fs/btrfs/disk-io.h     |  3 ++-
 fs/btrfs/export.c      |  2 +-
 fs/btrfs/file.c        |  2 +-
 fs/btrfs/inode.c       |  2 +-
 fs/btrfs/ioctl.c       | 31 +++++++++++++++++++++++++------
 fs/btrfs/relocation.c  | 11 ++++++-----
 fs/btrfs/root-tree.c   |  2 +-
 fs/btrfs/scrub.c       |  2 +-
 fs/btrfs/send.c        |  4 ++--
 fs/btrfs/super.c       |  2 +-
 fs/btrfs/transaction.c |  3 ++-
 fs/btrfs/transaction.h |  2 ++
 fs/btrfs/tree-log.c    |  2 +-
 fs/btrfs/uuid-tree.c   |  2 +-
 16 files changed, 83 insertions(+), 33 deletions(-)
Quoted 8 lines
Quoted 1 line

Do we really go that direction?

You're right, I don't like the summary and I don't like the code either.

Adding the anon_dev argument to btrfs_get_fs_root is wrong and I have never suggested that. What I meant is to put the actual id allocation to the callers where the subvolume is created, ie only 2 places.

You mean to extract btrfs_init_fs_root() out of btrfs_get_fs_root()?

That looks a little risky and I can't find any good solution to make it more elegant than the current one.

Although I would definitely remove the "__" prefix as we shouldn't add such prefix anymore.

Thanks,
Qu

#17 lore raw

Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation

Quoted 10 lines · Qu Wenruo
Quoted 4 lines · David Sterba

On Wed, Jul 01, 2020 at 11:25:27AM +0800, Qu Wenruo wrote:
Adding the anon_dev argument to btrfs_get_fs_root is wrong and I have never suggested that. What I meant is to put the actual id allocation to the callers where the subvolume is created, ie only 2 places.

You mean to extract btrfs_init_fs_root() out of btrfs_get_fs_root()?

That looks a little risky and I can't find any good solution to make it more elegant than the current one.

I spent more time reading through the get-fs-root functions and the main problem is that btrfs_get_fs_root is doing several things, and it makes a lot of code simple, I certainly want to keep it that way.

The idea was to pre-insert the new root (similar to the root item insertion, btrfs_insert_root) and not letting btrfs_get_fs_root call to btrfs_init_fs_info where the anon_bdev allocation happens for all the other non-ioctl cases.

Which could be done by factoring out btrfs_init_fs_root from btrfs_get_fs_root. This would allow to extend only btrfs_init_fs_root arguments with the anon_bdev, and keep btrfs_get_fs_root intact. So this is splitting the API from the end.

What you originally proposed is a split from the begnning, ie. add a common implementation for existing and new and provide btrfs_get_fs_root and btrfs_get_new_fs_root that would hide the additional parameters.

Both ways are IMO valid but I thought it would be easier to pass the anon bdev inside ioctl callbacks. The problem that makes my proposal less appealing is that btrfs_read_tree_root gets called earlier than I'd like so factoring everything after btrfs_init_fs_root would not be so straightforward.

In conclusion, your proposal is better and I'm going to merge it.

Quoted 2 lines

Although I would definitely remove the "__" prefix as we shouldn't add such prefix anymore.

Yeah with the small naming fixups.

#18 lore raw

Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation

Quoted 36 lines · David Sterba
Quoted 10 lines · Qu Wenruo
Quoted 4 lines · David Sterba

On Wed, Jul 01, 2020 at 11:25:27AM +0800, Qu Wenruo wrote:
Adding the anon_dev argument to btrfs_get_fs_root is wrong and I have never suggested that. What I meant is to put the actual id allocation to the callers where the subvolume is created, ie only 2 places.

You mean to extract btrfs_init_fs_root() out of btrfs_get_fs_root()?

That looks a little risky and I can't find any good solution to make it more elegant than the current one.

I spent more time reading through the get-fs-root functions and the main problem is that btrfs_get_fs_root is doing several things, and it makes a lot of code simple, I certainly want to keep it that way.

The idea was to pre-insert the new root (similar to the root item insertion, btrfs_insert_root) and not letting btrfs_get_fs_root call to btrfs_init_fs_info where the anon_bdev allocation happens for all the other non-ioctl cases.

Which could be done by factoring out btrfs_init_fs_root from btrfs_get_fs_root. This would allow to extend only btrfs_init_fs_root arguments with the anon_bdev, and keep btrfs_get_fs_root intact. So this is splitting the API from the end.

What you originally proposed is a split from the begnning, ie. add a common implementation for existing and new and provide btrfs_get_fs_root and btrfs_get_new_fs_root that would hide the additional parameters.

Both ways are IMO valid but I thought it would be easier to pass the anon bdev inside ioctl callbacks. The problem that makes my proposal less appealing is that btrfs_read_tree_root gets called earlier than I'd like so factoring everything after btrfs_init_fs_root would not be so straightforward.

Quoted 6 lines

In conclusion, your proposal is better and I'm going to merge it.

Quoted 2 lines

Although I would definitely remove the "__" prefix as we shouldn't add such prefix anymore.

Yeah with the small naming fixups.

It's in for-next-20200703. I've updated the changelogs to reflect what we found during debugging the issue, the __ function renamed to btrfs_get_root_ref and some function comments added. All patches reordered and tagged for stable though the preallocation is not within the size limit.

#19 lore raw

Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation

Quoted 42 lines · David Sterba
Quoted 41 lines · David Sterba
Quoted 10 lines · Qu Wenruo
Quoted 4 lines · David Sterba

On Wed, Jul 01, 2020 at 11:25:27AM +0800, Qu Wenruo wrote:
Adding the anon_dev argument to btrfs_get_fs_root is wrong and I have never suggested that. What I meant is to put the actual id allocation to the callers where the subvolume is created, ie only 2 places.

You mean to extract btrfs_init_fs_root() out of btrfs_get_fs_root()?

That looks a little risky and I can't find any good solution to make it more elegant than the current one.

I spent more time reading through the get-fs-root functions and the main problem is that btrfs_get_fs_root is doing several things, and it makes a lot of code simple, I certainly want to keep it that way.

The idea was to pre-insert the new root (similar to the root item insertion, btrfs_insert_root) and not letting btrfs_get_fs_root call to btrfs_init_fs_info where the anon_bdev allocation happens for all the other non-ioctl cases.

Which could be done by factoring out btrfs_init_fs_root from btrfs_get_fs_root. This would allow to extend only btrfs_init_fs_root arguments with the anon_bdev, and keep btrfs_get_fs_root intact. So this is splitting the API from the end.

What you originally proposed is a split from the begnning, ie. add a common implementation for existing and new and provide btrfs_get_fs_root and btrfs_get_new_fs_root that would hide the additional parameters.

Both ways are IMO valid but I thought it would be easier to pass the anon bdev inside ioctl callbacks. The problem that makes my proposal less appealing is that btrfs_read_tree_root gets called earlier than I'd like so factoring everything after btrfs_init_fs_root would not be so straightforward.

In conclusion, your proposal is better and I'm going to merge it.

Quoted 2 lines

Although I would definitely remove the "__" prefix as we shouldn't add such prefix anymore.

Quoted 8 lines
Quoted 1 line

Yeah with the small naming fixups.

It's in for-next-20200703. I've updated the changelogs to reflect what we found during debugging the issue, the __ function renamed to btrfs_get_root_ref and some function comments added. All patches reordered and tagged for stable though the preallocation is not within the size limit.

Thanks for the merge and dropping the unneeded check patch.

All the modification looks good to me.

Just a small nitpick for commit a561defc34aa ("btrfs: don't allocate anonymous block device for user invisible roots"), there is an unnecessary new line after "[CAUSE]".

Thanks for your daily work of maintaining btrfs,
Qu

#20 lore raw

Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation

Quoted 14 lines · Qu Wenruo
Quoted 5 lines
Quoted 5 lines

In conclusion, your proposal is better and I'm going to merge it.

Quoted 2 lines

Although I would definitely remove the "__" prefix as we shouldn't add such prefix anymore.

Quoted 8 lines
Quoted 1 line

Yeah with the small naming fixups.

It's in for-next-20200703. I've updated the changelogs to reflect what we found during debugging the issue, the __ function renamed to btrfs_get_root_ref and some function comments added. All patches reordered and tagged for stable though the preallocation is not within the size limit.

Quoted 7 lines

Thanks for the merge and dropping the unneeded check patch.

All the modification looks good to me.

Just a small nitpick for commit a561defc34aa ("btrfs: don't allocate anonymous block device for user invisible roots"), there is an unnecessary new line after "[CAUSE]".

Fixed, thanks for checking. I'm not entirely satisfied with the name of btrfs_get_root_ref, as it could be confused with the on-disk btrfs_root_ref. The get-ref functions could use some cleanup as btrfs_get_fs_root is sometimes used for non-filesystem roots. Adding a generic get_any_root that would accept any tree and btrfs_get_fs_root would be only for subvolume roots or perhaps related trees like data reloc. But this is not essential for the anon bdev fixes so that's for later.

#21 lore raw

Re: [PATCH 3/4] btrfs: preallocate anon_dev for subvolume and snapshot creation

Quoted 19 lines · David Sterba
Quoted 18 lines · Qu Wenruo
Quoted 5 lines
Quoted 5 lines

In conclusion, your proposal is better and I'm going to merge it.

Quoted 2 lines

Although I would definitely remove the "__" prefix as we shouldn't add such prefix anymore.

Quoted 8 lines
Quoted 1 line

Yeah with the small naming fixups.

It's in for-next-20200703. I've updated the changelogs to reflect what we found during debugging the issue, the __ function renamed to btrfs_get_root_ref and some function comments added. All patches reordered and tagged for stable though the preallocation is not within the size limit.

Thanks for the merge and dropping the unneeded check patch.

All the modification looks good to me.

Quoted 7 lines
Quoted 3 lines

Just a small nitpick for commit a561defc34aa ("btrfs: don't allocate anonymous block device for user invisible roots"), there is an unnecessary new line after "[CAUSE]".

Fixed, thanks for checking. I'm not entirely satisfied with the name of btrfs_get_root_ref, as it could be confused with the on-disk btrfs_root_ref.

My original plan is even simpler, just get_fs_root(), remove the btrfs_ prefix...

Quoted 6 lines

The get-ref functions could use some cleanup as btrfs_get_fs_root is sometimes used for non-filesystem roots. Adding a generic get_any_root that would accept any tree and btrfs_get_fs_root would be only for subvolume roots or perhaps related trees like data reloc. But this is not essential for the anon bdev fixes so that's for later.

No problem.

Thanks,
Qu

author
#22 lore raw

PATCH 4/4 btrfs: free anon_dev earlier to prevent exhausting anonymous block device pool

In reply to Qu Wenruo

[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:

  ------------[ cut here ]------------
  BTRFS: Transaction aborted (error -24)
  WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs]
  RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs]
  Call Trace:
   create_pending_snapshots+0x82/0xa0 [btrfs]
   btrfs_commit_transaction+0x275/0x8c0 [btrfs]
   btrfs_mksubvol+0x4b9/0x500 [btrfs]
   btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs]
   btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs]
   btrfs_ioctl+0x11a4/0x2da0 [btrfs]
   do_vfs_ioctl+0xa9/0x640
   ksys_ioctl+0x67/0x90
   __x64_sys_ioctl+0x1a/0x20
   do_syscall_64+0x5a/0x110
   entry_SYSCALL_64_after_hwframe+0x44/0xa9
  ---[ end trace 33f2f83f3d5250e9 ]---
  BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown
  BTRFS info (device sda1): forced readonly
  BTRFS warning (device sda1): Skipping commit of aborted transaction.
  BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown

[CAUSE]
The root cause is we don't have unlimited resource for anonymous block device number.
The anonymous block device pool only contains 1<<20 devices, and is shared across a several fses, like ceph and overlayfs.

While btrfs has support for 1<<48 subvolumes, so it's just a problem of time to hit such limit.

[WORKAROUND]
Here we can free btrfs_root::anon_dev as long as the subvolume is no longer visible to users.

By freeing it earlier we reclaim the anon_dev quicker, hopefully to reduce the chance of exhausting the pool.

---
 fs/btrfs/inode.c | 2 ++
 1 file changed, 2 insertions(+)
fs/btrfs/inode.c +2 -0
index 195aac71fe32..e9f5a166d056 100644
@@ -4045,6 +4045,8 @@ int btrfs_delete_subvolume(struct inode *dir, struct dentry *dentry)
 		}
 	}
 
+	free_anon_bdev(dest->anon_dev);
+	dest->anon_dev = 0;
 out_end_trans:
 	trans->block_rsv = NULL;
 	trans->bytes_reserved = 0;
Signature · 1 line
2.27.0
#23 lore raw

Re: [PATCH 4/4] btrfs: free anon_dev earlier to prevent exhausting anonymous block device pool

Quoted 35 lines · Qu Wenruo

[BUG]
When a lot of subvolumes are created, there is a user report about transaction aborted:

   ------------[ cut here ]------------
   BTRFS: Transaction aborted (error -24)
   WARNING: CPU: 17 PID: 17041 at fs/btrfs/transaction.c:1576 create_pending_snapshot+0xbc4/0xd10 [btrfs]
   RIP: 0010:create_pending_snapshot+0xbc4/0xd10 [btrfs]
   Call Trace:
    create_pending_snapshots+0x82/0xa0 [btrfs]
    btrfs_commit_transaction+0x275/0x8c0 [btrfs]
    btrfs_mksubvol+0x4b9/0x500 [btrfs]
    btrfs_ioctl_snap_create_transid+0x174/0x180 [btrfs]
    btrfs_ioctl_snap_create_v2+0x11c/0x180 [btrfs]
    btrfs_ioctl+0x11a4/0x2da0 [btrfs]
    do_vfs_ioctl+0xa9/0x640
    ksys_ioctl+0x67/0x90
    __x64_sys_ioctl+0x1a/0x20
    do_syscall_64+0x5a/0x110
    entry_SYSCALL_64_after_hwframe+0x44/0xa9
   ---[ end trace 33f2f83f3d5250e9 ]---
   BTRFS: error (device sda1) in create_pending_snapshot:1576: errno=-24 unknown
   BTRFS info (device sda1): forced readonly
   BTRFS warning (device sda1): Skipping commit of aborted transaction.
   BTRFS: error (device sda1) in cleanup_transaction:1831: errno=-24 unknown

[CAUSE]
The root cause is we don't have unlimited resource for anonymous block device number.
The anonymous block device pool only contains 1<<20 devices, and is shared across a several fses, like ceph and overlayfs.

While btrfs has support for 1<<48 subvolumes, so it's just a problem of time to hit such limit.

Quoted 7 lines

[WORKAROUND]
Here we can free btrfs_root::anon_dev as long as the subvolume is no longer visible to users.

By freeing it earlier we reclaim the anon_dev quicker, hopefully to reduce the chance of exhausting the pool.

Why isn't this happening as part of the root teardown once all the references to it are gone? Thanks,

Josef

#24 lore raw

Re: [PATCH 4/4] btrfs: free anon_dev earlier to prevent exhausting anonymous block device pool

Quoted 5 lines · Josef Bacik
Quoted 2 lines

By freeing it earlier we reclaim the anon_dev quicker, hopefully to reduce the chance of exhausting the pool.

Why isn't this happening as part of the root teardown once all the references to it are gone? Thanks,

This is where it happens now and is correct. The problem is that deleted subvolumes keep the id allocated until they are cleaned up, ie. all the dead roots consume the id though we don't need it anymore. Creating and deleting snapshots at large will produce a long list of dead subvolumes. THis patch will return the ids at the earlies possible moment so they can get reused.

Reviewed-by
#25 lore raw

Re: [PATCH 4/4] btrfs: free anon_dev earlier to prevent exhausting anonymous block device pool

Quoted 7 lines · David Sterba
Quoted 5 lines · Josef Bacik
Quoted 2 lines

By freeing it earlier we reclaim the anon_dev quicker, hopefully to reduce the chance of exhausting the pool.

Why isn't this happening as part of the root teardown once all the references to it are gone? Thanks,

Quoted 7 lines

This is where it happens now and is correct. The problem is that deleted subvolumes keep the id allocated until they are cleaned up, ie. all the dead roots consume the id though we don't need it anymore. Creating and deleting snapshots at large will produce a long list of dead subvolumes. THis patch will return the ids at the earlies possible moment so they can get reused.

Oh ok I misread, we're doing it earlier on purpose. Alright that's fine, you can add

Thanks,

Josef

#26 lore raw
In reply to Qu Wenruo
Quoted 8 lines · Qu Wenruo

Qu Wenruo (4):
btrfs: disk-io: don't allocate anonymous block device for user
invisible roots
btrfs: detect uninitialized btrfs_root::anon_dev for user visible
subvolumes
btrfs: preallocate anon_dev for subvolume and snapshot creation
btrfs: free anon_dev earlier to prevent exhausting anonymous block
device pool

I haven't seen an update to this patchset and would really like to get it fixed. Let me know if I missed an update or you need help, there weren't any big issues left.