Skip to contentSkip to conversation

btrfs: qgroup: Fix the wrong target io_tree when freeing reserved data space

PATCH Superseded Qu Wenruo 3 participants 5 messages Sep 13, 2019 Btrfs file system

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

PATCH 1/2 btrfs: qgroup: Fix the wrong target io_tree when freeing reserved data space

[BUG]
Under the follow case with qgroup enabled, if some error happened after we have reserved delalloc space, then in error handling path, we could cause qgroup data space leakage:

From btrfs_truncate_block() in inode.c:

	ret = btrfs_delalloc_reserve_space(inode, &data_reserved,
					   block_start, blocksize);
	if (ret)
		goto out;

again:

	page = find_or_create_page(mapping, index, mask);
	if (!page) {
		btrfs_delalloc_release_space(inode, data_reserved,
					     block_start, blocksize, true);
		btrfs_delalloc_release_extents(BTRFS_I(inode), blocksize, true);
		ret = -ENOMEM;
		goto out;
	}

[CAUSE]
In above case, btrfs_delalloc_reserve_space() will call btrfs_qgroup_reserve_data() and mark the io_tree range with EXTENT_QGROUP_RESERVED flag.

In the error handling path, btrfs_delalloc_release_space() calls btrfs_qgroup_free_data() which should clear EXTENT_QGROUP_RESERVED flag and reduce the reserved data space accroding to the cleared range.

However due to a completion bug, btrfs_qgroup_free_data() will clear EXTENT_QGROUP_RESERVED flag in BTRFS_I(inode)->io_failure_tree, other than the correct BTRFS_I(inode)->io_tree.
Since io_failure_tree is never marked with that flag, btrfs_qgroup_free_data() will not free any data reserved space at all, causing a leakage.

All of such error handling cases can only be triggered some errors not from qgroup, so regular EDQUOT error won't trigger the bug. Normally we need error injection to trigger such bug.

[FIX]
Fix the wrong target io_tree.

Fixes bc42bda22345 ("btrfs: qgroup: Fix qgroup reserved space underflow by only freeing reserved ranges")
---
 fs/btrfs/qgroup.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/qgroup.c +1 -1
index 2891b57b9e1e..64bdc3e3652d 100644
@@ -3492,7 +3492,7 @@ static int qgroup_free_reserved_data(struct inode *inode,
 		 * EXTENT_QGROUP_RESERVED, we won't double free.
 		 * So not need to rush.
 		 */
-		ret = clear_record_extent_bits(&BTRFS_I(inode)->io_failure_tree,
+		ret = clear_record_extent_bits(&BTRFS_I(inode)->io_tree,
 				free_start, free_start + free_len - 1,
 				EXTENT_QGROUP_RESERVED, &changeset);
 		if (ret < 0)
Signature · 1 line
2.23.0
author
#1 lore raw

PATCH 2/2 btrfs: qgroup: Fix reserved data space leak if we have multiple reserve calls

[BUG]
The following script can cause btrfs qgroup data space leak:

mkfs.btrfs -f $dev
mount $dev -o nospace_cache $mnt

btrfs subv create $mnt/subv
btrfs quota en $mnt
btrfs quota rescan -w $mnt
btrfs qgroup limit 128m $mnt/subv

for (( i = 0; i < 3; i++)); do
# Create 3 64M holes for latter fallocate to fail truncate -s 192m $mnt/subv/file
xfs_io -c "pwrite 64m 4k" $mnt/subv/file > /dev/null xfs_io -c "pwrite 128m 4k" $mnt/subv/file > /dev/null sync

# it's supposed to fail, and each failure will leak at least 64M # data space
xfs_io -f -c "falloc 0 192m" $mnt/subv/file &> /dev/null rm $mnt/subv/file
sync
done

# Shouldn't fail after we removed the file
xfs_io -f -c "falloc 0 64m" $mnt/subv/file

[CAUSE]
Btrfs qgroup data reserve code allows multiple reserve happen on a single extent_changeset:

The only usage is in btrfs_fallocate():

	struct extent_changeset *data_reserved = NULL;
	btrfs_qgroup_reserve_data(inode, &data_reserved,
				  range_start, range_len);
	...
	btrfs_qgroup_reserve_data(inode, &data_reserved,
				  new_range_start, new_range_len);
	extent_changeset_free(data_reserved);

However in btrfs_qgroup_reserve_data(), if one of the call failed, it will cleanup all reserved space.
The cleanup itself is OK, but it only cleans up all EXTENT_QGROUP_RESERVED flag, forget to release the reserved bytes.

So if multiple btrfs_qgroup_reserve_data() get called, and the last one failed, then previously reserved data space will get leaked.

And due to the fact that EXTENT_QGROUP_RESERVED flag is cleaned correctly, btrfs_qgroup_check_reserved_leak() won't catch the leakage.

[FIX]
Also free previously reserved data bytes when btrfs_qgroup_reserve_data fails.

Fixes 524725537023 ("btrfs: qgroup: Introduce btrfs_qgroup_reserve_data function")
---
 fs/btrfs/qgroup.c | 3 +++
 1 file changed, 3 insertions(+)
fs/btrfs/qgroup.c +3 -0
index 64bdc3e3652d..59f6a9981087 100644
@@ -3448,6 +3448,9 @@ int btrfs_qgroup_reserve_data(struct inode *inode,
 	while ((unode = ulist_next(&reserved->range_changed, &uiter)))
 		clear_extent_bit(&BTRFS_I(inode)->io_tree, unode->val,
 				 unode->aux, EXTENT_QGROUP_RESERVED, 0, 0, NULL);
+	/* Also free data bytes of already reserved one */
+	btrfs_qgroup_free_refroot(root->fs_info, root->root_key.objectid,
+				  orig_reserved, BTRFS_QGROUP_RSV_DATA);
 	extent_changeset_release(reserved);
 	return ret;
 }
Signature · 1 line
2.23.0
#2 lore raw

Re: [PATCH 2/2] btrfs: qgroup: Fix reserved data space leak if we have multiple reserve calls

Quoted 25 lines · Qu Wenruo

[BUG]
The following script can cause btrfs qgroup data space leak:

mkfs.btrfs -f $dev
mount $dev -o nospace_cache $mnt

btrfs subv create $mnt/subv
btrfs quota en $mnt
btrfs quota rescan -w $mnt
btrfs qgroup limit 128m $mnt/subv

for (( i = 0; i < 3; i++)); do
# Create 3 64M holes for latter fallocate to fail truncate -s 192m $mnt/subv/file
xfs_io -c "pwrite 64m 4k" $mnt/subv/file > /dev/null xfs_io -c "pwrite 128m 4k" $mnt/subv/file > /dev/null sync

# it's supposed to fail, and each failure will leak at least 64M # data space
xfs_io -f -c "falloc 0 192m" $mnt/subv/file &> /dev/null rm $mnt/subv/file
sync
done

Quoted 5 lines

# Shouldn't fail after we removed the file
xfs_io -f -c "falloc 0 64m" $mnt/subv/file

[CAUSE]
Btrfs qgroup data reserve code allows multiple reserve happen on a

^
reservations to happen

Quoted 10 lines

single extent_changeset:

The only usage is in btrfs_fallocate():

	struct extent_changeset *data_reserved = NULL;
	btrfs_qgroup_reserve_data(inode, &data_reserved,
				  range_start, range_len);
	...
	btrfs_qgroup_reserve_data(inode, &data_reserved,
				  new_range_start, new_range_len);
	extent_changeset_free(data_reserved);

I take it you refer to the while() loop in btrfs_fallocate. The code above is really just a _VERY_ condensed version. extent_changeset_free is at the end of the function. Instead of putting random lines of code just explicitly state it, something along the lines of:

"The only such pattern is in btrfs_fallocate in the main while loop in that function".

Quoted 10 lines

However in btrfs_qgroup_reserve_data(), if one of the call failed, it > will cleanup all reserved space.
The cleanup itself is OK, but it only cleans up all EXTENT_QGROUP_RESERVED flag, forget to release the reserved bytes.

So if multiple btrfs_qgroup_reserve_data() get called, and the last one failed, then previously reserved data space will get leaked.

And due to the fact that EXTENT_QGROUP_RESERVED flag is cleaned correctly, btrfs_qgroup_check_reserved_leak() won't catch the leakage.

How about rephraing the above 3 paragraphs along the lines of:

"btrfs_qgroup_reserve_data's error handling has a bug in that on error it clears all ranges in the io_tree with EXTENT_QGROUP_RESERVED flag and doesn't free the reserved bytes. This behavior has a two fold effect:

1. Clearing EXTENT_QGROUP_RESERVED ranges prevents
btrfs_qgroup_check_reserved_leak to catch the leakage.
2. Leak the previously reserved data bytes.

The bug manifests when N calls to btrfs_qgroup_reserve_data are made and the last one fails, leaking space allocated in the previous ones. "

Quoted 26 lines

[FIX]
Also free previously reserved data bytes when btrfs_qgroup_reserve_data fails.

Fixes 524725537023 ("btrfs: qgroup: Introduce btrfs_qgroup_reserve_data function")
---
 fs/btrfs/qgroup.c | 3 +++
 1 file changed, 3 insertions(+)
fs/btrfs/qgroup.c +3 -0
index 64bdc3e3652d..59f6a9981087 100644
@@ -3448,6 +3448,9 @@ int btrfs_qgroup_reserve_data(struct inode *inode,
 	while ((unode = ulist_next(&reserved->range_changed, &uiter)))
 		clear_extent_bit(&BTRFS_I(inode)->io_tree, unode->val,
 				 unode->aux, EXTENT_QGROUP_RESERVED, 0, 0, NULL);
+	/* Also free data bytes of already reserved one */
+	btrfs_qgroup_free_refroot(root->fs_info, root->root_key.objectid,
+				  orig_reserved, BTRFS_QGROUP_RSV_DATA);
 	extent_changeset_release(reserved);
 	return ret;
 }
#3 lore raw
In reply to Qu Wenruo
Quoted 27 lines · Qu Wenruo

[BUG]
Under the follow case with qgroup enabled, if some error happened after we have reserved delalloc space, then in error handling path, we could cause qgroup data space leakage:

From btrfs_truncate_block() in inode.c:

	ret = btrfs_delalloc_reserve_space(inode, &data_reserved,
					   block_start, blocksize);
	if (ret)
		goto out;

again:

	page = find_or_create_page(mapping, index, mask);
	if (!page) {
		btrfs_delalloc_release_space(inode, data_reserved,
					     block_start, blocksize, true);
		btrfs_delalloc_release_extents(BTRFS_I(inode), blocksize, true);
		ret = -ENOMEM;
		goto out;
	}

[CAUSE]
In above case, btrfs_delalloc_reserve_space() will call btrfs_qgroup_reserve_data() and mark the io_tree range with EXTENT_QGROUP_RESERVED flag.

Quoted 7 lines

In the error handling path, btrfs_delalloc_release_space() calls btrfs_qgroup_free_data() which should clear EXTENT_QGROUP_RESERVED flag and reduce the reserved data space accroding to the cleared range.

However due to a completion bug, btrfs_qgroup_free_data() will clear EXTENT_QGROUP_RESERVED flag in BTRFS_I(inode)->io_failure_tree, other than the correct BTRFS_I(inode)->io_tree.

This is a bit confusing because the error is actually in qgroup_free_reserved_data, which is called from __btrfs_qgroup_release_data. But in the latter function there is also a call to clear_record_extent_bits with the correct tree. Just fix the function name by using qgroup_free_reserved_data.

Quoted 5 lines

Since io_failure_tree is never marked with that flag, btrfs_qgroup_free_data() will not free any data reserved space at all, causing a leakage.

All of such error handling cases can only be triggered some errors not

I take it you meant:

This error handling can only be triggered by errors outside of qgroup e.g. EDQUOT can't triger the bug?

The first part of the sentence is hard to parse.

Quoted 27 lines

from qgroup, so regular EDQUOT error won't trigger the bug. Normally we need error injection to trigger such bug.

[FIX]
Fix the wrong target io_tree.

Fixes bc42bda22345 ("btrfs: qgroup: Fix qgroup reserved space underflow by only freeing reserved ranges")
---
 fs/btrfs/qgroup.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/qgroup.c +1 -1
index 2891b57b9e1e..64bdc3e3652d 100644
@@ -3492,7 +3492,7 @@ static int qgroup_free_reserved_data(struct inode *inode,
 		 * EXTENT_QGROUP_RESERVED, we won't double free.
 		 * So not need to rush.
 		 */
-		ret = clear_record_extent_bits(&BTRFS_I(inode)->io_failure_tree,
+		ret = clear_record_extent_bits(&BTRFS_I(inode)->io_tree,
 				free_start, free_start + free_len - 1,
 				EXTENT_QGROUP_RESERVED, &changeset);
 		if (ret < 0)
#4 lore raw
Quoted 38 lines · Nikolay Borisov
Quoted 27 lines · Qu Wenruo

[BUG]
Under the follow case with qgroup enabled, if some error happened after we have reserved delalloc space, then in error handling path, we could cause qgroup data space leakage:

From btrfs_truncate_block() in inode.c:

	ret = btrfs_delalloc_reserve_space(inode, &data_reserved,
					   block_start, blocksize);
	if (ret)
		goto out;

again:

	page = find_or_create_page(mapping, index, mask);
	if (!page) {
		btrfs_delalloc_release_space(inode, data_reserved,
					     block_start, blocksize, true);
		btrfs_delalloc_release_extents(BTRFS_I(inode), blocksize, true);
		ret = -ENOMEM;
		goto out;
	}

[CAUSE]
In above case, btrfs_delalloc_reserve_space() will call btrfs_qgroup_reserve_data() and mark the io_tree range with EXTENT_QGROUP_RESERVED flag.

Quoted 7 lines

In the error handling path, btrfs_delalloc_release_space() calls btrfs_qgroup_free_data() which should clear EXTENT_QGROUP_RESERVED flag and reduce the reserved data space accroding to the cleared range.

However due to a completion bug, btrfs_qgroup_free_data() will clear EXTENT_QGROUP_RESERVED flag in BTRFS_I(inode)->io_failure_tree, other than the correct BTRFS_I(inode)->io_tree.

Quoted 5 lines

This is a bit confusing because the error is actually in qgroup_free_reserved_data, which is called from __btrfs_qgroup_release_data. But in the latter function there is also a call to clear_record_extent_bits with the correct tree. Just fix the function name by using qgroup_free_reserved_data.

Right, I ignored some caller here, as the caller chain is not only dependent on btrfs_qgroup_free_data() but also on the parameter. E.g. only when reserved is non-null we go qgroup_free_reserved_data().

Quoted 11 lines
Quoted 6 lines

Since io_failure_tree is never marked with that flag, btrfs_qgroup_free_data() will not free any data reserved space at all, causing a leakage.

All of such error handling cases can only be triggered some errors not

I take it you meant:

This error handling can only be triggered by errors outside of qgroup e.g. EDQUOT can't triger the bug?

Right.

I'll change it too something like "such leakage can only be triggered by errors outside of qgroup."

Thanks,
Qu

Quoted 30 lines

The first part of the sentence is hard to parse.

Quoted 27 lines

from qgroup, so regular EDQUOT error won't trigger the bug. Normally we need error injection to trigger such bug.

[FIX]
Fix the wrong target io_tree.

Fixes bc42bda22345 ("btrfs: qgroup: Fix qgroup reserved space underflow by only freeing reserved ranges")
---
 fs/btrfs/qgroup.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/qgroup.c +1 -1
index 2891b57b9e1e..64bdc3e3652d 100644
@@ -3492,7 +3492,7 @@ static int qgroup_free_reserved_data(struct inode *inode,
 		 * EXTENT_QGROUP_RESERVED, we won't double free.
 		 * So not need to rush.
 		 */
-		ret = clear_record_extent_bits(&BTRFS_I(inode)->io_failure_tree,
+		ret = clear_record_extent_bits(&BTRFS_I(inode)->io_tree,
 				free_start, free_start + free_len - 1,
 				EXTENT_QGROUP_RESERVED, &changeset);
 		if (ret < 0)