Skip to contentSkip to conversation

btrfs: qgroup: Fix the long existing regression of btrfs/153

PATCH Superseded Qu Wenruo 4 participants 13 messages Jul 8, 2020 – Jul 13, 2020 Btrfs file system

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

v3 cover btrfs: qgroup: Fix the long existing regression of btrfs/153

Since commit c6887cd11149 ("Btrfs: don't do nocow check unless we have to"), btrfs/153 always fails with early EDQUOT.

This is caused by the fact that:
- We always reserved data space for even NODATACOW buffered write This is mostly to improve performance, and not pratical to revert.

- Btrfs qgroup data and meta reserved space share the same limit So it's not ensured to return EDQUOT just for that data reservation, metadata reservation can also get EDQUOT, means we can't go the same solution as that commit.

This patchset will solve it by doing extra qgroup space flushing when EDQUOT is hit.

This is a little like what we do in ticketing space reservation system, but since there are very limited ways for qgroup to reclaim space, currently it's still handled in qgroup realm, not reusing the ticketing system yet.

By this, this patch could solve the btrfs/153 problem, while still keep btrfs qgroup space usage under the limit.

The only cost is, when we're near qgroup limit, we will cause more dirty inodes flush and transaction commit, much like what we do when the metadata space is near exhausted.
So the cost should be still acceptable.

Changelog:
v2:
- Use existing ulist infrastructure
Thanks to the advice from Josef, we can just iterate the ulist nodes inside the failure range to remove the offending nodes. And since already dirtied range won't show up in the current extent changeset, we won't clear existing range.
This saves around 100 new lines

- Remove the use of "Revert" in the 3rd cleanup patch We're not reverting to fix some regression, but just remove some deprecated mechanism which was designed to partially solve the problem.

v3:
- Fix a variable shadowing bug
In the 2nd patch, there was a variable shadowing bug, which overrides the @start and @len parameter.

- Update the commit message for the first patch
To make the cause more obvious

- Use cleaner if condition before try_flush_qgroup()

Qu Wenruo (3):
btrfs: qgroup: allow btrfs_qgroup_reserve_data() to revert the range
it just set without releasing other existing ranges
btrfs: qgroup: try to flush qgroup space when we get -EDQUOT
btrfs: qgroup: remove the ASYNC_COMMIT mechanism in favor of qgroup
reserve retry-after-EDQUOT

 fs/btrfs/ctree.h       |   6 +-
 fs/btrfs/disk-io.c     |   2 +-
 fs/btrfs/qgroup.c      | 234 ++++++++++++++++++++++++++++++-----------
 fs/btrfs/transaction.c |   1 -
 fs/btrfs/transaction.h |  14 ---
 5 files changed, 172 insertions(+), 85 deletions(-)
Signature · 1 line
2.27.0
author
#1 lore raw

PATCH v3 1/3 btrfs: qgroup: allow btrfs_qgroup_reserve_data() to revert the range it just set without releasing other existing ranges

[PROBLEM]
Before this patch, when btrfs_qgroup_reserve_data() fails, we free all reserved space of the changeset.

For example:

	ret = btrfs_qgroup_reserve_data(inode, changeset, 0, SZ_1M);
	ret = btrfs_qgroup_reserve_data(inode, changeset, SZ_1M, SZ_1M);
	ret = btrfs_qgroup_reserve_data(inode, changeset, SZ_2M, SZ_1M);

If the last btrfs_qgroup_reserve_data() failed, it will release all [0, 3M) range.

This behavior is kinda OK for now, as when we hit -EDQUOT, we normally go error handling and need to release all reserved ranges anyway.

But this also means the following call is not possible:

	ret = btrfs_qgroup_reserve_data();
	if (ret == -EDQUOT) {
		/* Do something to free some qgroup space */
		ret = btrfs_qgroup_reserve_data();
	}

As if the first btrfs_qgroup_reserve_data() fails, it will free all reserved qgroup space.

[CAUSE]
This is because we release all reserved ranges when btrfs_qgroup_reserve_data() fails.

[FIX]
This patch will implement a new function, qgroup_revert(), to iterate through the ulist nodes, to find any nodes in the failure range, and remove the EXTENT_QGROUP_RESERVED bits from the io_tree, and decrease the extent_changeset::bytes_changed, so that we can revert to previous status.

This allows later patches to retry btrfs_qgroup_reserve_data() if EDQUOT happens.

---
 fs/btrfs/qgroup.c | 90 +++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 75 insertions(+), 15 deletions(-)
fs/btrfs/qgroup.c +75 -15
index ee0ad33b659c..84a452dea3f9 100644
@@ -3447,6 +3447,71 @@ btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info)
 	}
 }
 
+static int qgroup_revert(struct btrfs_inode *inode,
+			struct extent_changeset *reserved, u64 start,
+			u64 len)
+{
+	struct rb_node *n = reserved->range_changed.root.rb_node;
+	struct ulist_node *entry = NULL;
+	int ret = 0;
+
+	while (n) {
+		entry = rb_entry(n, struct ulist_node, rb_node);
+		if (entry->val < start)
+			n = n->rb_right;
+		else if (entry)
+			n = n->rb_left;
+		else
+			break;
+	}
+	/* Empty changeset */
+	if (!entry)
+		goto out;
+
+	if (entry->val > start && rb_prev(&entry->rb_node))
+		entry = rb_entry(rb_prev(&entry->rb_node), struct ulist_node,
+				 rb_node);
+
+	n = &entry->rb_node;
+	while (n) {
+		struct rb_node *tmp = rb_next(n);
+		u64 entry_start;
+		u64 entry_end;
+		u64 entry_len;
+		int clear_ret;
+
+		entry = rb_entry(n, struct ulist_node, rb_node);
+		entry_start = entry->val;
+		entry_end = entry->aux;
+		entry_len = entry_end - entry_start + 1;
+
+		if (entry_start >= start + len)
+			break;
+		if (entry_start + entry_len <= start)
+			goto next;
+		/*
+		 * Now the entry is in [start, start + len), revert the
+		 * EXTENT_QGROUP_RESERVED bit.
+		 */
+		clear_ret = clear_extent_bits(&inode->io_tree, entry_start,
+					      entry_end, EXTENT_QGROUP_RESERVED);
+		if (!ret && clear_ret < 0)
+			ret = clear_ret;
+
+		ulist_del(&reserved->range_changed, entry->val, entry->aux);
+		if (likely(reserved->bytes_changed >= entry_len)) {
+			reserved->bytes_changed -= entry_len;
+		} else {
+			WARN_ON(1);
+			reserved->bytes_changed = 0;
+		}
+next:
+		n = tmp;
+	}
+out:
+	return ret;
+}
+
 /*
  * Reserve qgroup space for range [start, start + len).
  *
@@ -3457,18 +3522,14 @@ btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info)
  * Return <0 for error (including -EQUOT)
  *
  * NOTE: this function may sleep for memory allocation.
- *       if btrfs_qgroup_reserve_data() is called multiple times with
- *       same @reserved, caller must ensure when error happens it's OK
- *       to free *ALL* reserved space.
  */
 int btrfs_qgroup_reserve_data(struct btrfs_inode *inode,
 			struct extent_changeset **reserved_ret, u64 start,
 			u64 len)
 {
 	struct btrfs_root *root = inode->root;
-	struct ulist_node *unode;
-	struct ulist_iterator uiter;
 	struct extent_changeset *reserved;
+	bool new_reserved = false;
 	u64 orig_reserved;
 	u64 to_reserve;
 	int ret;
@@ -3481,6 +3542,7 @@ int btrfs_qgroup_reserve_data(struct btrfs_inode *inode,
 	if (WARN_ON(!reserved_ret))
 		return -EINVAL;
 	if (!*reserved_ret) {
+		new_reserved = true;
 		*reserved_ret = extent_changeset_alloc();
 		if (!*reserved_ret)
 			return -ENOMEM;
@@ -3496,7 +3558,7 @@ int btrfs_qgroup_reserve_data(struct btrfs_inode *inode,
 	trace_btrfs_qgroup_reserve_data(&inode->vfs_inode, start, len,
 					to_reserve, QGROUP_RESERVE);
 	if (ret < 0)
-		goto cleanup;
+		goto out;
 	ret = qgroup_reserve(root, to_reserve, true, BTRFS_QGROUP_RSV_DATA);
 	if (ret < 0)
 		goto cleanup;
@@ -3504,15 +3566,13 @@ int btrfs_qgroup_reserve_data(struct btrfs_inode *inode,
 	return ret;
 
 cleanup:
-	/* cleanup *ALL* already reserved ranges */
-	ULIST_ITER_INIT(&uiter);
-	while ((unode = ulist_next(&reserved->range_changed, &uiter)))
-		clear_extent_bit(&inode->io_tree, unode->val,
-				 unode->aux, EXTENT_QGROUP_RESERVED, 0, 0, NULL);
-	/* Also free data bytes of already reserved one */
-	btrfs_qgroup_free_refroot(root->fs_info, root->root_key.objectid,
-				  orig_reserved, BTRFS_QGROUP_RSV_DATA);
-	extent_changeset_release(reserved);
+	qgroup_revert(inode, reserved, start, len);
+out:
+	if (new_reserved) {
+		extent_changeset_release(reserved);
+		kfree(reserved);
+		*reserved_ret = NULL;
+	}
 	return ret;
 }
 
Signature · 1 line
2.27.0
#2 lore raw

Re: [PATCH v3 1/3] btrfs: qgroup: allow btrfs_qgroup_reserve_data() to revert the range it just set without releasing other existing ranges

Quoted 63 lines · Qu Wenruo

[PROBLEM]
Before this patch, when btrfs_qgroup_reserve_data() fails, we free all reserved space of the changeset.

For example:

	ret = btrfs_qgroup_reserve_data(inode, changeset, 0, SZ_1M);
	ret = btrfs_qgroup_reserve_data(inode, changeset, SZ_1M, SZ_1M);
	ret = btrfs_qgroup_reserve_data(inode, changeset, SZ_2M, SZ_1M);

If the last btrfs_qgroup_reserve_data() failed, it will release all [0, 3M) range.

This behavior is kinda OK for now, as when we hit -EDQUOT, we normally go error handling and need to release all reserved ranges anyway.

But this also means the following call is not possible:

	ret = btrfs_qgroup_reserve_data();
	if (ret == -EDQUOT) {
		/* Do something to free some qgroup space */
		ret = btrfs_qgroup_reserve_data();
	}

As if the first btrfs_qgroup_reserve_data() fails, it will free all reserved qgroup space.

[CAUSE]
This is because we release all reserved ranges when btrfs_qgroup_reserve_data() fails.

[FIX]
This patch will implement a new function, qgroup_revert(), to iterate through the ulist nodes, to find any nodes in the failure range, and remove the EXTENT_QGROUP_RESERVED bits from the io_tree, and decrease the extent_changeset::bytes_changed, so that we can revert to previous status.

This allows later patches to retry btrfs_qgroup_reserve_data() if EDQUOT happens.

---
  fs/btrfs/qgroup.c | 90 +++++++++++++++++++++++++++++++++++++++--------
  1 file changed, 75 insertions(+), 15 deletions(-)
fs/btrfs/qgroup.c +10 -0
index ee0ad33b659c..84a452dea3f9 100644
@@ -3447,6 +3447,71 @@ btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info)
  	}
  }
 
+static int qgroup_revert(struct btrfs_inode *inode,
+			struct extent_changeset *reserved, u64 start,
+			u64 len)
+{
+	struct rb_node *n = reserved->range_changed.root.rb_node;
+	struct ulist_node *entry = NULL;
+	int ret = 0;
+
+	while (n) {
+		entry = rb_entry(n, struct ulist_node, rb_node);

fs/btrfs/qgroup.c btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info) · line 3469

Quoted 10 lines
fs/btrfs/qgroup.c +10 -0
+		if (entry->val < start)
+			n = n->rb_right;
+		else if (entry)
+			n = n->rb_left;
+		else
+			break;
+	}
+	/* Empty changeset */
+	if (!entry)
+		goto out;

Don't need the goto out here, just return ret;

fs/btrfs/qgroup.c btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info) · line 3476

Quoted 7 lines
fs/btrfs/qgroup.c +7 -0
+
+	if (entry->val > start && rb_prev(&entry->rb_node))
+		entry = rb_entry(rb_prev(&entry->rb_node), struct ulist_node,
+				 rb_node);
+
+	n = &entry->rb_node;
+	while (n) {

for (n = &entry->rb_node; n; n = rb_next(n)) {

Thanks,

Josef

#3 lore raw

Re: [PATCH v3 1/3] btrfs: qgroup: allow btrfs_qgroup_reserve_data() to revert the range it just set without releasing other existing ranges

Quoted 77 lines · Josef Bacik
Quoted 63 lines · Qu Wenruo

[PROBLEM]
Before this patch, when btrfs_qgroup_reserve_data() fails, we free all reserved space of the changeset.

For example:
    ret = btrfs_qgroup_reserve_data(inode, changeset, 0, SZ_1M); ret = btrfs_qgroup_reserve_data(inode, changeset, SZ_1M, SZ_1M); ret = btrfs_qgroup_reserve_data(inode, changeset, SZ_2M, SZ_1M);

If the last btrfs_qgroup_reserve_data() failed, it will release all [0, 3M) range.

This behavior is kinda OK for now, as when we hit -EDQUOT, we normally go error handling and need to release all reserved ranges anyway.

But this also means the following call is not possible:
    ret = btrfs_qgroup_reserve_data();
    if (ret == -EDQUOT) {
        /* Do something to free some qgroup space */ ret = btrfs_qgroup_reserve_data();
    }

As if the first btrfs_qgroup_reserve_data() fails, it will free all reserved qgroup space.

[CAUSE]
This is because we release all reserved ranges when btrfs_qgroup_reserve_data() fails.

[FIX]
This patch will implement a new function, qgroup_revert(), to iterate through the ulist nodes, to find any nodes in the failure range, and remove the EXTENT_QGROUP_RESERVED bits from the io_tree, and decrease the extent_changeset::bytes_changed, so that we can revert to previous status.

This allows later patches to retry btrfs_qgroup_reserve_data() if EDQUOT happens.

---
  fs/btrfs/qgroup.c | 90 +++++++++++++++++++++++++++++++++++++++--------

  1 file changed, 75 insertions(+), 15 deletions(-)
fs/btrfs/qgroup.c +9 -0
index ee0ad33b659c..84a452dea3f9 100644
@@ -3447,6 +3447,71 @@ btrfs_qgroup_rescan_resume(struct btrfs_fs_info
 *fs_info)
       }
   }
   +static int qgroup_revert(struct btrfs_inode *inode,
+            struct extent_changeset *reserved, u64 start,
+            u64 len)
+{
+    struct rb_node *n = reserved->range_changed.root.rb_node;
+    struct ulist_node *entry = NULL;
+    int ret = 0;
+
+    while (n) {
+        entry = rb_entry(n, struct ulist_node, rb_node);

fs/btrfs/qgroup.c btrfs_qgroup_rescan_resume(struct btrfs_fs_info · line 3469

Quoted 10 lines
fs/btrfs/qgroup.c +10 -0
+        if (entry->val < start)
+            n = n->rb_right;
+        else if (entry)
+            n = n->rb_left;
+        else
+            break;
+    }
+    /* Empty changeset */
+    if (!entry)
+        goto out;

Don't need the goto out here, just return ret;

Quoted 9 lines

fs/btrfs/qgroup.c btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info) · line 3476

Quoted 7 lines
fs/btrfs/qgroup.c +7 -0
+
+    if (entry->val > start && rb_prev(&entry->rb_node))
+        entry = rb_entry(rb_prev(&entry->rb_node), struct ulist_node,
+                 rb_node);
+
+    n = &entry->rb_node;
+    while (n) {

for (n = &entry->rb_node; n; n = rb_next(n)) {

You get into the trap!

Since @n can be deleted from the tree, it needs the @tmp to record what's the real next node.

But I can still use for loop with @tmp involved though.

Thanks,
Qu

Quoted 4 lines

Thanks,

Josef

#4 lore raw

Re: [PATCH v3 1/3] btrfs: qgroup: allow btrfs_qgroup_reserve_data() to revert the range it just set without releasing other existing ranges

In reply to Qu Wenruo
Quoted 50 lines · Qu Wenruo

[PROBLEM]
Before this patch, when btrfs_qgroup_reserve_data() fails, we free all reserved space of the changeset.

For example:

	ret = btrfs_qgroup_reserve_data(inode, changeset, 0, SZ_1M);
	ret = btrfs_qgroup_reserve_data(inode, changeset, SZ_1M, SZ_1M);
	ret = btrfs_qgroup_reserve_data(inode, changeset, SZ_2M, SZ_1M);

If the last btrfs_qgroup_reserve_data() failed, it will release all [0, 3M) range.

This behavior is kinda OK for now, as when we hit -EDQUOT, we normally go error handling and need to release all reserved ranges anyway.

But this also means the following call is not possible:

	ret = btrfs_qgroup_reserve_data();
	if (ret == -EDQUOT) {
		/* Do something to free some qgroup space */
		ret = btrfs_qgroup_reserve_data();
	}

As if the first btrfs_qgroup_reserve_data() fails, it will free all reserved qgroup space.

[CAUSE]
This is because we release all reserved ranges when btrfs_qgroup_reserve_data() fails.

[FIX]
This patch will implement a new function, qgroup_revert(), to iterate through the ulist nodes, to find any nodes in the failure range, and remove the EXTENT_QGROUP_RESERVED bits from the io_tree, and decrease the extent_changeset::bytes_changed, so that we can revert to previous status.

This allows later patches to retry btrfs_qgroup_reserve_data() if EDQUOT happens.

---
 fs/btrfs/qgroup.c | 90 +++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 75 insertions(+), 15 deletions(-)
fs/btrfs/qgroup.c +0 -0
index ee0ad33b659c..84a452dea3f9 100644

fs/btrfs/qgroup.c btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info) · line 3452

Quoted 6 lines
fs/btrfs/qgroup.c +3 -0
@@ -3447,6 +3447,71 @@ btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info)
 	}
 }
 
+static int qgroup_revert(struct btrfs_inode *inode,
+			struct extent_changeset *reserved, u64 start,
+			u64 len)

I've renamed it to qgroup_unreserve_range, as it's not clear what is being reverted.

fs/btrfs/qgroup.c btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info) · line 3463

Quoted 11 lines
fs/btrfs/qgroup.c +11 -0
+{
+	struct rb_node *n = reserved->range_changed.root.rb_node;
+	struct ulist_node *entry = NULL;
+	int ret = 0;
+
+	while (n) {
+		entry = rb_entry(n, struct ulist_node, rb_node);
+		if (entry->val < start)
+			n = n->rb_right;
+		else if (entry)
+			n = n->rb_left;

Please don't use single letter variables in new code unless it's 'i' for integer indexing. 'node' is fine.

fs/btrfs/qgroup.c btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info) · line 3469

Quoted 6 lines
fs/btrfs/qgroup.c +6 -0
+		else
+			break;
+	}
+	/* Empty changeset */
+	if (!entry)
+		goto out;

Switched to return as suggested.

fs/btrfs/qgroup.c btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info) · line 3477

Quoted 8 lines
fs/btrfs/qgroup.c +8 -0
+
+	if (entry->val > start && rb_prev(&entry->rb_node))
+		entry = rb_entry(rb_prev(&entry->rb_node), struct ulist_node,
+				 rb_node);
+
+	n = &entry->rb_node;
+	while (n) {
+		struct rb_node *tmp = rb_next(n);

Renamed to 'next'

All non-functional changes, no need to resend.

#5 lore raw

Re: [PATCH v3 1/3] btrfs: qgroup: allow btrfs_qgroup_reserve_data() to revert the range it just set without releasing other existing ranges

Quoted 96 lines · David Sterba
Quoted 50 lines · Qu Wenruo

[PROBLEM]
Before this patch, when btrfs_qgroup_reserve_data() fails, we free all reserved space of the changeset.

For example:

	ret = btrfs_qgroup_reserve_data(inode, changeset, 0, SZ_1M);
	ret = btrfs_qgroup_reserve_data(inode, changeset, SZ_1M, SZ_1M);
	ret = btrfs_qgroup_reserve_data(inode, changeset, SZ_2M, SZ_1M);

If the last btrfs_qgroup_reserve_data() failed, it will release all [0, 3M) range.

This behavior is kinda OK for now, as when we hit -EDQUOT, we normally go error handling and need to release all reserved ranges anyway.

But this also means the following call is not possible:

	ret = btrfs_qgroup_reserve_data();
	if (ret == -EDQUOT) {
		/* Do something to free some qgroup space */
		ret = btrfs_qgroup_reserve_data();
	}

As if the first btrfs_qgroup_reserve_data() fails, it will free all reserved qgroup space.

[CAUSE]
This is because we release all reserved ranges when btrfs_qgroup_reserve_data() fails.

[FIX]
This patch will implement a new function, qgroup_revert(), to iterate through the ulist nodes, to find any nodes in the failure range, and remove the EXTENT_QGROUP_RESERVED bits from the io_tree, and decrease the extent_changeset::bytes_changed, so that we can revert to previous status.

This allows later patches to retry btrfs_qgroup_reserve_data() if EDQUOT happens.

---
 fs/btrfs/qgroup.c | 90 +++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 75 insertions(+), 15 deletions(-)
fs/btrfs/qgroup.c +0 -0
index ee0ad33b659c..84a452dea3f9 100644

fs/btrfs/qgroup.c btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info) · line 3452

Quoted 6 lines
fs/btrfs/qgroup.c +3 -0
@@ -3447,6 +3447,71 @@ btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info)
 	}
 }
 
+static int qgroup_revert(struct btrfs_inode *inode,
+			struct extent_changeset *reserved, u64 start,
+			u64 len)

I've renamed it to qgroup_unreserve_range, as it's not clear what is being reverted.

fs/btrfs/qgroup.c btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info) · line 3463

Quoted 11 lines
fs/btrfs/qgroup.c +11 -0
+{
+	struct rb_node *n = reserved->range_changed.root.rb_node;
+	struct ulist_node *entry = NULL;
+	int ret = 0;
+
+	while (n) {
+		entry = rb_entry(n, struct ulist_node, rb_node);
+		if (entry->val < start)
+			n = n->rb_right;
+		else if (entry)
+			n = n->rb_left;

Please don't use single letter variables in new code unless it's 'i' for integer indexing. 'node' is fine.

fs/btrfs/qgroup.c btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info) · line 3469

Quoted 6 lines
fs/btrfs/qgroup.c +6 -0
+		else
+			break;
+	}
+	/* Empty changeset */
+	if (!entry)
+		goto out;

Switched to return as suggested.

fs/btrfs/qgroup.c btrfs_qgroup_rescan_resume(struct btrfs_fs_info *fs_info) · line 3477

Quoted 8 lines
fs/btrfs/qgroup.c +8 -0
+
+	if (entry->val > start && rb_prev(&entry->rb_node))
+		entry = rb_entry(rb_prev(&entry->rb_node), struct ulist_node,
+				 rb_node);
+
+	n = &entry->rb_node;
+	while (n) {
+		struct rb_node *tmp = rb_next(n);

Renamed to 'next'

Quoted 2 lines

All non-functional changes, no need to resend.

I haven't see it in misc-next yet, and consider the remaining two patches will need some modification anyway, would you mind to update the patchset?

Thanks,
Qu

author
#6 lore raw

PATCH v3 2/3 btrfs: qgroup: try to flush qgroup space when we get -EDQUOT

In reply to Qu Wenruo

[PROBLEM]
There are known problem related to how btrfs handles qgroup reserved space.
One of the most obvious case is the the test case btrfs/153, which do fallocate, then write into the preallocated range.

btrfs/153 1s ... - output mismatch (see xfstests-dev/results//btrfs/153.out.bad)
--- tests/btrfs/153.out 2019-10-22 15:18:14.068965341 +0800
+++ xfstests-dev/results//btrfs/153.out.bad 2020-07-01 20:24:40.730000089 +0800
@@ -1,2 +1,5 @@
QA output created by 153
+pwrite: Disk quota exceeded
+/mnt/scratch/testfile2: Disk quota exceeded
+/mnt/scratch/testfile2: Disk quota exceeded
Silence is golden
...
(Run 'diff -u xfstests-dev/tests/btrfs/153.out xfstests-dev/results//btrfs/153.out.bad' to see the entire diff)

[CAUSE]
Since commit c6887cd11149 ("Btrfs: don't do nocow check unless we have to"), we always reserve space no matter if it's COW or not.

Such behavior change is mostly for performance, and reverting it is not a good idea anyway.

For preallcoated extent, we reserve qgroup data space for it already, and since we also reserve data space for qgroup at buffered write time, it needs twice the space for us to write into preallocated space.

This leads to the -EDQUOT in buffered write routine.

And we can't follow the same solution, unlike data/meta space check, qgroup reserved space is shared between data/meta.
The EDQUOT can happen at the metadata reservation, so doing NODATACOW check after qgroup reservation failure is not a solution.

[FIX]
To solve the problem, we don't return -EDQUOT directly, but every time we got a -EDQUOT, we try to flush qgroup space by:
- Flush all inodes of the root
NODATACOW writes will free the qgroup reserved at run_dealloc_range(). However we don't have the infrastructure to only flush NODATACOW inodes, here we flush all inodes anyway.

- Wait ordered extents
This would convert the preallocated metadata space into per-trans metadata, which can be freed in later transaction commit.

- Commit transaction
This will free all per-trans metadata space.

Also we don't want to trigger flush too racy, so here we introduce a per-root mutex to ensure if there is a running qgroup flushing, we wait for it to end and don't start re-flush.

Fixes c6887cd11149 ("Btrfs: don't do nocow check unless we have to")
---
 fs/btrfs/ctree.h   |   1 +
 fs/btrfs/disk-io.c |   1 +
 fs/btrfs/qgroup.c  | 101 +++++++++++++++++++++++++++++++++++++++++----
 3 files changed, 95 insertions(+), 8 deletions(-)
fs/btrfs/ctree.h +1 -0
index 4dd478b4fe3a..891f47c7891f 100644
@@ -1162,6 +1162,7 @@ struct btrfs_root {
 	spinlock_t qgroup_meta_rsv_lock;
 	u64 qgroup_meta_rsv_pertrans;
 	u64 qgroup_meta_rsv_prealloc;
+	struct mutex qgroup_flushing_mutex;
 
 	/* Number of active swapfiles */
 	atomic_t nr_swapfiles;
fs/btrfs/disk-io.c +1 -0
index c27022f13150..0116e0b487c9 100644
@@ -1116,6 +1116,7 @@ static void __setup_root(struct btrfs_root *root, struct btrfs_fs_info *fs_info,
 	mutex_init(&root->log_mutex);
 	mutex_init(&root->ordered_extent_mutex);
 	mutex_init(&root->delalloc_mutex);
+	mutex_init(&root->qgroup_flushing_mutex);
 	init_waitqueue_head(&root->log_writer_wait);
 	init_waitqueue_head(&root->log_commit_wait[0]);
 	init_waitqueue_head(&root->log_commit_wait[1]);
fs/btrfs/qgroup.c +93 -8

129 lines not on this page · raw message

Signature · 1 line
2.27.0
Reviewed-by on v3 2/3
#7 lore raw

Re: [PATCH v3 2/3] btrfs: qgroup: try to flush qgroup space when we get -EDQUOT

Quoted 51 lines · Qu Wenruo

[PROBLEM]
There are known problem related to how btrfs handles qgroup reserved space.
One of the most obvious case is the the test case btrfs/153, which do fallocate, then write into the preallocated range.

btrfs/153 1s ... - output mismatch (see xfstests-dev/results//btrfs/153.out.bad)
--- tests/btrfs/153.out 2019-10-22 15:18:14.068965341 +0800
+++ xfstests-dev/results//btrfs/153.out.bad 2020-07-01 20:24:40.730000089 +0800
@@ -1,2 +1,5 @@
QA output created by 153
+pwrite: Disk quota exceeded
+/mnt/scratch/testfile2: Disk quota exceeded
+/mnt/scratch/testfile2: Disk quota exceeded
Silence is golden
...
(Run 'diff -u xfstests-dev/tests/btrfs/153.out xfstests-dev/results//btrfs/153.out.bad' to see the entire diff)

[CAUSE]
Since commit c6887cd11149 ("Btrfs: don't do nocow check unless we have to"), we always reserve space no matter if it's COW or not.

Such behavior change is mostly for performance, and reverting it is not a good idea anyway.

For preallcoated extent, we reserve qgroup data space for it already, and since we also reserve data space for qgroup at buffered write time, it needs twice the space for us to write into preallocated space.

This leads to the -EDQUOT in buffered write routine.

And we can't follow the same solution, unlike data/meta space check, qgroup reserved space is shared between data/meta.
The EDQUOT can happen at the metadata reservation, so doing NODATACOW check after qgroup reservation failure is not a solution.

[FIX]
To solve the problem, we don't return -EDQUOT directly, but every time we got a -EDQUOT, we try to flush qgroup space by:
- Flush all inodes of the root
NODATACOW writes will free the qgroup reserved at run_dealloc_range(). However we don't have the infrastructure to only flush NODATACOW inodes, here we flush all inodes anyway.

- Wait ordered extents
This would convert the preallocated metadata space into per-trans metadata, which can be freed in later transaction commit.

- Commit transaction
This will free all per-trans metadata space.

Quoted 6 lines

Also we don't want to trigger flush too racy, so here we introduce a per-root mutex to ensure if there is a running qgroup flushing, we wait for it to end and don't start re-flush.

Fixes c6887cd11149 ("Btrfs: don't do nocow check unless we have to")

Thanks,

Josef

#8 lore raw

Re: [PATCH v3 2/3] btrfs: qgroup: try to flush qgroup space when we get -EDQUOT

In reply to Qu Wenruo
Quoted 4 lines · Qu Wenruo
fs/btrfs/qgroup.c +3 -1
-int btrfs_qgroup_reserve_data(struct btrfs_inode *inode,
+static int try_flush_qgroup(struct btrfs_root *root)
+{
+	struct btrfs_trans_handle *trans;

fs/btrfs/qgroup.c static int qgroup_revert(struct btrfs_inode *inode, · line 3547

Quoted 10 lines
fs/btrfs/qgroup.c +10 -0
+	int ret;
+
+	/*
+	 * We don't want to run flush again and again, so if there is a running
+	 * one, we won't try to start a new flush, but exit directly.
+	 */
+	ret = mutex_trylock(&root->qgroup_flushing_mutex);
+	if (!ret) {
+		mutex_lock(&root->qgroup_flushing_mutex);
+		mutex_unlock(&root->qgroup_flushing_mutex);

This is abuse of mutex, for status tracking "is somebody flushing" and for waiting until it's over.

We have many root::status bits (the BTRFS_ROOT_* namespace) so that qgroups are flushing should another one. The bit atomically set when it starts and cleared when it ends.

All waiting tasks should queue in a normal wait_queue_head.

Quoted 7 lines
fs/btrfs/qgroup.c +7 -0
+		return 0;
+	}
+
+	ret = btrfs_start_delalloc_snapshot(root);
+	if (ret < 0)
+		goto unlock;
+	btrfs_wait_ordered_extents(root, U64_MAX, 0, (u64)-1);

fs/btrfs/qgroup.c static int qgroup_revert(struct btrfs_inode *inode, · line 3564

Quoted 10 lines
fs/btrfs/qgroup.c +10 -0
+
+	trans = btrfs_join_transaction(root);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		goto unlock;
+	}
+
+	ret = btrfs_commit_transaction(trans);
+unlock:
+	mutex_unlock(&root->qgroup_flushing_mutex);

And also the whole wait/join/commit combo is in one huge mutex, that's really an anti-pattern.

#9 lore raw

Re: [PATCH v3 2/3] btrfs: qgroup: try to flush qgroup space when we get -EDQUOT

Quoted 19 lines · David Sterba
Quoted 4 lines · Qu Wenruo
(quoted diff) +3 -1
-int btrfs_qgroup_reserve_data(struct btrfs_inode *inode,
+static int try_flush_qgroup(struct btrfs_root *root)
+{
+	struct btrfs_trans_handle *trans;

Review

Quoted 10 lines
(quoted diff) +10 -0
+	int ret;
+
+	/*
+	 * We don't want to run flush again and again, so if there is a running
+	 * one, we won't try to start a new flush, but exit directly.
+	 */
+	ret = mutex_trylock(&root->qgroup_flushing_mutex);
+	if (!ret) {
+		mutex_lock(&root->qgroup_flushing_mutex);
+		mutex_unlock(&root->qgroup_flushing_mutex);

This is abuse of mutex, for status tracking "is somebody flushing" and for waiting until it's over.

Quoted 5 lines

We have many root::status bits (the BTRFS_ROOT_* namespace) so that qgroups are flushing should another one. The bit atomically set when it starts and cleared when it ends.

All waiting tasks should queue in a normal wait_queue_head.

Something like that (untested):

fs/btrfs/ctree.h +7 -1
index 2cc1fcaa7cfa..5dbb6b7300b7 100644
@@ -1007,6 +1007,12 @@ enum {
 	BTRFS_ROOT_DEAD_TREE,
 	/* The root has a log tree. Used only for subvolume roots. */
 	BTRFS_ROOT_HAS_LOG_TREE,
+
+	/*
+	 * Indicate that qgroup flushing is in progress to prevent multiple
+	 * processes attempting that
+	 */
+	BTRFS_ROOT_QGROUP_FLUSHING,
 };
 
 /*
@@ -1159,7 +1165,7 @@ struct btrfs_root {
 	spinlock_t qgroup_meta_rsv_lock;
 	u64 qgroup_meta_rsv_pertrans;
 	u64 qgroup_meta_rsv_prealloc;
-	struct mutex qgroup_flushing_mutex;
+	wait_queue_head_t qgroup_flush_wait;
 
 	/* Number of active swapfiles */
 	atomic_t nr_swapfiles;
fs/btrfs/disk-io.c +1 -1
index 8029127df537..e124e3376208 100644
@@ -1116,7 +1116,7 @@ static void __setup_root(struct btrfs_root *root, struct btrfs_fs_info *fs_info,
 	mutex_init(&root->log_mutex);
 	mutex_init(&root->ordered_extent_mutex);
 	mutex_init(&root->delalloc_mutex);
-	mutex_init(&root->qgroup_flushing_mutex);
+	init_waitqueue_head(&root->qgroup_flush_wait);
 	init_waitqueue_head(&root->log_writer_wait);
 	init_waitqueue_head(&root->log_commit_wait[0]);
 	init_waitqueue_head(&root->log_commit_wait[1]);
fs/btrfs/qgroup.c +4 -5
index 494ab2b1bbf2..95695aca7aa9 100644
@@ -3503,10 +3503,9 @@ static int try_flush_qgroup(struct btrfs_root *root)
 	 * We don't want to run flush again and again, so if there is a running
 	 * one, we won't try to start a new flush, but exit directly.
 	 */
-	ret = mutex_trylock(&root->qgroup_flushing_mutex);
-	if (!ret) {
-		mutex_lock(&root->qgroup_flushing_mutex);
-		mutex_unlock(&root->qgroup_flushing_mutex);
+	if (test_and_set_bit(BTRFS_ROOT_QGROUP_FLUSHING, &root->state)) {
+		wait_event(root->qgroup_flush_wait,
+			!test_bit(BTRFS_ROOT_QGROUP_FLUSHING, &root->state));
 		return 0;
 	}
 
@@ -3523,7 +3522,7 @@ static int try_flush_qgroup(struct btrfs_root *root)
 
 	ret = btrfs_commit_transaction(trans);
 unlock:
-	mutex_unlock(&root->qgroup_flushing_mutex);
+	clear_bit(BTRFS_ROOT_QGROUP_FLUSHING, &root->state);
 	return ret;
 }
 
#10 lore raw

Re: [PATCH v3 2/3] btrfs: qgroup: try to flush qgroup space when we get -EDQUOT

In reply to David Sterba
Quoted 19 lines · David Sterba
Quoted 4 lines · Qu Wenruo
fs/btrfs/qgroup.c +3 -1
-int btrfs_qgroup_reserve_data(struct btrfs_inode *inode,
+static int try_flush_qgroup(struct btrfs_root *root)
+{
+	struct btrfs_trans_handle *trans;

fs/btrfs/qgroup.c static int qgroup_revert(struct btrfs_inode *inode, · line 3547

Quoted 10 lines
fs/btrfs/qgroup.c +10 -0
+	int ret;
+
+	/*
+	 * We don't want to run flush again and again, so if there is a running
+	 * one, we won't try to start a new flush, but exit directly.
+	 */
+	ret = mutex_trylock(&root->qgroup_flushing_mutex);
+	if (!ret) {
+		mutex_lock(&root->qgroup_flushing_mutex);
+		mutex_unlock(&root->qgroup_flushing_mutex);

This is abuse of mutex, for status tracking "is somebody flushing" and for waiting until it's over.

Quoted 3 lines

We have many root::status bits (the BTRFS_ROOT_* namespace) so that qgroups are flushing should another one. The bit atomically set when it starts and cleared when it ends.

In fact, I want to avoid plain wait_queue usage if possible.

Unlike mutex, wait_queue doesn't have good enough debug mechanism like lockdep.

I see no reason re-implementing the existing mutex code by ourselves could bring any benefit here.

It may looks like an abuse of mutex, but I could wrap it into something like wait_or_lock_mutex(), which may slightly improve the readability.

Or am I missing anything?

Quoted 15 lines

All waiting tasks should queue in a normal wait_queue_head.

Quoted 12 lines
fs/btrfs/qgroup.c +12 -0
+		return 0;
+	}
+
+	ret = btrfs_start_delalloc_snapshot(root);
+	if (ret < 0)
+		goto unlock;
+	btrfs_wait_ordered_extents(root, U64_MAX, 0, (u64)-1);
+
+	trans = btrfs_join_transaction(root);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		goto unlock;
Quoted 9 lines

fs/btrfs/qgroup.c static int qgroup_revert(struct btrfs_inode *inode, · line 3564

Quoted 5 lines
fs/btrfs/qgroup.c +5 -0
+	}
+
+	ret = btrfs_commit_transaction(trans);
+unlock:
+	mutex_unlock(&root->qgroup_flushing_mutex);

And also the whole wait/join/commit combo is in one huge mutex, that's really an anti-pattern.

But that mutex is per-root, and is the slow path.

Converting to wait_queue won't change the pattern either.

Thanks,
Qu

author
#11 lore raw

PATCH v3 3/3 btrfs: qgroup: remove the ASYNC_COMMIT mechanism in favor of qgroup reserve retry-after-EDQUOT

In reply to Qu Wenruo

commit a514d63882c3 ("btrfs: qgroup: Commit transaction in advance to reduce early EDQUOT") tries to reduce the early EDQUOT problems by checking the qgroup free against threshold and try to wake up commit kthread to free some space.

The problem of that mechanism is, it can only free qgroup per-trans metadata space, can't do anything to data, nor prealloc qgroup space.

Now since we have the ability to flush qgroup space, and implements retry-after-EDQUOT behavior, such mechanism is completely replaced.

So this patch will cleanup such mechanism in favor of retry-after-EDQUOT.

---
 fs/btrfs/ctree.h       |  5 -----
 fs/btrfs/disk-io.c     |  1 -
 fs/btrfs/qgroup.c      | 43 ++----------------------------------------
 fs/btrfs/transaction.c |  1 -
 fs/btrfs/transaction.h | 14 --------------
 5 files changed, 2 insertions(+), 62 deletions(-)
fs/btrfs/ctree.h +0 -5
index 891f47c7891f..373567c168ac 100644
@@ -545,11 +545,6 @@ enum {
 	 * (device replace, resize, device add/delete, balance)
 	 */
 	BTRFS_FS_EXCL_OP,
-	/*
-	 * To info transaction_kthread we need an immediate commit so it
-	 * doesn't need to wait for commit_interval
-	 */
-	BTRFS_FS_NEED_ASYNC_COMMIT,
 	/*
 	 * Indicate that balance has been set up from the ioctl and is in the
 	 * main phase. The fs_info::balance_ctl is initialized.
fs/btrfs/disk-io.c +0 -1
index 0116e0b487c9..3b80d88e6ab2 100644
@@ -1746,7 +1746,6 @@ static int transaction_kthread(void *arg)
 
 		now = ktime_get_seconds();
 		if (cur->state < TRANS_STATE_COMMIT_START &&
-		    !test_bit(BTRFS_FS_NEED_ASYNC_COMMIT, &fs_info->flags) &&
 		    (now < cur->start_time ||
 		     now - cur->start_time < fs_info->commit_interval)) {
 			spin_unlock(&fs_info->trans_lock);
fs/btrfs/qgroup.c +2 -41

68 lines not on this page · raw message

fs/btrfs/transaction.c +0 -1
index b359d4b17658..d0a6150bf82d 100644
@@ -2351,7 +2351,6 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans)
 	 */
 	cur_trans->state = TRANS_STATE_COMPLETED;
 	wake_up(&cur_trans->commit_wait);
-	clear_bit(BTRFS_FS_NEED_ASYNC_COMMIT, &fs_info->flags);
 
 	spin_lock(&fs_info->trans_lock);
 	list_del_init(&cur_trans->list);
fs/btrfs/transaction.h +0 -14
index 6f65fff6cf50..0b18d25aa9b6 100644
@@ -208,20 +208,6 @@ int btrfs_clean_one_deleted_snapshot(struct btrfs_root *root);
 int btrfs_commit_transaction(struct btrfs_trans_handle *trans);
 int btrfs_commit_transaction_async(struct btrfs_trans_handle *trans,
 				   int wait_for_unblock);
-
-/*
- * Try to commit transaction asynchronously, so this is safe to call
- * even holding a spinlock.
- *
- * It's done by informing transaction_kthread to commit transaction without
- * waiting for commit interval.
- */
-static inline void btrfs_commit_transaction_locksafe(
-		struct btrfs_fs_info *fs_info)
-{
-	set_bit(BTRFS_FS_NEED_ASYNC_COMMIT, &fs_info->flags);
-	wake_up_process(fs_info->transaction_kthread);
-}
 int btrfs_end_transaction_throttle(struct btrfs_trans_handle *trans);
 int btrfs_should_end_transaction(struct btrfs_trans_handle *trans);
 void btrfs_throttle(struct btrfs_fs_info *fs_info);
Signature · 1 line
2.27.0
#12 lore raw

Re: [PATCH v3 3/3] btrfs: qgroup: remove the ASYNC_COMMIT mechanism in favor of qgroup reserve retry-after-EDQUOT

Quoted 8 lines · Qu Wenruo

commit a514d63882c3 ("btrfs: qgroup: Commit transaction in advance to reduce early EDQUOT") tries to reduce the early EDQUOT problems by checking the qgroup free against threshold and try to wake up commit kthread to free some space.

The problem of that mechanism is, it can only free qgroup per-trans metadata space, can't do anything to data, nor prealloc qgroup space.

Quoted 8 lines

Now since we have the ability to flush qgroup space, and implements retry-after-EDQUOT behavior, such mechanism is completely replaced.

So this patch will cleanup such mechanism in favor of retry-after-EDQUOT.

That one looks good to me.