Skip to contentSkip to conversation

btrfs: squota lifetime fixes

PATCH In review Boris Burkov 3 participants 9 messages May 12 – May 13 Btrfs file system

lore
Share & tools
lore
#0 lore raw

cover btrfs: squota lifetime fixes

A recent report of read-only filesystems since enabling squotas:

https://lore.kernel.org/linux-btrfs/[email protected]/

led me to investigate a handful of issues in squotas and find some trivial and not so trivial bugs in their lifetime management. The aim of the series is to strengthen enforcement of the invariant: "a squota qgroup must outlast extents that are accounted to it".

This series consists of two fixes to squota lifetime bugs and two patches that reduce the consequences of errors in lifetime management from full fs goes read-only abort to "warning and inaccurate squotas which can be deleted eventually"

Additional improvements to btrfs check and fstests to follow.

Boris Burkov (4):
btrfs: check for subvolume before deleting squota qgroup
btrfs: fix squota accounting during enable generation
btrfs: clamp to avoid squota underflow
btrfs: swallow btrfs_record_squota_delta() enoent

 fs/btrfs/fs.h     |  1 +
 fs/btrfs/qgroup.c | 99 ++++++++++++++++++++++++++++++++---------------
 2 files changed, 68 insertions(+), 32 deletions(-)
Signature · 1 line
2.54.0
#1 lore raw

PATCH 1/4 btrfs: check for subvolume before deleting squota qgroup

The invariant that we want to maintain with subvolume qgroups is that the qgroup can only be deleted if there is no root. With squotas, we thought that it was sufficient to just check the usage, because we assumed that deleting a subvolume will drive it's qgroups usage to 0, and thus 0 usage implies no subvolume.

However, this is false, for two reasons:
- A subvol whose extents are all from before squotas was enabled.
- A subvol that was created in this transaction and for which we have not yet run any delayed refs.

In both cases, deleting the qgroup breaks the desired invariant and we are left with a subvolume with no qgroup but squotas are enabled.

Fix this by unifying the deletion check logic between full qgroups and squotas. Squotas do all the same checks *and* the additional usage == 0 check, which is the one extra rule peculiar to squotas.

Fixes a8df35619948 ("btrfs: forbid deleting live subvol qgroup")
---
 fs/btrfs/qgroup.c | 51 ++++++++++++++++++++++++-----------------------
 1 file changed, 26 insertions(+), 25 deletions(-)
fs/btrfs/qgroup.c +26 -25
index 0511b6dc72b0..5cd2e0ceb1d8 100644
@@ -1715,32 +1715,25 @@ int btrfs_create_qgroup(struct btrfs_trans_handle *trans, u64 qgroupid)
 	return ret;
 }
 
-static bool can_delete_parent_qgroup(struct btrfs_qgroup *qgroup)
-
+static bool can_delete_parent_qgroup(struct btrfs_fs_info *fs_info,
+				     struct btrfs_qgroup *qgroup)
 {
 	ASSERT(btrfs_qgroup_level(qgroup->qgroupid));
+	if (btrfs_qgroup_mode(fs_info) == BTRFS_QGROUP_MODE_SIMPLE)
+		squota_check_parent_usage(fs_info, qgroup);
 	return list_empty(&qgroup->members);
 }
 
 /*
- * Return true if we can delete the squota qgroup and false otherwise.
- *
- * Rules for whether we can delete:
- *
- * A subvolume qgroup can be removed iff the subvolume is fully deleted, which
- * is iff there is 0 usage in the qgroup.
- *
- * A higher level qgroup can be removed iff it has no members.
- * Note: We audit its usage to warn on inconsitencies without blocking deletion.
+ * Because a shared extent can outlive its owning subvolume, we cannot delete a
+ * subvol squota qgroup until all of the extents it owns are gone, even if the
+ * subvolume itself has been deleted.
  */
-static bool can_delete_squota_qgroup(struct btrfs_fs_info *fs_info, struct btrfs_qgroup *qgroup)
+static bool can_delete_squota_subvol_qgroup(struct btrfs_fs_info *fs_info,
+					    struct btrfs_qgroup *qgroup)
 {
 	ASSERT(btrfs_qgroup_mode(fs_info) == BTRFS_QGROUP_MODE_SIMPLE);
-
-	if (btrfs_qgroup_level(qgroup->qgroupid) > 0) {
-		squota_check_parent_usage(fs_info, qgroup);
-		return can_delete_parent_qgroup(qgroup);
-	}
+	ASSERT(btrfs_qgroup_level(qgroup->qgroupid) == 0);
 
 	return !(qgroup->rfer || qgroup->excl || qgroup->rfer_cmpr || qgroup->excl_cmpr);
 }
@@ -1754,14 +1747,11 @@ static int can_delete_qgroup(struct btrfs_fs_info *fs_info, struct btrfs_qgroup
 {
 	struct btrfs_key key;
 	BTRFS_PATH_AUTO_FREE(path);
-
-	/* Since squotas cannot be inconsistent, they have special rules for deletion. */
-	if (btrfs_qgroup_mode(fs_info) == BTRFS_QGROUP_MODE_SIMPLE)
-		return can_delete_squota_qgroup(fs_info, qgroup);
+	int ret;
 
 	/* For higher level qgroup, we can only delete it if it has no child. */
 	if (btrfs_qgroup_level(qgroup->qgroupid))
-		return can_delete_parent_qgroup(qgroup);
+		return can_delete_parent_qgroup(fs_info, qgroup);
 
 	/*
 	 * For level-0 qgroups, we can only delete it if it has no subvolume
@@ -1777,10 +1767,21 @@ static int can_delete_qgroup(struct btrfs_fs_info *fs_info, struct btrfs_qgroup
 		return -ENOMEM;
 
 	/*
-	 * The @ret from btrfs_find_root() exactly matches our definition for
-	 * the return value, thus can be returned directly.
+	 * Any subvol qgroup, regardless of mode, cannot be deleted if the
+	 * subvol still exists.
+	 */
+	ret = btrfs_find_root(fs_info->tree_root, &key, path, NULL, NULL);
+	/*
+	 * btrfs_find_root returns <0 on error, 0 if found, and >0 if not,
+	 * so the "found" and "error" cases match our desired return values.
 	 */
-	return btrfs_find_root(fs_info->tree_root, &key, path, NULL, NULL);
+	if (ret <= 0)
+		return ret;
+
+	/* Squotas require additional checks, even if the subvol is deleted. */
+	if (btrfs_qgroup_mode(fs_info) == BTRFS_QGROUP_MODE_SIMPLE)
+		return can_delete_squota_subvol_qgroup(fs_info, qgroup);
+	return 1;
 }
 
 int btrfs_remove_qgroup(struct btrfs_trans_handle *trans, u64 qgroupid)
Signature · 1 line
2.54.0
#2 lore raw

PATCH 2/4 btrfs: fix squota accounting during enable generation

In reply to Boris Burkov

The first transaction that enables squotas is special and a bit tricky. We have to set BTRFS_FS_QUOTA_ENABLED after the transaction to avoid a deadlock, so any delayed refs that run before we set the bit are not squota accounted. For data this is fine, we don't get an owner_ref, so there is no real harm, it's as if the extent predated squotas. However for metadata, the tree block will have gen == enable_gen so when we free it later, we will decrement the squota accounting, which can result in an underflow. Before it is freed, btrfs check shows errors, as we have mismatched usage between the node generations/owners and the squota values.

There are two angles to this fix:
1. For extents that come in delayed_refs that run during the enable_gen transaction, we must actually set enable_gen to the *next* transaction. That is the first transaction that we can really properly account in any way.
2. For extents that come in between the end of our transaction handle and the time we set the BTRFS_FS_QUOTA_ENABLED bit, we need an additional bit, BTRFS_FS_SQUOTA_ENABLING which only affects recording squota deltas, so we do pick up those extents. Otherwise, we would miss them, even for enable_gen + 1.

Fixes bd7c1ea3a302 ("btrfs: qgroup: check generation when recording simple quota delta")
---
 fs/btrfs/fs.h     |  1 +
 fs/btrfs/qgroup.c | 31 +++++++++++++++++++++++++++----
 2 files changed, 28 insertions(+), 4 deletions(-)
fs/btrfs/fs.h +1 -0
index f821aa762613..1725611a3450 100644
@@ -145,6 +145,7 @@ enum {
 	BTRFS_FS_LOG_RECOVERING,
 	BTRFS_FS_OPEN,
 	BTRFS_FS_QUOTA_ENABLED,
+	BTRFS_FS_SQUOTA_ENABLING,
 	BTRFS_FS_UPDATE_UUID_TREE_GEN,
 	BTRFS_FS_CREATING_FREE_SPACE_TREE,
 	BTRFS_FS_BTREE_ERR,
fs/btrfs/qgroup.c +27 -4
index 5cd2e0ceb1d8..96746ca19e17 100644
@@ -1107,7 +1107,13 @@ int btrfs_quota_enable(struct btrfs_fs_info *fs_info,
 	if (simple) {
 		fs_info->qgroup_flags |= BTRFS_QGROUP_STATUS_FLAG_SIMPLE_MODE;
 		btrfs_set_fs_incompat(fs_info, SIMPLE_QUOTA);
-		btrfs_set_qgroup_status_enable_gen(leaf, ptr, trans->transid);
+		/*
+		 * Set the enable generation to the next transaction, as we cannot
+		 * ensure that extents written during this transaction will see any
+		 * state we have set here. So we should treat all extents of the
+		 * transaction as coming in before squotas was enabled.
+		 */
+		btrfs_set_qgroup_status_enable_gen(leaf, ptr, trans->transid + 1);
 	} else {
 		fs_info->qgroup_flags |= BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT;
 	}
@@ -1210,7 +1216,15 @@ int btrfs_quota_enable(struct btrfs_fs_info *fs_info,
 		goto out_free_path;
 	}
 
-	fs_info->qgroup_enable_gen = trans->transid;
+	/*
+	 * Set fs_info->qgroup_enable_gen and BTRFS_FS_SQUOTA_ENABLING
+	 * under the transaction handle. We want to ensure that all extents in
+	 * the next transaction definitely see them.
+	 */
+	if (simple) {
+		fs_info->qgroup_enable_gen = trans->transid + 1;
+		set_bit(BTRFS_FS_SQUOTA_ENABLING, &fs_info->flags);
+	}
 
 	mutex_unlock(&fs_info->qgroup_ioctl_lock);
 	/*
@@ -1224,9 +1238,15 @@ int btrfs_quota_enable(struct btrfs_fs_info *fs_info,
 	 */
 	ret = btrfs_commit_transaction(trans);
 	trans = NULL;
+
 	mutex_lock(&fs_info->qgroup_ioctl_lock);
-	if (ret)
+	if (ret) {
+		if (simple) {
+			clear_bit(BTRFS_FS_SQUOTA_ENABLING, &fs_info->flags);
+			fs_info->qgroup_enable_gen = 0;
+		}
 		goto out_free_path;
+	}
 
 	/*
 	 * Set quota enabled flag after committing the transaction, to avoid
@@ -1236,6 +1256,8 @@ int btrfs_quota_enable(struct btrfs_fs_info *fs_info,
 	spin_lock(&fs_info->qgroup_lock);
 	fs_info->quota_root = quota_root;
 	set_bit(BTRFS_FS_QUOTA_ENABLED, &fs_info->flags);
+	if (simple)
+		clear_bit(BTRFS_FS_SQUOTA_ENABLING, &fs_info->flags);
 	spin_unlock(&fs_info->qgroup_lock);
 
 	/* Skip rescan for simple qgroups. */
@@ -4923,7 +4945,8 @@ int btrfs_record_squota_delta(struct btrfs_fs_info *fs_info,
 	u64 num_bytes = delta->num_bytes;
 	const int sign = (delta->is_inc ? 1 : -1);
 
-	if (btrfs_qgroup_mode(fs_info) != BTRFS_QGROUP_MODE_SIMPLE)
+	if (btrfs_qgroup_mode(fs_info) != BTRFS_QGROUP_MODE_SIMPLE &&
+	    !test_bit(BTRFS_FS_SQUOTA_ENABLING, &fs_info->flags))
 		return 0;
 
 	if (!btrfs_is_fstree(root))
Signature · 1 line
2.54.0
#3 lore raw

PATCH 3/4 btrfs: clamp to avoid squota underflow

In reply to Boris Burkov

Simple quota accounting can undercount metadata tree block allocations in certain scenarios. When an undercounted subvolume is deleted and its tree blocks freed, the free deltas decrement rfer/excl past zero, wrapping the u64 to a value near U64_MAX.

Once wrapped, can_delete_squota_qgroup() sees non-zero rfer and refuses to delete the qgroup. The qgroup becomes permanently orphaned in the quota tree, since there is no subvolume left to generate frees that would bring the counter back to zero.

While we ultimately want to fix any mis-accounting at the source, it is also helpful and worthwhile to mitigate the damage by clamping rfer and excl to zero on underflow rather than allowing the u64 to wrap. This at least allows us to clean up the messed up qgroups on subvol deletion.

---
 fs/btrfs/qgroup.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
fs/btrfs/qgroup.c +13 -2
index 96746ca19e17..a93d49015fc7 100644
@@ -4968,8 +4968,19 @@ int btrfs_record_squota_delta(struct btrfs_fs_info *fs_info,
 	list_for_each_entry(qg, &qgroup_list, iterator) {
 		struct btrfs_qgroup_list *glist;
 
-		qg->excl += num_bytes * sign;
-		qg->rfer += num_bytes * sign;
+		ASSERT(qg->excl == qg->rfer);
+		if (sign < 0 && qg->excl < num_bytes) {
+			btrfs_warn(fs_info,
+				   "squota underflow qg %hu/%llu excl %llu num_bytes %llu",
+				   btrfs_qgroup_level(qg->qgroupid),
+				   btrfs_qgroup_subvolid(qg->qgroupid),
+				   qg->excl, num_bytes);
+			qg->excl = 0;
+			qg->rfer = 0;
+		} else {
+			qg->excl += num_bytes * sign;
+			qg->rfer += num_bytes * sign;
+		}
 		qgroup_dirty(fs_info, qg);
 
 		list_for_each_entry(glist, &qg->groups, next_group)
Signature · 1 line
2.54.0
#5 lore raw

PATCH 4/4 btrfs: swallow btrfs_record_squota_delta() enoent

In reply to Boris Burkov

I thought that it was likely I could harden squota deletion to the point that it was impossible to end up with an extent accounted to a qgroup outliving its qgroup. Several recent bugs have made me re-consider that position.

Ultimately, this is a tradeoff between short term stability and long term strictness, but I think given that there could be another layer of bugs behind the 2-3 I just fixed, I would feel much more confident in people using squotas if the risk was "your values can get a bit out of whack which you can fix by deleting stuff or disabling/re-enabling/repairing" vs "it will abort your filesystem".

As the final nail in the coffin, the Meta production kernel was lacking earlier fixes from me and Qu regarding subvol qgroup lifetime, so this is what we have been testing at scale, so I think at least for now upstream should have the same extra layer of protection.

---
 fs/btrfs/qgroup.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/qgroup.c +1 -1
index a93d49015fc7..f9e5005028dc 100644
@@ -4959,7 +4959,7 @@ int btrfs_record_squota_delta(struct btrfs_fs_info *fs_info,
 	spin_lock(&fs_info->qgroup_lock);
 	qgroup = find_qgroup_rb(fs_info, root);
 	if (!qgroup) {
-		ret = -ENOENT;
+		ret = 0;
 		goto out;
 	}
 
Signature · 1 line
2.54.0
#4 lore raw

Re: [PATCH 3/4] btrfs: clamp to avoid squota underflow

In reply to Boris Burkov
Quoted 27 lines · Boris Burkov

Simple quota accounting can undercount metadata tree block allocations in certain scenarios. When an undercounted subvolume is deleted and its tree blocks freed, the free deltas decrement rfer/excl past zero, wrapping the u64 to a value near U64_MAX.

Once wrapped, can_delete_squota_qgroup() sees non-zero rfer and refuses to delete the qgroup. The qgroup becomes permanently orphaned in the quota tree, since there is no subvolume left to generate frees that would bring the counter back to zero.

While we ultimately want to fix any mis-accounting at the source, it is also helpful and worthwhile to mitigate the damage by clamping rfer and excl to zero on underflow rather than allowing the u64 to wrap. This at least allows us to clean up the messed up qgroups on subvol deletion.

---
  fs/btrfs/qgroup.c | 15 +++++++++++++--
  1 file changed, 13 insertions(+), 2 deletions(-)
fs/btrfs/qgroup.c +0 -0
index 96746ca19e17..a93d49015fc7 100644
@@ -4968,8 +4968,19 @@ int btrfs_record_squota_delta(struct btrfs_fs_info *fs_info,
  	list_for_each_entry(qg, &qgroup_list, iterator) {
  		struct btrfs_qgroup_list *glist;

fs/btrfs/qgroup.c int btrfs_record_squota_delta(struct btrfs_fs_info *fs_info, · line 4977

Quoted 10 lines
fs/btrfs/qgroup.c +7 -2
 
-		qg->excl += num_bytes * sign;
-		qg->rfer += num_bytes * sign;
+		ASSERT(qg->excl == qg->rfer);
+		if (sign < 0 && qg->excl < num_bytes) {
+			btrfs_warn(fs_info,
+				   "squota underflow qg %hu/%llu excl %llu num_bytes %llu",
+				   btrfs_qgroup_level(qg->qgroupid),
+				   btrfs_qgroup_subvolid(qg->qgroupid),
+				   qg->excl, num_bytes);

This may not be noisy enough for fstests to catch.

Maybe a WARN_ON_ONCE()?

Thanks,
Qu

Quoted 9 lines
fs/btrfs/qgroup.c +6 -0
+			qg->excl = 0;
+			qg->rfer = 0;
+		} else {
+			qg->excl += num_bytes * sign;
+			qg->rfer += num_bytes * sign;
+		}
  		qgroup_dirty(fs_info, qg);
 
  		list_for_each_entry(glist, &qg->groups, next_group)
#6 lore raw

Re: [PATCH 4/4] btrfs: swallow btrfs_record_squota_delta() enoent

In reply to Boris Burkov
Quoted 27 lines · Boris Burkov

I thought that it was likely I could harden squota deletion to the point that it was impossible to end up with an extent accounted to a qgroup outliving its qgroup. Several recent bugs have made me re-consider that position.

Ultimately, this is a tradeoff between short term stability and long term strictness, but I think given that there could be another layer of bugs behind the 2-3 I just fixed, I would feel much more confident in people using squotas if the risk was "your values can get a bit out of whack which you can fix by deleting stuff or disabling/re-enabling/repairing" vs "it will abort your filesystem".

As the final nail in the coffin, the Meta production kernel was lacking earlier fixes from me and Qu regarding subvol qgroup lifetime, so this is what we have been testing at scale, so I think at least for now upstream should have the same extra layer of protection.

---
  fs/btrfs/qgroup.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/qgroup.c +0 -0
index a93d49015fc7..f9e5005028dc 100644

fs/btrfs/qgroup.c int btrfs_record_squota_delta(struct btrfs_fs_info *fs_info, · line 4962

Quoted 5 lines
fs/btrfs/qgroup.c +1 -1
@@ -4959,7 +4959,7 @@ int btrfs_record_squota_delta(struct btrfs_fs_info *fs_info,
  	spin_lock(&fs_info->qgroup_lock);
  	qgroup = find_qgroup_rb(fs_info, root);
  	if (!qgroup) {
-		ret = -ENOENT;
+		ret = 0;

I appreciate that end users won't be bothered with transaction abort, but on the other hand, this seems to mask real bugs.

Can we add a WARN_ON_ONCE(IS_ENABLED(CONFIG_BTRFS_DEBUG))) so that we can catch such problems for debug builds?

Thanks,
Qu> goto out;

Quoted 2 lines
fs/btrfs/qgroup.c +0 -0
  	}
 
Reviewed-by on the v1 series
#7 lore raw
In reply to Boris Burkov
Quoted 9 lines · Boris Burkov

A recent report of read-only filesystems since enabling squotas:

https://lore.kernel.org/linux-btrfs/[email protected]/

led me to investigate a handful of issues in squotas and find some trivial and not so trivial bugs in their lifetime management. The aim of the series is to strengthen enforcement of the invariant: "a squota qgroup must outlast extents that are accounted to it".

Quoted 6 lines

This series consists of two fixes to squota lifetime bugs and two patches that reduce the consequences of errors in lifetime management from full fs goes read-only abort to "warning and inaccurate squotas which can be deleted eventually"

Additional improvements to btrfs check and fstests to follow.

Only minor debug related comments in the last two patches.

Otherwise looks good to me.

Thanks,
Qu

Quoted 11 lines

Boris Burkov (4):
btrfs: check for subvolume before deleting squota qgroup
btrfs: fix squota accounting during enable generation
btrfs: clamp to avoid squota underflow
btrfs: swallow btrfs_record_squota_delta() enoent

  fs/btrfs/fs.h     |  1 +
  fs/btrfs/qgroup.c | 99 ++++++++++++++++++++++++++++++++---------------
  2 files changed, 68 insertions(+), 32 deletions(-)
#8 lore raw
In reply to Boris Burkov
Quoted 14 lines · Boris Burkov

A recent report of read-only filesystems since enabling squotas:

https://lore.kernel.org/linux-btrfs/[email protected]/

led me to investigate a handful of issues in squotas and find some trivial and not so trivial bugs in their lifetime management. The aim of the series is to strengthen enforcement of the invariant: "a squota qgroup must outlast extents that are accounted to it".

This series consists of two fixes to squota lifetime bugs and two patches that reduce the consequences of errors in lifetime management from full fs goes read-only abort to "warning and inaccurate squotas which can be deleted eventually"

Quoted 7 lines

Additional improvements to btrfs check and fstests to follow.

Boris Burkov (4):
btrfs: check for subvolume before deleting squota qgroup
btrfs: fix squota accounting during enable generation
btrfs: clamp to avoid squota underflow
btrfs: swallow btrfs_record_squota_delta() enoent

Looks good to me, as much I can understand it. Please add it to for-next.