Skip to contentSkip to conversation

btrfs-progs: mkfs: Quota support through -Q|--quota

PATCH Applied Marcos Paulo de Souza 2 participants 13 messages Mar 18, 2020 – May 25, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v4 cover btrfs-progs: mkfs: Quota support through -Q|--quota

From: Marcos Paulo de Souza <[email protected]>

Hi guys,

This if the forth version of this patchset. The last submission of these patches was in 2018[1]. This version is based on top on the current devel branch, with minor cleanups, minor conflicts and only a real fix in patch 0008. I would like to ask you guys to review these patches, since v3 didn't receive any feedback at the time.

I only added my SoB in three patches, which were those were I needed a manual intervention, or a specific fix as I mentioned.

Thanks for your review,
Marcos

Original cover letter for Wenruo:
This patchset adds quota support, which means the result fs will have quota enabled by default, and its accounting is already consistent, no manually rescan or quota enable is needed.

The overall design of such support is:
1) Create needed tree
Both btrfs_root and real root item and tree root leaf. For this, a new infrastructure, btrfs_create_tree(), is added for this.

2) Fill quota root with basic skeleton
Only 3 items are really needed
a) global quota status item
b) quota info for specified qgroup
c) quota limit for specified qgroup

Currently we insert all qgroup items for all existing file trees. If we're going to support extra subvolume at mkfs time, just pass the subvolume id into insert_qgroup_items().

The content doesn't matter at all.

3) Repair qgroups using infrastructure from qgroup-verify In fact, qgroup repair is just offline rescan.
Although the original qgroup-verify infrastructure is mostly noisy, modify it a little to make it silent to function as offline quota rescan.

And such support is mainly designed for developers and QA guys.

As to enable quota, before we must normally mount the fs, enable quota (and rescan if needed).
This ioctl based procedure is not common, and fstests doesn't provide such support.
(Not to mention sometimes rescan itself can be buggy)

There are several attempts to make fstests to support it, but due to different reasons, all these attempts failed.

To make it easier to test all existing test cases with btrfs quota enabled, the current best method is to support quota at mkfs time, and here comes the patchset.

[1]: https://lore.kernel.org/linux-btrfs/[email protected]/T/#m107735cecbf4729b599e6e4eee0a54802909b30d

Qu Wenruo (11):
btrfs-progs: qgroup-verify: Avoid NULL pointer dereference for later
silent qgroup repair
btrfs-progs: qgroup-verify: Also repair qgroup status version
btrfs-progs: qgroup-verify: Use fs_info->readonly to check if we
should repair qgroups
btrfs-progs: qgroup-verify: Move qgroup classification out of
report_qgroups
btrfs-progs: qgroup-verify: Allow repair_qgroups function to do silent
repair
btrfs-progs: ctree: Introduce function to create an empty tree
btrfs-progs: mkfs: Introduce function to insert qgroup info and limit
items
btrfs-progs: mkfs: Introduce function to setup quota root and rescan
btrfs-progs: mkfs: Introduce mkfs time quota support
btrfs-progs: test/mkfs: Add test case for -Q|--quota option
btrfs-progs: test/mkfs: Add test case for --rootdir and --quota

 Documentation/mkfs.btrfs.asciidoc             |   5 +
 check/main.c                                  |  20 +--
 check/qgroup-verify.c                         |  93 ++++++++----
 check/qgroup-verify.h                         |   4 +-
 ctree.c                                       | 109 +++++++++++++-
 ctree.h                                       |   3 +
 mkfs/main.c                                   | 136 +++++++++++++++++-
 tests/mkfs-tests/001-basic-profiles/test.sh   |  10 ++
 .../mkfs-tests/018-rootdir-with-quota/test.sh |  51 +++++++
 9 files changed, 389 insertions(+), 42 deletions(-)
 create mode 100755 tests/mkfs-tests/018-rootdir-with-quota/test.sh
Signature · 1 line
2.25.0
#1 lore raw

PATCH v4 1/11 btrfs-progs: qgroup-verify: Avoid NULL pointer dereference for later silent qgroup repair

From: Qu Wenruo <[email protected]>

Commit 078e9a1cc973 ("btrfs-progs: check: enhanced progress indicator") introduced @qgroup_item_count for progress indicator.

However since we will later introduce silent qgroup rescan functionality, the @qgroup_item_count pointer can be NULL.

So check if @qgroup_item_count is NULL before accessing it.

---
 check/qgroup-verify.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
check/qgroup-verify.c +2 -1
index afe15acf..17c266d4 100644
@@ -743,7 +743,8 @@ static int travel_tree(struct btrfs_fs_info *info, struct btrfs_root *root,
 	 */
 	nr = btrfs_header_nritems(eb);
 	for (i = 0; i < nr; i++) {
-		(*qgroup_item_count)++;
+		if (qgroup_item_count)
+			(*qgroup_item_count)++;
 		new_bytenr = btrfs_node_blockptr(eb, i);
 		new_num_bytes = info->nodesize;
 
Signature · 1 line
2.25.0
#2 lore raw

PATCH v4 2/11 btrfs-progs: qgroup-verify: Also repair qgroup status version

In reply to Marcos Paulo de Souza

From: Qu Wenruo <[email protected]>

Current kernel only supports qgroup version 1.
Make qgroup-verify to follow this standard.

---
 check/qgroup-verify.c | 2 ++
 ctree.h               | 1 +
 2 files changed, 3 insertions(+)
check/qgroup-verify.c +2 -0
index 17c266d4..0509ecec 100644
@@ -1598,6 +1598,8 @@ static int repair_qgroup_status(struct btrfs_fs_info *info)
 	btrfs_set_qgroup_status_rescan(path.nodes[0], status_item, 0);
 	btrfs_set_qgroup_status_generation(path.nodes[0], status_item,
 					   trans->transid);
+	btrfs_set_qgroup_status_version(path.nodes[0], status_item,
+					BTRFS_QGROUP_STATUS_VERSION);
 
 	btrfs_mark_buffer_dirty(path.nodes[0]);
 
ctree.h +1 -0
index 36f62732..083bde3c 100644
@@ -1034,6 +1034,7 @@ struct btrfs_qgroup_status_item {
 	__le64 rescan;		/* progress during scanning */
 } __attribute__ ((__packed__));
 
+#define BTRFS_QGROUP_STATUS_VERSION		1
 struct btrfs_block_group_item {
 	__le64 used;
 	__le64 chunk_objectid;
Signature · 1 line
2.25.0
#3 lore raw

PATCH v4 3/11 btrfs-progs: qgroup-verify: Use fs_info->readonly to check if we should repair qgroups

In reply to Marcos Paulo de Souza

From: Qu Wenruo <[email protected]>

In fact qgroup-verify is just kind of offline qgroup rescan, and later mkfs qgroup support will reuse it.

So qgroup-verify doesn't really need to rely the global variable @repair to check if it should repair qgroups.

Instead check fs_info->readonly to do the repair.

---
 check/qgroup-verify.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
check/qgroup-verify.c +1 -1
index 0509ecec..b7b63095 100644
@@ -1617,7 +1617,7 @@ int repair_qgroups(struct btrfs_fs_info *info, int *repaired)
 
 	*repaired = 0;
 
-	if (!repair)
+	if (info->readonly)
 		return 0;
 
 	list_for_each_entry_safe(count, tmpcount, &bad_qgroups, bad_list) {
Signature · 1 line
2.25.0
#4 lore raw

PATCH v4 4/11 btrfs-progs: qgroup-verify: Move qgroup classification out of report_qgroups

In reply to Marcos Paulo de Souza

From: Qu Wenruo <[email protected]>

The original qgroup-verify integrates qgroup classification into report_qgroups().
This behavior makes silent qgroup repair (or offline rescan) impossible.

To repair qgroup, we must call report_qgroups() to trigger bad qgroup classification, which will output error message.

This patch moves bad qgroup classification from report_qgroups() to qgroup_verify_all().
Now report_qgroups() is pretty lightweight, only doing basic qgroup difference report thus change it type to void.

And since the functionality of qgroup_verify_all() changes, change callers to handle the new return value correctly.

[ removed some comments ]
---
 check/main.c          | 18 ++++++------
 check/qgroup-verify.c | 67 +++++++++++++++++++++++++++++++------------
 check/qgroup-verify.h |  2 +-
 3 files changed, 59 insertions(+), 28 deletions(-)
check/main.c +9 -9
index 49bdbfec..7632b60a 100644
@@ -9941,7 +9941,7 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)
 	int clear_space_cache = 0;
 	int qgroup_report = 0;
 	int qgroups_repaired = 0;
-	int qgroup_report_ret;
+	int qgroup_verify_ret;
 	unsigned ctree_flags = OPEN_CTREE_EXCLUSIVE;
 	int force = 0;
 
@@ -10198,8 +10198,8 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)
 		       uuidbuf);
 		ret = qgroup_verify_all(info);
 		err |= !!ret;
-		if (ret == 0)
-			err |= !!report_qgroups(1);
+		if (ret >= 0)
+			report_qgroups(1);
 		goto close_out;
 	}
 	if (subvolid) {
@@ -10433,21 +10433,21 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)
 			ctx.tp = TASK_QGROUPS;
 			task_start(ctx.info, &ctx.start_time, &ctx.item_count);
 		}
-		ret = qgroup_verify_all(info);
+		qgroup_verify_ret = qgroup_verify_all(info);
 		task_stop(ctx.info);
-		err |= !!ret;
-		if (ret) {
+		if (qgroup_verify_ret < 0) {
 			error("failed to check quota groups");
+			err |= !!qgroup_verify_ret;
 			goto out;
 		}
-		qgroup_report_ret = report_qgroups(0);
+		report_qgroups(0);
 		ret = repair_qgroups(info, &qgroups_repaired);
 		if (ret) {
 			error("failed to repair quota groups");
 			goto out;
 		}
-		if (qgroup_report_ret && (!qgroups_repaired || ret))
-			err |= qgroup_report_ret;
+		if (qgroup_verify_ret && (!qgroups_repaired || ret))
+			err |= !!qgroup_verify_ret;
 		ret = 0;
 	} else {
 		fprintf(stderr,
check/qgroup-verify.c +49 -18

128 lines not on this page · raw message

check/qgroup-verify.h +1 -1
index 20e93708..6495dd18 100644
@@ -23,7 +23,7 @@
 #include "ctree.h"
 
 int qgroup_verify_all(struct btrfs_fs_info *info);
-int report_qgroups(int all);
+void report_qgroups(int all);
 int repair_qgroups(struct btrfs_fs_info *info, int *repaired);
 
 int print_extent_state(struct btrfs_fs_info *info, u64 subvol);
Signature · 1 line
2.25.0
#5 lore raw

PATCH v4 5/11 btrfs-progs: qgroup-verify: Allow repair_qgroups function to do silent repair

In reply to Marcos Paulo de Souza

From: Qu Wenruo <[email protected]>

Allow repair_qgroups() to do silent repair, so it can acts as offline qgroup rescan.

This provides the basis for later mkfs quota support.

---
 check/main.c          |  2 +-
 check/qgroup-verify.c | 19 +++++++++++--------
 check/qgroup-verify.h |  2 +-
 3 files changed, 13 insertions(+), 10 deletions(-)
check/main.c +1 -1
index 7632b60a..375aef8f 100644
@@ -10441,7 +10441,7 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)
 			goto out;
 		}
 		report_qgroups(0);
-		ret = repair_qgroups(info, &qgroups_repaired);
+		ret = repair_qgroups(info, &qgroups_repaired, false);
 		if (ret) {
 			error("failed to repair quota groups");
 			goto out;
check/qgroup-verify.c +11 -8
index b1e6b26c..b1736aab 100644
@@ -1540,7 +1540,7 @@ out:
 }
 
 static int repair_qgroup_info(struct btrfs_fs_info *info,
-			      struct qgroup_count *count)
+			      struct qgroup_count *count, bool silent)
 {
 	int ret;
 	struct btrfs_root *root = info->quota_root;
@@ -1549,8 +1549,10 @@ static int repair_qgroup_info(struct btrfs_fs_info *info,
 	struct btrfs_qgroup_info_item *info_item;
 	struct btrfs_key key;
 
-	printf("Repair qgroup %llu/%llu\n", btrfs_qgroup_level(count->qgroupid),
-	       btrfs_qgroup_subvid(count->qgroupid));
+	if (!silent)
+		printf("Repair qgroup %llu/%llu\n",
+			btrfs_qgroup_level(count->qgroupid),
+			btrfs_qgroup_subvid(count->qgroupid));
 
 	trans = btrfs_start_transaction(root, 1);
 	if (IS_ERR(trans))
@@ -1595,7 +1597,7 @@ out:
 	return ret;
 }
 
-static int repair_qgroup_status(struct btrfs_fs_info *info)
+static int repair_qgroup_status(struct btrfs_fs_info *info, bool silent)
 {
 	int ret;
 	struct btrfs_root *root = info->quota_root;
@@ -1604,7 +1606,8 @@ static int repair_qgroup_status(struct btrfs_fs_info *info)
 	struct btrfs_key key;
 	struct btrfs_qgroup_status_item *status_item;
 
-	printf("Repair qgroup status item\n");
+	if (!silent)
+		printf("Repair qgroup status item\n");
 
 	trans = btrfs_start_transaction(root, 1);
 	if (IS_ERR(trans))
@@ -1641,7 +1644,7 @@ out:
 	return ret;
 }
 
-int repair_qgroups(struct btrfs_fs_info *info, int *repaired)
+int repair_qgroups(struct btrfs_fs_info *info, int *repaired, bool silent)
 {
 	int ret = 0;
 	struct qgroup_count *count, *tmpcount;
@@ -1652,7 +1655,7 @@ int repair_qgroups(struct btrfs_fs_info *info, int *repaired)
 		return 0;
 
 	list_for_each_entry_safe(count, tmpcount, &bad_qgroups, bad_list) {
-		ret = repair_qgroup_info(info, count);
+		ret = repair_qgroup_info(info, count, silent);
 		if (ret) {
 			goto out;
 		}
@@ -1668,7 +1671,7 @@ int repair_qgroups(struct btrfs_fs_info *info, int *repaired)
 	 * mount.
 	 */
 	if (*repaired || counts.qgroup_inconsist || counts.rescan_running) {
-		ret = repair_qgroup_status(info);
+		ret = repair_qgroup_status(info, silent);
 		if (ret)
 			goto out;
 
check/qgroup-verify.h +1 -1
index 6495dd18..8a8694b6 100644
@@ -24,7 +24,7 @@
 
 int qgroup_verify_all(struct btrfs_fs_info *info);
 void report_qgroups(int all);
-int repair_qgroups(struct btrfs_fs_info *info, int *repaired);
+int repair_qgroups(struct btrfs_fs_info *info, int *repaired, bool silent);
 
 int print_extent_state(struct btrfs_fs_info *info, u64 subvol);
 
Signature · 1 line
2.25.0
#6 lore raw

PATCH v4 6/11 btrfs-progs: ctree: Introduce function to create an empty tree

In reply to Marcos Paulo de Souza

From: Qu Wenruo <[email protected]>

Introduce a new function, btrfs_create_tree(), to create an empty tree.

Currently, there is only one caller to create new tree, namely data reloc tree in mkfs.
However it's copying fs tree to create a new root.

This copy fs tree method is not a good idea if we only need an empty tree.

So here introduce a new function, btrfs_create_tree() to create new tree.
Which will handle the following things:
1) New tree root leaf
Using generic tree allocation

2) New root item in tree root

3) Modify special tree root pointers in fs_info
Only quota_root is supported yet, but can be expended easily

This patch provides the basis to implement quota support in mkfs.

[ solved minor conflicts ]
---
 ctree.c | 109 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 ctree.h |   2 ++
 2 files changed, 110 insertions(+), 1 deletion(-)
ctree.c +108 -1
index 97b44d48..49598a45 100644
@@ -21,8 +21,9 @@
 #include "print-tree.h"
 #include "repair.h"
 #include "common/internal.h"
-#include "kernel-lib/sizes.h"
 #include "common/messages.h"
+#include "common/utils.h"
+#include "kernel-lib/sizes.h"
 #include "volumes.h"
 
 static int split_node(struct btrfs_trans_handle *trans, struct btrfs_root
@@ -182,6 +183,112 @@ int btrfs_copy_root(struct btrfs_trans_handle *trans,
 	return 0;
 }
 
+/*
+ * Create a new tree root, with root objectid set to @objectid.
+ *
+ * NOTE: Doesn't support tree with non-zero offset, like data reloc tree.
+ */
+int btrfs_create_root(struct btrfs_trans_handle *trans,
+		      struct btrfs_fs_info *fs_info, u64 objectid)
+{
+	struct extent_buffer *node;
+	struct btrfs_root *new_root;
+	struct btrfs_disk_key disk_key;
+	struct btrfs_key location;
+	struct btrfs_root_item root_item = { 0 };
+	int ret;
+
+	new_root = malloc(sizeof(*new_root));
+	if (!new_root)
+		return -ENOMEM;
+
+	btrfs_setup_root(new_root, fs_info, objectid);
+	if (!is_fstree(objectid))
+		new_root->track_dirty = 1;
+	add_root_to_dirty_list(new_root);
+
+	new_root->objectid = objectid;
+	new_root->root_key.objectid = objectid;
+	new_root->root_key.type = BTRFS_ROOT_ITEM_KEY;
+	new_root->root_key.offset = 0;
+
+	node = btrfs_alloc_free_block(trans, new_root, fs_info->nodesize,
+				      objectid, &disk_key, 0, 0, 0);
+	if (IS_ERR(node)) {
+		ret = PTR_ERR(node);
+		error("failed to create root node for tree %llu: %d (%s)",
+		      objectid, ret, strerror(-ret));
+		return ret;
+	}
+	new_root->node = node;
+
+	btrfs_set_header_generation(node, trans->transid);
+	btrfs_set_header_backref_rev(node, BTRFS_MIXED_BACKREF_REV);
+	btrfs_clear_header_flag(node, BTRFS_HEADER_FLAG_RELOC |
+				      BTRFS_HEADER_FLAG_WRITTEN);
+	btrfs_set_header_owner(node, objectid);
+	btrfs_set_header_nritems(node, 0);
+	btrfs_set_header_level(node, 0);
+	ret = btrfs_inc_ref(trans, new_root, node, 0);
+	if (ret < 0)
+		goto free;
+
+	/*
+	 * Special tree roots may need to modify pointers in @fs_info
+	 * Only quota is supported yet.
+	 */
+	switch (objectid) {
+	case BTRFS_QUOTA_TREE_OBJECTID:
+		if (fs_info->quota_root) {
+			error("quota root already exists");
+			ret = -EEXIST;
+			goto free;
+		}
+		fs_info->quota_root = new_root;
+		fs_info->quota_enabled = 1;
+		break;
+	/*
+	 * Essential trees can't be created by this function, yet.
+	 * As we expect such skeleton exists, or a lot of functions like
+	 * btrfs_alloc_free_block() doesn't work at all
+	 */
+	case BTRFS_ROOT_TREE_OBJECTID:
+	case BTRFS_EXTENT_TREE_OBJECTID:
+	case BTRFS_CHUNK_TREE_OBJECTID:
+	case BTRFS_FS_TREE_OBJECTID:
+		ret = -EEXIST;
+		goto free;
+	default:
+		/* Subvolume trees don't need special handles */
+		if (is_fstree(objectid))
+			break;
+		/* Other special trees are not supported yet */
+		ret = -ENOTTY;
+		goto free;
+	}
+	btrfs_mark_buffer_dirty(node);
+	btrfs_set_root_bytenr(&root_item, btrfs_header_bytenr(node));
+	btrfs_set_root_level(&root_item, 0);
+	btrfs_set_root_generation(&root_item, trans->transid);
+	btrfs_set_root_dirid(&root_item, 0);
+	btrfs_set_root_refs(&root_item, 1);
+	btrfs_set_root_used(&root_item, fs_info->nodesize);
+	location.objectid = objectid;
+	location.type = BTRFS_ROOT_ITEM_KEY;
+	location.offset = 0;
+
+	ret = btrfs_insert_root(trans, fs_info->tree_root, &location,
+				&root_item);
+	if (ret < 0)
+		goto free;
+	return ret;
+
+free:
+	free_extent_buffer(node);
+	free(new_root);
+	return ret;
+}
+
 /*
  * check if the tree block can be shared by multiple trees
  */
ctree.h +2 -0
index 083bde3c..41565b52 100644
@@ -2641,6 +2641,8 @@ int btrfs_copy_root(struct btrfs_trans_handle *trans,
 		      struct btrfs_root *root,
 		      struct extent_buffer *buf,
 		      struct extent_buffer **cow_ret, u64 new_root_objectid);
+int btrfs_create_root(struct btrfs_trans_handle *trans,
+		      struct btrfs_fs_info *fs_info, u64 objectid);
 int btrfs_extend_item(struct btrfs_root *root, struct btrfs_path *path,
 		u32 data_size);
 int btrfs_truncate_item(struct btrfs_root *root, struct btrfs_path *path,
Signature · 1 line
2.25.0
#7 lore raw

PATCH v4 7/11 btrfs-progs: mkfs: Introduce function to insert qgroup info and limit items

In reply to Marcos Paulo de Souza

From: Qu Wenruo <[email protected]>

Introduce a new function, insert_qgroup_items(), to insert qgroup info item and qgroup limit item for later mkfs qgroup support.

---
 mkfs/main.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
mkfs/main.c +34 -0
index 316ea82e..6d9b3265 100644
@@ -798,6 +798,40 @@ out:
 	return ret;
 }
 
+static int insert_qgroup_items(struct btrfs_trans_handle *trans,
+			       struct btrfs_fs_info *fs_info,
+			       u64 qgroupid)
+{
+	struct btrfs_path path;
+	struct btrfs_root *quota_root = fs_info->quota_root;
+	struct btrfs_key key;
+	int ret;
+
+	if (qgroupid >> BTRFS_QGROUP_LEVEL_SHIFT) {
+		error("qgroup level other than 0 is not supported yet");
+		return -ENOTTY;
+	}
+
+	key.objectid = 0;
+	key.type = BTRFS_QGROUP_INFO_KEY;
+	key.offset = qgroupid;
+
+	btrfs_init_path(&path);
+	ret = btrfs_insert_empty_item(trans, quota_root, &path, &key,
+				      sizeof(struct btrfs_qgroup_info_item));
+	btrfs_release_path(&path);
+	if (ret < 0)
+		return ret;
+
+	key.objectid = 0;
+	key.type = BTRFS_QGROUP_LIMIT_KEY;
+	key.offset = qgroupid;
+	ret = btrfs_insert_empty_item(trans, quota_root, &path, &key,
+				      sizeof(struct btrfs_qgroup_limit_item));
+	btrfs_release_path(&path);
+	return ret;
+}
+
 int BOX_MAIN(mkfs)(int argc, char **argv)
 {
 	char *file;
Signature · 1 line
2.25.0
#8 lore raw

PATCH v4 8/11 btrfs-progs: mkfs: Introduce function to setup quota root and rescan

In reply to Marcos Paulo de Souza

From: Qu Wenruo <[email protected]>

Introduce a new function, setup_quota_root(), which will create quota root, and do an offline rescan to ensure all quota accounting numbers are correct.

[ minor improvement in the fail path ]
---
 mkfs/main.c | 86 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 86 insertions(+)
mkfs/main.c +86 -0
index 6d9b3265..1fb25a9d 100644
@@ -48,6 +48,7 @@
 #include "crypto/crc32c.h"
 #include "common/fsfeatures.h"
 #include "common/box.h"
+#include "check/qgroup-verify.h"
 
 static int verbose = 1;
 
@@ -832,6 +833,91 @@ static int insert_qgroup_items(struct btrfs_trans_handle *trans,
 	return ret;
 }
 
+static int setup_quota_root(struct btrfs_fs_info *fs_info)
+{
+	struct btrfs_trans_handle *trans;
+	struct btrfs_qgroup_status_item *qsi;
+	struct btrfs_root *quota_root;
+	struct btrfs_path path;
+	struct btrfs_key key;
+	int qgroup_repaired = 0;
+	int ret;
+
+	/* One to modify tree root, one for quota root */
+	trans = btrfs_start_transaction(fs_info->tree_root, 2);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		error("failed to start transaction: %d (%s)",
+			ret, strerror(-ret));
+		return ret;
+	}
+	ret = btrfs_create_root(trans, fs_info, BTRFS_QUOTA_TREE_OBJECTID);
+	if (ret < 0) {
+		error("failed to create quota root: %d (%s)",
+			ret, strerror(-ret));
+		goto fail;
+	}
+	quota_root = fs_info->quota_root;
+
+	key.objectid = 0;
+	key.type = BTRFS_QGROUP_STATUS_KEY;
+	key.offset = 0;
+
+	btrfs_init_path(&path);
+	ret = btrfs_insert_empty_item(trans, quota_root, &path, &key,
+				      sizeof(*qsi));
+	if (ret < 0) {
+		error("failed to insert qgroup status item: %d (%s)",
+			ret, strerror(-ret));
+		goto fail;
+	}
+
+	qsi = btrfs_item_ptr(path.nodes[0], path.slots[0],
+			     struct btrfs_qgroup_status_item);
+	btrfs_set_qgroup_status_generation(path.nodes[0], qsi, 0);
+	btrfs_set_qgroup_status_rescan(path.nodes[0], qsi, 0);
+
+	/* Mark current status info inconsistent, and fix it later */
+	btrfs_set_qgroup_status_flags(path.nodes[0], qsi,
+			BTRFS_QGROUP_STATUS_FLAG_ON |
+			BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT);
+	btrfs_release_path(&path);
+
+	/* Currently mkfs will only create one subvolume */
+	ret = insert_qgroup_items(trans, fs_info, BTRFS_FS_TREE_OBJECTID);
+	if (ret < 0) {
+		error("failed to insert qgroup items: %d (%s)",
+			ret, strerror(-ret));
+		goto fail;
+	}
+
+	ret = btrfs_commit_transaction(trans, fs_info->tree_root);
+	if (ret < 0) {
+		error("failed to commit current transaction: %d (%s)",
+			ret, strerror(-ret));
+		return ret;
+	}
+
+	/*
+	 * Qgroup is setup but with wrong info, use qgroup-verify
+	 * infrastructure to repair them.
+	 * (Just acts as offline rescan)
+	 */
+	ret = qgroup_verify_all(fs_info);
+	if (ret < 0) {
+		error("qgroup rescan failed: %d (%s)", ret, strerror(-ret));
+		return ret;
+	}
+	ret = repair_qgroups(fs_info, &qgroup_repaired, true);
+	if (ret < 0)
+		error("failed to fill qgroup info: %d (%s)", ret,
+			strerror(-ret));
+	return ret;
+fail:
+	btrfs_abort_transaction(trans, ret);
+	return ret;
+}
+
 int BOX_MAIN(mkfs)(int argc, char **argv)
 {
 	char *file;
Signature · 1 line
2.25.0
#9 lore raw

PATCH v4 9/11 btrfs-progs: mkfs: Introduce mkfs time quota support

In reply to Marcos Paulo de Souza

From: Qu Wenruo <[email protected]>

The result fs will has quota enabled, with consistent qgroup accounting.

This is quite handy to test quota with fstests, which doesn't support to call ioctl for btrfs at mount time.

---
 Documentation/mkfs.btrfs.asciidoc |  5 +++++
 mkfs/main.c                       | 16 +++++++++++++++-
 2 files changed, 20 insertions(+), 1 deletion(-)
Documentation/mkfs.btrfs.asciidoc +5 -0
index 0502d1d8..611ea9c2 100644
@@ -87,6 +87,11 @@ updating the metadata blocks.
 +
 NOTE: versions up to 3.11 set the nodesize to 4k.
 
+*-Q|--quota*::
+Enable btrfs quota support. Result filesystem will have quota enabled and all
+qgroup accounting correct.
+See also `btrfs-quota`(8).
+
 *-s|--sectorsize <size>*::
 Specify the sectorsize, the minimum data block allocation unit.
 +
mkfs/main.c +15 -1
index 1fb25a9d..70a66cef 100644
@@ -949,6 +949,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 	char *source_dir = NULL;
 	bool source_dir_set = false;
 	bool shrink_rootdir = false;
+	bool enable_quota = false;
 	u64 source_dir_size = 0;
 	u64 min_dev_size;
 	u64 shrink_size;
@@ -985,13 +986,14 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 			{ "nodiscard", no_argument, NULL, 'K' },
 			{ "features", required_argument, NULL, 'O' },
 			{ "uuid", required_argument, NULL, 'U' },
+			{ "quota", required_argument, NULL, 'Q' },
 			{ "quiet", 0, NULL, 'q' },
 			{ "shrink", no_argument, NULL, GETOPT_VAL_SHRINK },
 			{ "help", no_argument, NULL, GETOPT_VAL_HELP },
 			{ NULL, 0, NULL, 0}
 		};
 
-		c = getopt_long(argc, argv, "A:b:fl:n:s:m:d:L:O:r:U:VMKq",
+		c = getopt_long(argc, argv, "A:b:fl:n:s:m:d:L:O:r:U:VMKqQ",
 				long_options, NULL);
 		if (c < 0)
 			break;
@@ -1066,6 +1068,9 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 			case 'q':
 				verbose = 0;
 				break;
+			case 'Q':
+				enable_quota = true;
+				break;
 			case GETOPT_VAL_SHRINK:
 				shrink_rootdir = true;
 				break;
@@ -1470,6 +1475,15 @@ raid_groups:
 		}
 	}
 
+	if (enable_quota) {
+		ret = setup_quota_root(fs_info);
+		if (ret < 0) {
+			error("failed to initialize quota: %d (%s)", ret,
+				strerror(-ret));
+			goto out;
+		}
+	}
+
 	if (verbose) {
 		char features_buf[64];
 
Signature · 1 line
2.25.0
#10 lore raw

PATCH v4 10/11 btrfs-progs: test/mkfs: Add test case for -Q|--quota option

In reply to Marcos Paulo de Souza

From: Qu Wenruo <[email protected]>

Only test if btrfs check (which will check qgroup by default) and kernel mount success.

Comprehensive qgroup test cases still belongs to fstests.

---
 tests/mkfs-tests/001-basic-profiles/test.sh | 10 ++++++++++
 1 file changed, 10 insertions(+)
tests/mkfs-tests/001-basic-profiles/test.sh +10 -0
index 4b6c2f42..c273f1df 100755
@@ -47,6 +47,16 @@ test_mkfs_single  -d  dup     -m  single
 test_mkfs_single  -d  dup     -m  dup
 test_mkfs_single  -d  dup     -m  dup     --mixed
 
+test_mkfs_single -Q
+test_mkfs_single -Q -d  single  -m  single
+test_mkfs_single -Q -d  single  -m  single  --mixed
+test_mkfs_single -Q -d  single  -m  dup
+test_mkfs_single -Q -d  dup     -m  single
+test_mkfs_single -Q -d  dup     -m  dup
+test_mkfs_single -Q -d  dup     -m  dup     --mixed
+
+# Profile doesn't really affect quota, skip them to save some time
+
 test_mkfs_multi
 test_mkfs_multi   -d  single  -m  single
 test_mkfs_multi   -d  single  -m  single  --mixed
Signature · 1 line
2.25.0
#11 lore raw

PATCH v4 11/11 btrfs-progs: test/mkfs: Add test case for --rootdir and --quota

In reply to Marcos Paulo de Souza

From: Qu Wenruo <[email protected]>

Nothing interesting, since such combination can be handled easily by qgroup-verify.

---
 .../mkfs-tests/018-rootdir-with-quota/test.sh | 51 +++++++++++++++++++
 1 file changed, 51 insertions(+)
 create mode 100755 tests/mkfs-tests/018-rootdir-with-quota/test.sh
tests/mkfs-tests/018-rootdir-with-quota/test.sh new file +51 -0
new file mode 100755
index 00000000..97e5f592
@@ -0,0 +1,51 @@
+#!/bin/bash
+# Check if runtime feature quota can handle rootdir
+#
+
+source "$TOP/tests/common"
+
+check_prereq mkfs.btrfs
+check_prereq btrfs
+
+setup_root_helper		# For mknod
+prepare_test_dev 1G		# make it large since we will fill the fs
+
+# mknod can create FIFO/CHAR/BLOCK file but not SOCK.
+# No neat tool to create socket file, unless using python or similar.
+# So no SOCK is tested here
+check_global_prereq mknod
+check_global_prereq dd
+
+tmp=$(mktemp -d --tmpdir btrfs-progs-mkfs.rootdirXXXXXXX)
+
+run_check mkdir "$tmp/dir"
+run_check mkdir -p "$tmp/dir/in/dir"
+
+# More dir, there is no good way to pump metadata since we have no trigger
+# to enable/disable inline extent data, so here create enough dirs to bump
+# metadata
+run_check mkdir "$tmp/a_lot_of_dirs"
+for i in $(seq -w 0 8192); do
+	run_check mkdir "$tmp/a_lot_of_dirs/dir_$i"
+done
+
+# Then some data
+run_check dd if=/dev/zero bs=1M count=1 of="$tmp/1M"
+run_check dd if=/dev/zero bs=2M count=1 of="$tmp/2M"
+run_check dd if=/dev/zero bs=4M count=1 of="$tmp/4M"
+run_check dd if=/dev/zero bs=8M count=1 of="$tmp/8M"
+
+run_check dd if=/dev/zero bs=1K count=1 of="$tmp/1K"
+run_check dd if=/dev/zero bs=2K count=1 of="$tmp/2K"
+run_check dd if=/dev/zero bs=4K count=1 of="$tmp/4K"
+run_check dd if=/dev/zero bs=8K count=1 of="$tmp/8K"
+
+run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f -r "$tmp" -Q "$TEST_DEV"
+
+rm -rf -- "$tmp"
+
+# Normal check already includes quota check
+run_check $SUDO_HELPER "$TOP/btrfs" check "$TEST_DEV"
+
+# Just in case
+run_check $SUDO_HELPER "$TOP/btrfs" check --qgroup-report "$TEST_DEV"
Signature · 1 line
2.25.0
Applied
#12 lore raw
In reply to Marcos Paulo de Souza
Quoted 62 lines · Marcos Paulo de Souza

From: Marcos Paulo de Souza <[email protected]>

Hi guys,

This if the forth version of this patchset. The last submission of these patches was in 2018[1]. This version is based on top on the current devel branch, with minor cleanups, minor conflicts and only a real fix in patch 0008. I would like to ask you guys to review these patches, since v3 didn't receive any feedback at the time.

I only added my SoB in three patches, which were those were I needed a manual intervention, or a specific fix as I mentioned.

Thanks for your review,
Marcos

Original cover letter for Wenruo:
This patchset adds quota support, which means the result fs will have quota enabled by default, and its accounting is already consistent, no manually rescan or quota enable is needed.

The overall design of such support is:
1) Create needed tree
Both btrfs_root and real root item and tree root leaf. For this, a new infrastructure, btrfs_create_tree(), is added for this.

2) Fill quota root with basic skeleton
Only 3 items are really needed
a) global quota status item
b) quota info for specified qgroup
c) quota limit for specified qgroup

Currently we insert all qgroup items for all existing file trees. If we're going to support extra subvolume at mkfs time, just pass the subvolume id into insert_qgroup_items().

The content doesn't matter at all.

3) Repair qgroups using infrastructure from qgroup-verify In fact, qgroup repair is just offline rescan.
Although the original qgroup-verify infrastructure is mostly noisy, modify it a little to make it silent to function as offline quota rescan.

And such support is mainly designed for developers and QA guys.

As to enable quota, before we must normally mount the fs, enable quota (and rescan if needed).
This ioctl based procedure is not common, and fstests doesn't provide such support.
(Not to mention sometimes rescan itself can be buggy)

There are several attempts to make fstests to support it, but due to different reasons, all these attempts failed.

To make it easier to test all existing test cases with btrfs quota enabled, the current best method is to support quota at mkfs time, and here comes the patchset.

[1]: https://lore.kernel.org/linux-btrfs/[email protected]/T/#m107735cecbf4729b599e6e4eee0a54802909b30d

Quoted 14 lines

Qu Wenruo (11):
btrfs-progs: qgroup-verify: Avoid NULL pointer dereference for later
silent qgroup repair
btrfs-progs: qgroup-verify: Also repair qgroup status version
btrfs-progs: qgroup-verify: Use fs_info->readonly to check if we
should repair qgroups
btrfs-progs: qgroup-verify: Move qgroup classification out of
report_qgroups
btrfs-progs: qgroup-verify: Allow repair_qgroups function to do silent
repair
btrfs-progs: ctree: Introduce function to create an empty tree
btrfs-progs: mkfs: Introduce function to insert qgroup info and limit
items
btrfs-progs: mkfs: Introduce function to setup quota root and rescan

I've applied the above with some fixes to devel.

Quoted 3 lines

btrfs-progs: mkfs: Introduce mkfs time quota support
btrfs-progs: test/mkfs: Add test case for -Q|--quota option
btrfs-progs: test/mkfs: Add test case for --rootdir and --quota

The option name needs to be -R as used to be in V2 of Qu's original patchset. I don't know why this got changed to the single purpose -Q but -R will be used to specifiy runtime options, similar to what -O does now.