mkfs: Quota support

The original qgroup-verify integrates qgroup classification into report_qgroups().
This behavior makes silent qgroup repair (or offline rescan) impossible.

To repair qgroup, we must call report_qgroups() to trigger bad qgroup classification, which will output error message.

This patch moves bad qgroup classification from report_qgroups() to qgroup_verify_all().
The functionality is not changed, but provides the basis for later silent qgroup repair.

Signed-off-by Qu Wenruo <[email protected]>

---
qgroup-verify.c | 28 ++++++++++++++++++++++++++--

1 file changed, 26 insertions(+), 2 deletions(-)

qgroup-verify.c +1 -2

index f509cdb9317a..67bd8b96ddd3 100644
@@ -1319,8 +1319,7 @@ void report_qgroups(int all)
 	while (node) {
 		c = rb_entry(node, struct qgroup_count, rb_node);
 
-		if (report_qgroup_difference(c, all))
-			list_add_tail(&c->bad_list, &bad_qgroups);
+		report_qgroup_difference(c, all);
 
 		node = rb_next(node);
 	}

@@ -1358,9 +1357,20 @@ void free_qgroup_counts(void)
 	}
 }
 

qgroup-verify.c +15 -0

 int qgroup_verify_all(struct btrfs_fs_info *info)
 {
 	int ret;
+	struct rb_node *node;
 
 	if (!info->quota_enabled)
 		return 0;

@@ -1395,6 +1405,20 @@ int qgroup_verify_all(struct btrfs_fs_info *info)
 
 	ret = account_all_refs(1, 0);
 
+	/*
+	 * Do the correctness check here, so for callers who don't want
+	 * verbose report can skip calling report_qgroups()
+	 */
+	node = rb_first(&counts.root);
+	while (node) {
+		struct qgroup_count *c;
+
+		c = rb_entry(node, struct qgroup_count, rb_node);
+		if (is_bad_qgroup(c))
+			list_add_tail(&c->bad_list, &bad_qgroups);
+		node = rb_next(node);
+	}
+

out:

	/*
	 * Don't free the qgroup count records as they will be walked

Signature · 6 lines

2.14.3

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Introduce a new function, btrfs_create_tree(), to create an empty tree.

Currently, there is only one caller to create new tree, namely data reloc tree in mkfs.
However it's copying fs tree to create a new root.

This copy fs tree method is not a good idea if we only need an empty tree.

So here introduce a new function, btrfs_create_tree() to create new tree.
Which will handle the following things:
1) New tree root leaf
Using generic tree allocation

2) New root item in tree root

3) Modify special tree root pointers in fs_info
Only quota_root is supported yet, but can be expended easily

This patch provides the basis to implement quota support in mkfs.

Signed-off-by Qu Wenruo <[email protected]>

---
  ctree.c | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  ctree.h |   2 ++
  2 files changed, 111 insertions(+)

ctree.c +14 -0

index 4fc33b14000a..c707be58c413 100644
@@ -22,6 +22,7 @@
  #include "repair.h"
  #include "internal.h"
  #include "sizes.h"
+#include "utils.h"
 
  static int split_node(struct btrfs_trans_handle *trans, struct btrfs_root
  		      *root, struct btrfs_path *path, int level);

@@ -136,6 +137,114 @@ int btrfs_copy_root(struct btrfs_trans_handle *trans,
  	return 0;
  }
 
+/*
+ * Create a new tree root, with root objectid set to @objectid.
+ *
+ * NOTE: Doesn't support tree with non-zero offset, like tree reloc tree.
+ */
+int btrfs_create_root(struct btrfs_trans_handle *trans,
+		      struct btrfs_fs_info *fs_info, u64 objectid)
+{
+	struct extent_buffer *node;
+	struct btrfs_root *new_root;
+	struct btrfs_disk_key disk_key;
+	struct btrfs_key location;
+	struct btrfs_root_item root_item = { 0 };

ctree.c +82 -0

+	new_root->root_key.type = BTRFS_ROOT_ITEM_KEY;
+	new_root->root_key.offset = 0;
+
+	node = btrfs_alloc_free_block(trans, new_root, fs_info->nodesize,
+				      objectid, &disk_key, 0, 0, 0);
+	if (IS_ERR(node)) {
+		ret = PTR_ERR(node);
+		error("failed to create root node for tree %llu: %d (%s)",
+		      objectid, ret, strerror(-ret));
+		return ret;
+	}
+	new_root->node = node;
+
+	btrfs_set_header_generation(node, trans->transid);
+	btrfs_set_header_backref_rev(node, BTRFS_MIXED_BACKREF_REV);
+	btrfs_clear_header_flag(node, BTRFS_HEADER_FLAG_RELOC |
+				      BTRFS_HEADER_FLAG_WRITTEN);
+	btrfs_set_header_owner(node, objectid);
+	btrfs_set_header_nritems(node, 0);
+	btrfs_set_header_level(node, 0);
+	write_extent_buffer(node, fs_info->fsid, btrfs_header_fsid(),
+			    BTRFS_FSID_SIZE);
+	ret = btrfs_inc_ref(trans, new_root, node, 0);
+	if (ret < 0)
+		goto free;
+
+	/*
+	 * Special tree roots may need to modify pointers in @fs_info
+	 * Only quota is supported yet.
+	 */
+	switch (objectid) {
+	case BTRFS_QUOTA_TREE_OBJECTID:
+		if (fs_info->quota_root) {
+			error("quota root already exists");
+			ret = -EEXIST;
+			goto free;
+		}
+		fs_info->quota_root = new_root;
+		fs_info->quota_enabled = 1;
+		break;
+	/*
+	 * Essential trees can't be created by this function, yet.
+	 * As we expect such skeleton exists, or a lot of functions like
+	 * btrfs_alloc_free_block() doesn't work at all
+	 */
+	case BTRFS_ROOT_TREE_OBJECTID:
+	case BTRFS_EXTENT_TREE_OBJECTID:
+	case BTRFS_CHUNK_TREE_OBJECTID:
+	case BTRFS_FS_TREE_OBJECTID:
+		ret = -EEXIST;
+		goto free;
+	default:
+		/* Subvolume trees don't need special handles */
+		if (is_fstree(objectid))
+			break;
+		/* Other special trees are not supported yet */
+		ret = -ENOTTY;
+		goto free;
+	}
+	btrfs_mark_buffer_dirty(node);
+	btrfs_set_root_bytenr(&root_item, btrfs_header_bytenr(node));
+	btrfs_set_root_level(&root_item, 0);
+	btrfs_set_root_generation(&root_item, trans->transid);
+	btrfs_set_root_dirid(&root_item, 0);
+	btrfs_set_root_refs(&root_item, 1);
+	btrfs_set_root_used(&root_item, fs_info->nodesize);
+	location.objectid = objectid;
+	location.type = BTRFS_ROOT_ITEM_KEY;
+	location.offset = 0;
+
+	ret = btrfs_insert_root(trans, fs_info->tree_root, &location,
+				&root_item);
+	if (ret < 0)
+		goto free;
+	return ret;
+
+free:
+	free_extent_buffer(node);
+	free(new_root);
+	return ret;
+}
+
  /*
   * check if the tree block can be shared by multiple trees
   */

ctree.h +2 -0

index 506b76766579..ac8ce70a55e7 100644
@@ -2558,6 +2558,8 @@ int btrfs_copy_root(struct btrfs_trans_handle *trans,
  		      struct btrfs_root *root,
  		      struct extent_buffer *buf,
  		      struct extent_buffer **cow_ret, u64 new_root_objectid);
+int btrfs_create_root(struct btrfs_trans_handle *trans,
+		      struct btrfs_fs_info *fs_info, u64 objectid);
  int btrfs_extend_item(struct btrfs_root *root, struct btrfs_path *path,
  		u32 data_size);
  int btrfs_truncate_item(struct btrfs_root *root, struct btrfs_path *path,

This patchset adds quota support, which means the result fs will have quota enabled by default, and its accounting is already consistent, no manually rescan or quota enable is needed.

2) Fill quota root with basic skeleton
Only 3 items are really needed
a) global quota status item
b) quota info for 0/5 qgroup
c) quota limit for 0/5 qgroup

The content doesn't matter at all.

Quoted 3 lines · Qu Wenruo

This patchset adds quota support, which means the result fs will have quota enabled by default, and its accounting is already consistent, no manually rescan or quota enable is needed.

Great, thanks for working on that. I'm going to merge the preparatory patches next. I'm not sure about the command line option, we might want to design it in a similar way as -O/--features. As this is a runtime feature (unlike the on-disk format features), a new option will be needed, backed by some research what other mkfs.* tools use.

The uuid_tree would at least use the btrfs_create_tree, but I don't think we'd need to create it only on request, so quotas will be the only option.

Quoted 6 lines

2) Fill quota root with basic skeleton
Only 3 items are really needed
a) global quota status item
b) quota info for 0/5 qgroup
c) quota limit for 0/5 qgroup

Quoted 1 line

The content doesn't matter at all.

How would --rootdir and quotas behave in that case? A related question is what will be the semantics of rootdir+quotas+subvolume created at mkfs time.

We can certainly start with only quotas on the empty filesystem, but I'm sure somebody will ask for the wild combination at some point.> -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 3 lines · Qu Wenruo

This patchset adds quota support, which means the result fs will have quota enabled by default, and its accounting is already consistent, no manually rescan or quota enable is needed.

Great, thanks for working on that. I'm going to merge the preparatory patches next. I'm not sure about the command line option, we might want to design it in a similar way as -O/--features. As this is a runtime feature (unlike the on-disk format features), a new option will be needed, backed by some research what other mkfs.* tools use.

The uuid_tree would at least use the btrfs_create_tree, but I don't think we'd need to create it only on request, so quotas will be the only option.

Quoted 6 lines

2) Fill quota root with basic skeleton
Only 3 items are really needed
a) global quota status item
b) quota info for 0/5 qgroup
c) quota limit for 0/5 qgroup

Quoted 1 line

The content doesn't matter at all.

How would --rootdir and quotas behave in that case? A related question is what will be the semantics of rootdir+quotas+subvolume created at mkfs time.

We can certainly start with only quotas on the empty filesystem, but I'm sure somebody will ask for the wild combination at some point.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

This patchset adds quota support, which means the result fs will have quota enabled by default, and its accounting is already consistent, no manually rescan or quota enable is needed.

The overall design of such support is:
1) Create needed tree
Both btrfs_root and real root item and tree root leaf. For this, a new infrastructure, btrfs_create_tree(), is added for this.

2) Fill quota root with basic skeleton
Only 3 items are really needed
a) global quota status item
b) quota info for 0/5 qgroup
c) quota limit for 0/5 qgroup

The content doesn't matter at all.

3) Repair qgroups using infrastructure from qgroup-verify In fact, qgroup repair is just offline rescan.
Although the original qgroup-verify infrastructure is mostly noisy, modify it a little to make it silent to function as offline quota rescan.

And such support is mainly designed for developers and QA guys.

As to enable quota, we must normally mount the fs, enable quota (and rescan if needed).
This ioctl based procedure is not common, and fstests doesn't provide such support.

There are several attempts to make fstests to support it, but due to different reasons, all these attempts failed.

To make it easier to test all existing test cases with btrfs quota enabled, the current best method is to support quota at mkfs time, and here comes the patchset.

Qu Wenruo (8):
btrfs-progs: qgroup-verify: Also repair qgroup status version
btrfs-progs: qgroup-verify: Use fs_info->readonly to check if we
should repair qgroups
btrfs-progs: qgroup-verify: Move qgroup classification out of
report_qgroups
btrfs-progs: qgroup-verify: Allow repair_qgroups function to do silent
repair
btrfs-progs: ctree: Introduce function to create an empty tree
btrfs-progs: mkfs: Introduce function to insert qgroup info and limit
items
btrfs-progs: mkfs: Introduce option to enable quota at mkfs time
btrfs-progs: test/mkfs: Add test case for --enable-quota option

Documentation/mkfs.btrfs.asciidoc           |   3 +
cmds-check.c                                |   2 +-
ctree.c                                     | 109 ++++++++++++++++++++++
ctree.h                                     |   3 +
mkfs/main.c                                 | 136 +++++++++++++++++++++++++++-
qgroup-verify.c                             |  51 ++++++++---
qgroup-verify.h                             |   2 +-
tests/mkfs-tests/001-basic-profiles/test.sh |  10 ++
8 files changed, 302 insertions(+), 14 deletions(-)

Signature · 6 lines

2.14.3

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html