Skip to contentSkip to conversation

Add support for object properties

PATCH In review Filipe David Borba Manana 3 participants 13 messages Nov 12, 2013 – Jan 23, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Add support for object properties

This is a revised version of the original proposal/work from Alexander Block to introduce a generic framework to set properties on btrfs filesystem objects (inodes, subvolumes, filesystems, devices).

I preserved most of Alexander's work, rebasing only his main patch against latest integration branch, adding some missing static qualifiers, malloc failure checks and fixing some checkpatch.pl warnings. On top of his work, I added a few fixes and enhancements and added support for 1 inode property, named compression, which requires the corresponding kernel patch.

Once there's agreement on this feature and patchset, I'll send my xfstests patch that tests this feature.

Alexander's original cover letter:

"
This patchset introduces the btrfs property subgroup. It is the result of a discussion we had on IRC. I tried to make the properties interface as generic and extensible as possible. Comments are welcome.

Currently the command group looks like this:
btrfs prop set [-t <type>] /path/to/object <name> <value>
btrfs prop get [-t <type>] /path/to/object [<name>] (omitting name dumps all) btrfs prop list [-t <type>] /path/to/object (lists properties with description)

The type is used to explicitly specify what type of object you mean. This is necessary in case the object+property combination is ambiguous. For example '/path/to/fs/root' could mean the root subvolume, the directory inode or the filesystem itself. Normally, btrfs-progs will try to detect the type automatically.

David suggested that it should also be possible to specify objects by their id/uuid/fsid. I like that idea, but would be happy if someone else could take over that part :)

For now, I've implemented two properties:

1. read-only. Usable on subvolumes to toggle the read-only flags.
2. label. I looked through btrfs to find good examples of things that could be moved to the new properties interface and the filesystem label looked like a good one. There are for sure more, but that is something for later (and maybe for someone else). I would suggest to move everthing that makes sense over to the props interface and mark the old interfaces as deprecated. Comments on this are welcome.

Patch version history:
v1
Initial version.
v2
- Removed the filesystem prefix and implemented it as new command group
- Switched from the <name>[=<value>] form to the set/get <name> [<value>] form.
- Removed patches "Btrfs-progs: make filesystem_cmd_group non const" and "Btrfs-progs: move skip_prefix and prefixcmp to utils.c". They are not needed anymore due to the 'btrfs prop list' command.
- Udjusted the subvol flags patch to be compatible to the "Btrfs: use _IOR for BTRFS_IOC_SUBVOL_GETFLAGS" patch.
- Using -t <type> instead of <type>: prefix now.
- Changes are based on feedback from Ilya and David.

Alex.
"

Thanks.

Alexander Block (1):
Btrfs-progs: introduce btrfs property subgroup

Filipe David Borba Manana (4):
Btrfs-progs: let get_label return the label instead of printing it
Btrfs-progs: fix detection of root objects in cmds-property.c
Btrfs-progs: add type root to label property
Btrfs-progs: add support for the compression property

 Makefile          |    4 +-
 btrfs.c           |    1 +
 cmds-filesystem.c |   14 +-
 cmds-property.c   |  467 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 commands.h        |    2 +
 ioctl.h           |   12 ++
 props.c           |  184 +++++++++++++++++++++
 props.h           |   43 +++++
 utils.c           |   15 +-
 utils.h           |    2 +-
 10 files changed, 729 insertions(+), 15 deletions(-)
 create mode 100644 cmds-property.c
 create mode 100644 props.c
 create mode 100644 props.h
Signature · 1 line
1.7.9.5
#1 lore raw

PATCH 1/5 Btrfs-progs: let get_label return the label instead of printing it

get_label prints the label at the moment. Change this so that the label is returned and printing is done by the caller.

---
 cmds-filesystem.c |   14 +++++++++++---
 utils.c           |   15 ++++++---------
 utils.h           |    2 +-
 3 files changed, 18 insertions(+), 13 deletions(-)
cmds-filesystem.c +11 -3
index d2cad81..9fda41a 100644
@@ -810,10 +810,18 @@ static int cmd_label(int argc, char **argv)
 	if (check_argc_min(argc, 2) || check_argc_max(argc, 3))
 		usage(cmd_label_usage);
 
-	if (argc > 2)
+	if (argc > 2) {
 		return set_label(argv[1], argv[2]);
-	else
-		return get_label(argv[1]);
+	} else {
+		char label[BTRFS_LABEL_SIZE];
+		int ret;
+
+		ret = get_label(argv[1], label);
+		if (!ret)
+			fprintf(stdout, "%s\n", label);
+
+		return ret;
+	}
 }
 
 const struct cmd_group filesystem_cmd_group = {
utils.c +6 -9
index 85682d9..9105af1 100644
@@ -1343,7 +1343,7 @@ static int set_label_mounted(const char *mount_path, const char *label)
 	return 0;
 }
 
-static int get_label_unmounted(const char *dev)
+static int get_label_unmounted(const char *dev, char *label)
 {
 	struct btrfs_root *root;
 	int ret;
@@ -1366,7 +1366,7 @@ static int get_label_unmounted(const char *dev)
 	if(!root)
 		return -1;
 
-	fprintf(stdout, "%s\n", root->fs_info->super_copy->label);
+	memcpy(label, root->fs_info->super_copy->label, BTRFS_LABEL_SIZE);
 
 	/* Now we close it since we are done. */
 	close_ctree(root);
@@ -1401,18 +1401,15 @@ int get_label_mounted(const char *mount_path, char *labelp)
 	return 0;
 }
 
-int get_label(const char *btrfs_dev)
+int get_label(const char *btrfs_dev, char *label)
 {
 	int ret;
-	char label[BTRFS_LABEL_SIZE];
 
 	if (is_existing_blk_or_reg_file(btrfs_dev))
-		ret = get_label_unmounted(btrfs_dev);
-	else {
+		ret = get_label_unmounted(btrfs_dev, label);
+	else
 		ret = get_label_mounted(btrfs_dev, label);
-		if (!ret)
-			fprintf(stdout, "%s\n", label);
-	}
+
 	return ret;
 }
 
utils.h +1 -1
index 6f4b10c..b38bd3a 100644
@@ -72,7 +72,7 @@ int open_file_or_dir(const char *fname, DIR **dirstream);
 void close_file_or_dir(int fd, DIR *dirstream);
 int get_fs_info(char *path, struct btrfs_ioctl_fs_info_args *fi_args,
 		struct btrfs_ioctl_dev_info_args **di_ret);
-int get_label(const char *btrfs_dev);
+int get_label(const char *btrfs_dev, char *label);
 int set_label(const char *btrfs_dev, const char *label);
 
 char *__strncpy__null(char *dest, const char *src, size_t n);
Signature · 1 line
1.7.9.5
#2 lore raw

Re: [PATCH 1/5] Btrfs-progs: let get_label return the label instead of printing it

Quoted 4 lines · Filipe David Borba Manana

get_label prints the label at the moment. Change this so that the label is returned and printing is done by the caller.

FYI, I'll add this patch to integration now because it's independent on the rest of the series.

david

#3 lore raw

PATCH 2/5 Btrfs-progs: introduce btrfs property subgroup

In reply to Filipe David Borba Manana

From: Alexander Block <[email protected]>

"btrfs filesystem property" is a generic interface to set/get properties on filesystem objects (inodes/subvolumes/filesystems /devs).

This patch adds the generic framework for properties and also implements two properties. The first is the read-only property for subvolumes and the second is the label property for devices.

---
 Makefile        |    4 +-
 btrfs.c         |    1 +
 cmds-property.c |  459 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 commands.h      |    2 +
 props.c         |  110 +++++++++++++
 props.h         |   43 ++++++
 6 files changed, 617 insertions(+), 2 deletions(-)
 create mode 100644 cmds-property.c
 create mode 100644 props.c
 create mode 100644 props.h
Makefile +2 -2
index 039fd81..eef0060 100644
@@ -9,12 +9,12 @@ CFLAGS = -g -O1
 objects = ctree.o disk-io.o radix-tree.o extent-tree.o print-tree.o \
 	  root-tree.o dir-item.o file-item.o inode-item.o inode-map.o \
 	  extent-cache.o extent_io.o volumes.o utils.o repair.o \
-	  qgroup.o raid6.o free-space-cache.o uuid-tree.o list_sort.o
+	  qgroup.o raid6.o free-space-cache.o uuid-tree.o list_sort.o props.o
 cmds_objects = cmds-subvolume.o cmds-filesystem.o cmds-device.o cmds-scrub.o \
 	       cmds-inspect.o cmds-balance.o cmds-send.o cmds-receive.o \
 	       cmds-quota.o cmds-qgroup.o cmds-replace.o cmds-check.o \
 	       cmds-restore.o cmds-rescue.o chunk-recover.o super-recover.o \
-	       cmds-dedup.o
+	       cmds-dedup.o cmds-property.o
 libbtrfs_objects = send-stream.o send-utils.o rbtree.o btrfs-list.o crc32c.o \
 		   uuid-tree.o
 libbtrfs_headers = send-stream.h send-utils.h send.h rbtree.h btrfs-list.h \
btrfs.c +1 -0
index dfae35f..16458ef 100644
@@ -250,6 +250,7 @@ static const struct cmd_group btrfs_cmd_group = {
 		{ "rescue", cmd_rescue, NULL, &rescue_cmd_group, 0 },
 		{ "restore", cmd_restore, cmd_restore_usage, NULL, 0 },
 		{ "inspect-internal", cmd_inspect, NULL, &inspect_cmd_group, 0 },
+		{ "property", cmd_property, NULL, &property_cmd_group, 0 },
 		{ "send", cmd_send, cmd_send_usage, NULL, 0 },
 		{ "receive", cmd_receive, cmd_receive_usage, NULL, 0 },
 		{ "quota", cmd_quota, NULL, &quota_cmd_group, 0 },
cmds-property.c new file +459 -0

459 lines not on this page · raw message

commands.h +2 -0
index 6fccc15..9ae2e10 100644
@@ -87,6 +87,7 @@ extern const struct cmd_group balance_cmd_group;
 extern const struct cmd_group device_cmd_group;
 extern const struct cmd_group scrub_cmd_group;
 extern const struct cmd_group inspect_cmd_group;
+extern const struct cmd_group property_cmd_group;
 extern const struct cmd_group quota_cmd_group;
 extern const struct cmd_group qgroup_cmd_group;
 extern const struct cmd_group replace_cmd_group;
@@ -110,6 +111,7 @@ int cmd_check(int argc, char **argv);
 int cmd_chunk_recover(int argc, char **argv);
 int cmd_super_recover(int argc, char **argv);
 int cmd_inspect(int argc, char **argv);
+int cmd_property(int argc, char **argv);
 int cmd_send(int argc, char **argv);
 int cmd_receive(int argc, char **argv);
 int cmd_quota(int argc, char **argv);
props.c new file +110 -0

110 lines not on this page · raw message

props.h new file +43 -0

43 lines not on this page · raw message

Signature · 1 line
1.7.9.5
#4 lore raw

PATCH 3/5 Btrfs-progs: fix detection of root objects in cmds-property.c

In reply to Filipe David Borba Manana

Several fixes:

1) The function check_is_root() returns 0 if the object is root;

2) Don't treat any error from get fsid ioctl as meaning the target is root. Only -ENOTTY means it's a root (parent directory is not a btrfs fs) and a -ENOTDIR means our target object is not a directory, therefore it can be the root;

3) Fix the comparison of the target and target's parent fs ids. If they are different, it means the target is a mount point in a btrfs fs, therefore it's a root, otherwise it isn't.

---
 cmds-property.c |   24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)
cmds-property.c +16 -8
index 6699e61..3d1f18c 100644
@@ -75,7 +75,7 @@ static int parse_prop(const char *arg, const struct prop_handler *props,
 	return -1;
 }
 
-static int get_fsid(const char *path, u8 *fsid)
+static int get_fsid(const char *path, u8 *fsid, int silent)
 {
 	int ret;
 	int fd;
@@ -84,7 +84,8 @@ static int get_fsid(const char *path, u8 *fsid)
 	fd = open(path, O_RDONLY);
 	if (fd < 0) {
 		ret = -errno;
-		fprintf(stderr, "ERROR: open %s failed. %s\n", path,
+		if (!silent)
+			fprintf(stderr, "ERROR: open %s failed. %s\n", path,
 				strerror(-ret));
 		goto out;
 	}
@@ -109,7 +110,7 @@ static int check_btrfs_object(const char *object)
 	int ret;
 	u8 fsid[BTRFS_FSID_SIZE];
 
-	ret = get_fsid(object, fsid);
+	ret = get_fsid(object, fsid, 0);
 	if (ret < 0)
 		ret = 0;
 	else
@@ -134,20 +135,27 @@ static int check_is_root(const char *object)
 		strcat(tmp, "/");
 	strcat(tmp, "..");
 
-	ret = get_fsid(object, fsid);
+	ret = get_fsid(object, fsid, 0);
 	if (ret < 0) {
 		fprintf(stderr, "ERROR: get_fsid for %s failed. %s\n", object,
 				strerror(-ret));
 		goto out;
 	}
 
-	ret = get_fsid(tmp, fsid2);
-	if (ret < 0) {
+	ret = get_fsid(tmp, fsid2, 1);
+	if (ret == -ENOTTY) {
 		ret = 0;
 		goto out;
+	} else if (ret == -ENOTDIR) {
+		ret = 1;
+		goto out;
+	} else if (ret < 0) {
+		fprintf(stderr, "ERROR: get_fsid for %s failed. %s\n", tmp,
+			strerror(-ret));
+		goto out;
 	}
 
-	if (!memcmp(fsid, fsid2, BTRFS_FSID_SIZE)) {
+	if (memcmp(fsid, fsid2, BTRFS_FSID_SIZE)) {
 		ret = 0;
 		goto out;
 	}
@@ -195,7 +203,7 @@ static int autodetect_object_types(const char *object, int *types_out)
 		ret = check_is_root(object);
 		if (ret < 0)
 			goto out;
-		if (ret)
+		if (!ret)
 			types |= prop_object_root;
 	}
 
Signature · 1 line
1.7.9.5
#5 lore raw

PATCH 4/5 Btrfs-progs: add type root to label property

In reply to Filipe David Borba Manana

So that we can get the label of a mounted filesystem.

Before this change:

        $ btrfs prop get /mnt/btrfs label
        ERROR: object is not compatible with property
        $ btrfs prop get /dev/sdb3 label
        ERROR: dev /dev/sdb3 is mounted, use mount point
        ERROR: failed to set/get property for object.

After this change:

        $ btrfs prop get /mnt/btrfs label
        label=foobar
---
 props.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
props.c +2 -1
index 763fd57..12fdc52 100644
@@ -105,6 +105,7 @@ static int prop_label(enum prop_object_type type,
 const struct prop_handler prop_handlers[] = {
 	{"ro", "Set/get read-only flag of subvolume.", 0, prop_object_subvol,
 	 prop_read_only},
-	{"label", "Set/get label of device.", 0, prop_object_dev, prop_label},
+	{"label", "Set/get label of device.", 0,
+	 prop_object_dev | prop_object_root, prop_label},
 	{0, 0, 0, 0, 0}
 };
Signature · 1 line
1.7.9.5
#6 lore raw

PATCH 5/5 Btrfs-progs: add support for the compression property

In reply to Filipe David Borba Manana

With this property, one can enable compression for individual files without the need to mount the filesystem with the compress or compress-force options, and specify the compression algorithm.

When applied against a directory, files created under that directory will inherit the compression property.

This requires the corresponding kernel patch, which adds the support for the property get and set ioctls and implementation of the compression property.

---
 ioctl.h |   12 +++++++++++
 props.c |   73 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 85 insertions(+)
ioctl.h +12 -0
index d21413f..b1f7928 100644
@@ -465,6 +465,14 @@ struct btrfs_ioctl_qgroup_create_args {
 	__u64 qgroupid;
 };
 
+struct btrfs_ioctl_getset_prop_args {
+	__u16 name_len;   /* in     */
+	__u16 value_len;  /* in/out */
+	__u8  unused[4];
+	__u64 name_ptr;   /* in     */
+	__u64 value_ptr;  /* in/out */
+};
+
 /* Error codes as returned by the kernel */
 enum btrfs_err_code {
 	notused,
@@ -551,6 +559,10 @@ struct btrfs_ioctl_clone_range_args {
 #define BTRFS_IOC_DEFAULT_SUBVOL _IOW(BTRFS_IOCTL_MAGIC, 19, u64)
 #define BTRFS_IOC_SPACE_INFO _IOWR(BTRFS_IOCTL_MAGIC, 20, \
 				    struct btrfs_ioctl_space_args)
+#define BTRFS_IOC_GET_PROP _IOR(BTRFS_IOCTL_MAGIC, 21, \
+				struct btrfs_ioctl_getset_prop_args)
+#define BTRFS_IOC_SET_PROP _IOW(BTRFS_IOCTL_MAGIC, 21, \
+				struct btrfs_ioctl_getset_prop_args)
 #define BTRFS_IOC_SNAP_CREATE_V2 _IOW(BTRFS_IOCTL_MAGIC, 23, \
 				   struct btrfs_ioctl_vol_args_v2)
 #define BTRFS_IOC_SUBVOL_CREATE_V2 _IOW(BTRFS_IOCTL_MAGIC, 24, \
props.c +73 -0

89 lines not on this page · raw message

Signature · 1 line
1.7.9.5
#7 lore raw

PATCH v2 5/5 Btrfs-progs: add support for the compression property

In reply to Filipe David Borba Manana

With this property, one can enable compression for individual files without the need to mount the filesystem with the compress or compress-force options, and specify the compression algorithm.

When applied against a directory, files created under that directory will inherit the compression property.

This requires the corresponding kernel patch, which adds the support for setting and getting properties and implements the compression property.

---

V2: Update to match latest kernel change - use regular set/get xattr
path to set/get compression property.

 props.c |   78 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)
props.c +78 -0
index 12fdc52..4d0aeea 100644
@@ -16,6 +16,8 @@
 
 #include <sys/stat.h>
 #include <sys/ioctl.h>
+#include <sys/types.h>
+#include <attr/xattr.h>
 #include <fcntl.h>
 #include <unistd.h>
 
@@ -24,6 +26,10 @@
 #include "utils.h"
 #include "props.h"
 
+#define XATTR_BTRFS_PREFIX     "btrfs."
+#define XATTR_BTRFS_PREFIX_LEN (sizeof(XATTR_BTRFS_PREFIX) - 1)
+
+
 static int prop_read_only(enum prop_object_type type,
 			  const char *object,
 			  const char *name,
@@ -102,10 +108,82 @@ static int prop_label(enum prop_object_type type,
 	return ret;
 }
 
+static int prop_compression(enum prop_object_type type,
+			    const char *object,
+			    const char *name,
+			    const char *value)
+{
+	int ret;
+	ssize_t sret;
+	int fd = -1;
+	DIR *dirstream = NULL;
+	char *buf = NULL;
+	char *xattr_name = NULL;
+
+	fd = open_file_or_dir(object, &dirstream);
+	if (fd == -1) {
+		ret = -errno;
+		fprintf(stderr, "ERROR: open %s failed. %s\n",
+			object, strerror(-ret));
+		goto out;
+	}
+
+	xattr_name = malloc(XATTR_BTRFS_PREFIX_LEN + strlen(name));
+	if (!xattr_name) {
+		ret = -ENOMEM;
+		goto out;
+	}
+	memcpy(xattr_name, XATTR_BTRFS_PREFIX, XATTR_BTRFS_PREFIX_LEN);
+	memcpy(xattr_name + XATTR_BTRFS_PREFIX_LEN, name, strlen(name));
+
+	if (value)
+		sret = fsetxattr(fd, xattr_name, value, strlen(value), 0);
+	else
+		sret = fgetxattr(fd, xattr_name, NULL, 0);
+	if (sret < 0) {
+		ret = -errno;
+		if (ret != -ENODATA)
+			fprintf(stderr,
+				"ERROR: failed to %s compression for %s. %s\n",
+				value ? "set" : "get", object, strerror(-ret));
+		goto out;
+	}
+	if (!value) {
+		size_t len = sret;
+
+		buf = malloc(len);
+		if (!buf) {
+			ret = -ENOMEM;
+			goto out;
+		}
+		sret = fgetxattr(fd, xattr_name, buf, len);
+		if (sret < 0) {
+			ret = -errno;
+			fprintf(stderr,
+				"ERROR: failed to get compression for %s. %s\n",
+				object, strerror(-ret));
+			goto out;
+		}
+		fprintf(stdout, "compression=%.*s\n", (int)len, buf);
+	}
+
+	ret = 0;
+out:
+	free(xattr_name);
+	free(buf);
+	if (fd >= 0)
+		close_file_or_dir(fd, dirstream);
+
+	return ret;
+}
+
+
 const struct prop_handler prop_handlers[] = {
 	{"ro", "Set/get read-only flag of subvolume.", 0, prop_object_subvol,
 	 prop_read_only},
 	{"label", "Set/get label of device.", 0,
 	 prop_object_dev | prop_object_root, prop_label},
+	{"compression", "Set/get compression for a file or directory", 0,
+	 prop_object_inode, prop_compression},
 	{0, 0, 0, 0, 0}
 };
Signature · 1 line
1.7.9.5
#8 lore raw
In reply to Filipe David Borba Manana
Quoted 3 lines · Filipe David Borba Manana

This is a revised version of the original proposal/work from Alexander Block to introduce a generic framework to set properties on btrfs filesystem objects (inodes, subvolumes, filesystems, devices).

Thank you very much for working on that. I've added patches 2-5 to the unstable part of integration branch. More comments later.

david

#9 lore raw
In reply to Filipe David Borba Manana
Quoted 9 lines · Filipe David Borba Manana

This is a revised version of the original proposal/work from Alexander Block to introduce a generic framework to set properties on btrfs filesystem objects (inodes, subvolumes, filesystems, devices).

Currently the command group looks like this:
btrfs prop set [-t <type>] /path/to/object <name> <value>
btrfs prop get [-t <type>] /path/to/object [<name>] (omitting name dumps all) btrfs prop list [-t <type>] /path/to/object (lists properties with description)

Quoted 5 lines

The type is used to explicitly specify what type of object you mean. This is necessary in case the object+property combination is ambiguous. For example '/path/to/fs/root' could mean the root subvolume, the directory inode or the filesystem itself. Normally, btrfs-progs will try to detect the type automatically.

The generic commandline UI still looks ok to me, storing properties as xattr’s is also ok (provided that we can capture the “btrfs.” xattr namespace).

I’ll dump my thoughts and questions about the rest.

1) Where are stored properties that are not directly attached to an inode? Ie. whole-filesystem and device. How can I access props for a subvolume that is not currently reachable in the directory tree?

A fs or device props must be accessible any time, eg. no matter which subvolume is currently mounted. This should be probably a special case where the property can be queried from any inode but will be internally routed to eg. toplevel subvolume that will store the respective xattrs.

2) if a property’s object can be ambiguous, how is that distinguished in the xattrs?

We don’t have a list of props yet, so I’m trying to use one that hopefully makes some sense. The example here can be persistent-mount-options that are attached to fs and a subvolume. The fs-wide props will apply when a different subvolume is explicitly mounted.

Assuming that the xattrs are stored with the toplevel subvolume, the fs-wide and per-subvolume property must have a differnt xattr name (another option is to store fs-wide elsewhere). So the question is, if we should encode the property object into the xattr name directly. Eg.:

btrfs.fs.persistent_mount
btrfs.subvol.persistent_mount

or if the fs-wide uses a reserved naming scheme that would appear as xattr named

btrfs.persistent_mount

but the value would differ if queried with ‘-t fs or ‘-t subvolume’.

3) property precedence, interaction with mount options

The precedence should follow the rule of the least surprise, ie. if I set eg. a compression of a file to zlib, set the subvolume compression type to ‘none’ and have fs-wide mount the filesystem with compress=lzo, I’m expecting that the file will use ‘zlib’.

The generic rule says that mount options (that have a corresponding property) take precedence. There may be exceptions.

What if there are no specific mount options, but the subvolume has eg. compression set? Should new files inherit that automatically or only if explicitly marked as such?

The background concern is how far do I need to look and whether this could be a potential performance problem. The lookup chain can be long: inode -> dir -> subvol -> fs, plus mount options along the path where applicable.

If the xattr values are cached in structs that are accessed anyway (containing subvol, fs_info) and updated when property xattr changes, this could work.

4) behaviour of tools that see the btrfs-specific props

What would cp or rsync do when copying the xattrs to a different filesystem? They may simply ignore it and just try to copy the user. namespace, I haven’t done any research here.

5) properties to consider, naming, xattr name namespaces, inheritable props

Here’s a list of properties that I’ve collected so far:

filesystem-wide
- persistent mount options
- default raid allocation profile
- default compression parameters
- label
- scrub ioprio

subvolume
- persistent mount options
- default raid allocation profile
- default compression parameters
- writable

device
- writable/ro/dead
- hot-spare
- preferred for metadata
- preferred data reads
- preferred data writes
- no new allocations if possible
- speed profile, io priorities
- allocation group - from project idea "Chunk allocation groups"
- "ssd cache", L2ARC
- scrub ioprio

file, directory
- compression parameters
- raid allocation profile

details on compression parameters:
- compression algo
- compression level
- desired compression ratio target
- file compressibility hint - from project idea “Compression updates”
- compression container type - dtto

The props should be basically self explanatory. This is not a complete or finalized list, feel free to suggest what you think should/not be here.

The number of compression parameters suggest that a namespace would be appropriate. This could establish a common practice for any set of properties that are not attached to any filesystem object but are grouped logically.

Most of the props are expected to be read-write, I don’t have a counterexample for a property that makes sense, is persistent and must be read-only. Such one can be exported through the sysfs, so s/Most of/All/ in the previous sentence.

I don’t have a proposal how to mark inheritable props yet.

david

#10 lore raw

Hi David,

Quoted 21 lines · David Sterba
Quoted 9 lines · Filipe David Borba Manana

This is a revised version of the original proposal/work from Alexander Block to introduce a generic framework to set properties on btrfs filesystem objects (inodes, subvolumes, filesystems, devices).

Currently the command group looks like this:
btrfs prop set [-t <type>] /path/to/object <name> <value>
btrfs prop get [-t <type>] /path/to/object [<name>] (omitting name dumps all) btrfs prop list [-t <type>] /path/to/object (lists properties with description)

Quoted 5 lines

The type is used to explicitly specify what type of object you mean. This is necessary in case the object+property combination is ambiguous. For example '/path/to/fs/root' could mean the root subvolume, the directory inode or the filesystem itself. Normally, btrfs-progs will try to detect the type automatically.

The generic commandline UI still looks ok to me, storing properties as xattr’s is also ok (provided that we can capture the “btrfs.” xattr namespace).

I’ll dump my thoughts and questions about the rest.

Quoted 8 lines

1) Where are stored properties that are not directly attached to an inode? Ie. whole-filesystem and device. How can I access props for a subvolume that is not currently reachable in the directory tree?

A fs or device props must be accessible any time, eg. no matter which subvolume is currently mounted. This should be probably a special case where the property can be queried from any inode but will be internally routed to eg. toplevel subvolume that will store the respective xattrs.

I think that we should divided "how access the properties" and "where the properties are stored".

- Storing the *inode* properties in xattrs makes sense: it is a clean interface, and the infrastructure is capable to hold all the informations. It could be discussed also if we need to use 1 property <-> 1 xattr or use one xattr to hold more properties (this could alleviate some performance problem)

- Storing the *subvolume* and the *filesystem* properties in an xattr associated to subvolue or '/' inode could be done. When it is accessed this subvolume or '/' inode (during the mount and/or inode traversing) all the information could be read from the xattr and stored in the btrfs_fs_info struct and btrfs_root struct (which are easily accessible from the inode, avoiding performance issues)

- For the *device* properties, we could think to use other trees like the device tree to store the information but still using the *xattr interfaces to access them.

Quoted 28 lines

2) if a property’s object can be ambiguous, how is that distinguished in the xattrs?

We don’t have a list of props yet, so I’m trying to use one that hopefully makes some sense. The example here can be persistent-mount-options that are attached to fs and a subvolume. The fs-wide props will apply when a different subvolume is explicitly mounted.

Assuming that the xattrs are stored with the toplevel subvolume, the fs-wide and per-subvolume property must have a differnt xattr name (another option is to store fs-wide elsewhere). So the question is, if we should encode the property object into the xattr name directly. Eg.:

btrfs.fs.persistent_mount
btrfs.subvol.persistent_mount

or if the fs-wide uses a reserved naming scheme that would appear as xattr named

btrfs.persistent_mount

but the value would differ if queried with ‘-t fs or ‘-t subvolume’.

3) property precedence, interaction with mount options

Quoted 8 lines

The precedence should follow the rule of the least surprise, ie. if I set eg. a compression of a file to zlib, set the subvolume compression type to ‘none’ and have fs-wide mount the filesystem with compress=lzo, I’m expecting that the file will use ‘zlib’.

The generic rule says that mount options (that have a corresponding property) take precedence. There may be exceptions.

As general rule I suggest the following priority list:

fs props, subvol props, dir props, inode props, mount opts

It should be possible to "override" via mount option all options (to force some behaviour).

However I have some doubts about:

1) in case of nested subvolumes, should the inner subvolumes inherit the properties of the outer subvolumes? I think no, because it is not so obvious the hierarchy when a subvolume is mounted (or moved).

2) there are properties that are "inode" related. Other no. For example does make sense to have inode-setting about compression, raid profile, datasum/cow ... when the data is shared between different inode/subvolume which could have different setup? Which should be the "least surprise" behaviour ? Or we should intend these properties only as hints for the new file/data/chunk ? Anyway what it would do a balance in these case ?
I am inclined to think that the "storage policy" (raid profile, compression, cow...) should be "per filesystem" and not per inode:
having different "storage policy" doesn't mix well with the sharing of data (like the snapshot).

Quoted 7 lines

What if there are no specific mount options, but the subvolume has eg. compression set? Should new files inherit that automatically or only if explicitly marked as such?

The background concern is how far do I need to look and whether this could be a potential performance problem. The lookup chain can be long: inode -> dir -> subvol -> fs, plus mount options along the path where applicable.

Looking at the acl, exists the concept of the "default" acl (the one inherited).

Quoted 2 lines

If the xattr values are cached in structs that are accessed anyway (containing subvol, fs_info) and updated when property xattr changes, this could work.

I reached the same conclusion

Quoted 10 lines

4) behaviour of tools that see the btrfs-specific props

What would cp or rsync do when copying the xattrs to a different filesystem? They may simply ignore it and just try to copy the user. namespace, I haven’t done any research here.

From rsync man page

This option causes rsync to update the destination extended attributes to be the same as the source ones.

For systems that support extended-attribute namespaces, a copy being done by a super-user copies all namespaces except sys‐ tem.*.

Quoted 45 lines

5) properties to consider, naming, xattr name namespaces, inheritable props

Here’s a list of properties that I’ve collected so far:

filesystem-wide
- persistent mount options
- default raid allocation profile
- default compression parameters
- label
- scrub ioprio

subvolume
- persistent mount options
- default raid allocation profile
- default compression parameters
- writable

device
- writable/ro/dead
- hot-spare
- preferred for metadata
- preferred data reads
- preferred data writes
- no new allocations if possible
- speed profile, io priorities
- allocation group - from project idea "Chunk allocation groups"
- "ssd cache", L2ARC
- scrub ioprio

file, directory
- compression parameters
- raid allocation profile

details on compression parameters:
- compression algo
- compression level
- desired compression ratio target
- file compressibility hint - from project idea “Compression updates”
- compression container type - dtto

The props should be basically self explanatory. This is not a complete or finalized list, feel free to suggest what you think should/not be here.

Quoted 7 lines

The number of compression parameters suggest that a namespace would be appropriate. This could establish a common practice for any set of properties that are not attached to any filesystem object but are grouped logically.

Most of the props are expected to be read-write, I don’t have a counterexample for a property that makes sense, is persistent and must be read-only. Such one can be exported through the sysfs, so s/Most of/All/ in the previous sentence.

Some properties which could be exported as readonly:
- the uuid of the subvolume
- the "generation" number (but in effect exists and ioctl for that)

Anyway sysfs is good for filesystem properties; it could handle also subvolume properties. Definitely it is not adapted to "per inode" properties.

Quoted 10 lines

I don’t have a proposal how to mark inheritable props yet.

david
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 6 lines
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo
#11 lore raw
In reply to David Sterba
Quoted 131 lines · David Sterba
Quoted 9 lines · Filipe David Borba Manana

This is a revised version of the original proposal/work from Alexander Block to introduce a generic framework to set properties on btrfs filesystem objects (inodes, subvolumes, filesystems, devices).

Currently the command group looks like this:
btrfs prop set [-t <type>] /path/to/object <name> <value>
btrfs prop get [-t <type>] /path/to/object [<name>] (omitting name dumps all) btrfs prop list [-t <type>] /path/to/object (lists properties with description)

Quoted 5 lines

The type is used to explicitly specify what type of object you mean. This is necessary in case the object+property combination is ambiguous. For example '/path/to/fs/root' could mean the root subvolume, the directory inode or the filesystem itself. Normally, btrfs-progs will try to detect the type automatically.

The generic commandline UI still looks ok to me, storing properties as xattr’s is also ok (provided that we can capture the “btrfs.” xattr namespace).

I’ll dump my thoughts and questions about the rest.

1) Where are stored properties that are not directly attached to an inode? Ie. whole-filesystem and device. How can I access props for a subvolume that is not currently reachable in the directory tree?

A fs or device props must be accessible any time, eg. no matter which subvolume is currently mounted. This should be probably a special case where the property can be queried from any inode but will be internally routed to eg. toplevel subvolume that will store the respective xattrs.

2) if a property’s object can be ambiguous, how is that distinguished in the xattrs?

We don’t have a list of props yet, so I’m trying to use one that hopefully makes some sense. The example here can be persistent-mount-options that are attached to fs and a subvolume. The fs-wide props will apply when a different subvolume is explicitly mounted.

Assuming that the xattrs are stored with the toplevel subvolume, the fs-wide and per-subvolume property must have a differnt xattr name (another option is to store fs-wide elsewhere). So the question is, if we should encode the property object into the xattr name directly. Eg.:

btrfs.fs.persistent_mount
btrfs.subvol.persistent_mount

or if the fs-wide uses a reserved naming scheme that would appear as xattr named

btrfs.persistent_mount

but the value would differ if queried with ‘-t fs or ‘-t subvolume’.

3) property precedence, interaction with mount options

The precedence should follow the rule of the least surprise, ie. if I set eg. a compression of a file to zlib, set the subvolume compression type to ‘none’ and have fs-wide mount the filesystem with compress=lzo, I’m expecting that the file will use ‘zlib’.

The generic rule says that mount options (that have a corresponding property) take precedence. There may be exceptions.

What if there are no specific mount options, but the subvolume has eg. compression set? Should new files inherit that automatically or only if explicitly marked as such?

The background concern is how far do I need to look and whether this could be a potential performance problem. The lookup chain can be long: inode -> dir -> subvol -> fs, plus mount options along the path where applicable.

If the xattr values are cached in structs that are accessed anyway (containing subvol, fs_info) and updated when property xattr changes, this could work.

4) behaviour of tools that see the btrfs-specific props

What would cp or rsync do when copying the xattrs to a different filesystem? They may simply ignore it and just try to copy the user. namespace, I haven’t done any research here.

5) properties to consider, naming, xattr name namespaces, inheritable props

Here’s a list of properties that I’ve collected so far:

filesystem-wide
- persistent mount options
- default raid allocation profile
- default compression parameters
- label
- scrub ioprio

subvolume
- persistent mount options
- default raid allocation profile
- default compression parameters
- writable

device
- writable/ro/dead
- hot-spare
- preferred for metadata
- preferred data reads
- preferred data writes
- no new allocations if possible
- speed profile, io priorities
- allocation group - from project idea "Chunk allocation groups"
- "ssd cache", L2ARC
- scrub ioprio

file, directory
- compression parameters
- raid allocation profile

details on compression parameters:
- compression algo
- compression level
- desired compression ratio target
- file compressibility hint - from project idea “Compression updates”
- compression container type - dtto

The props should be basically self explanatory. This is not a complete or finalized list, feel free to suggest what you think should/not be here.

The number of compression parameters suggest that a namespace would be appropriate. This could establish a common practice for any set of properties that are not attached to any filesystem object but are grouped logically.

Quoted 5 lines

Most of the props are expected to be read-write, I don’t have a counterexample for a property that makes sense, is persistent and must be read-only. Such one can be exported through the sysfs, so s/Most of/All/ in the previous sentence.

I don’t have a proposal how to mark inheritable props yet.

Thanks for sharing your thoughts on this David.

There seems to be a lot of details for this feature, and many other projects (such as those you point on the wiki) would benefit from properties.
It might be hard to get all requirements at once, so I think it might be better to start simple, but in a way that doesn't prevent adding the more advanced requirements later.

As for the xattrs namespace: yes, we can capture the "btrfs." namespace. hfsplus does this, it captures the "osx." namespace. See my reply to Hugo about this on this thread:
http://comments.gmane.org/gmane.comp.file-systems.btrfs/29972

My work so far, for the kernel side, was all about inode (and subvolume) properties. Haven't considered properties for devices so far.
I think the inheritable detail can depend on the specific property. What I did so far, adds generic support for both inheritable and non-inheritable ones.

And yes, so far it has some impact on performance. I added some results (and concerns) in the last rfc patch (v4) I just sent minutes ago.
By using xattrs, while simplifying the storage and not touching/extending the on-disk format, it will always have some impact on performance - they take leaf space, hence btrees will be larger than before.

I think we need to clearly define what are the initial requirements, start with a simple approach, always thinking on how to extend/refine it later without the need for breaking backwards compatibility later of course.

Thanks David.

Quoted 3 lines

david

Signature · 5 lines
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."
#12 lore raw
Quoted 6 lines · Filipe David Manana

There seems to be a lot of details for this feature, and many other projects (such as those you point on the wiki) would benefit from properties.
It might be hard to get all requirements at once, so I think it might be better to start simple, but in a way that doesn't prevent adding the more advanced requirements later.

Absolutely, that's why I'm trying to gather all potential future requirements and usecases now so we don't become surpristed later. Although it's detailed, there are groups that are covered by saying eg. "this is per-device, stored here, accessed trhough this xattr". If the proposal is able to give such answers, then it's what I'm expecting at this phase.

Quoted 4 lines

As for the xattrs namespace: yes, we can capture the "btrfs." namespace. hfsplus does this, it captures the "osx." namespace. See my reply to Hugo about this on this thread:
http://comments.gmane.org/gmane.comp.file-systems.btrfs/29972

JFYI, there's a discussion on fsdevel about replacing the existing file attributes ioctl with something else, xattr's are considered as the UI/storage, so it would need some coordination to be consistent with whatever gets decided.

Quoted 3 lines

My work so far, for the kernel side, was all about inode (and subvolume) properties. Haven't considered properties for devices so far.

I hope it's clear that I'm not asking about implementing all the properties from the beginning. I's fine that you're focused on inode props, get the things working so we have something tangible to play with. If the device props are not forgottent to add to some high-level xattr namespace, I'm satisfied.

Quoted 4 lines

By using xattrs, while simplifying the storage and not touching/extending the on-disk format, it will always have some impact on performance - they take leaf space, hence btrees will be larger than before.

Not all of the props need to end up as a xattr, we can still use the existing inode->flags field that's used for the file attributes, there are 11 bits occupied, plenty of room to add more where it would make sense. The rest shall be in xattr items.

Quoted 4 lines

I think we need to clearly define what are the initial requirements, start with a simple approach, always thinking on how to extend/refine it later without the need for breaking backwards compatibility later of course.

Well, I think once the xattr naming scheme is set it won't be changed or refined. I'm expecting only specific properties appended. Eg. if you propose btrfs.compression as an inode property, then it's a no-go to move it to btrfs.fs.compression once a whole-filesystem gains persistent option, and refine again when somebody like me comes and says that any *.compression should really be a napespace that groups all compression-related options.