Skip to contentSkip to conversation

Make btrfs_prepare_device parallel during mkfs.btrfs

PATCH Superseded Li Zhang 3 participants 11 messages Aug 24, 2022 – Aug 29, 2022 Btrfs file system

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Li Zhang

author
#0 lore raw

PATCH 1/1 Make btrfs_prepare_device parallel during mkfs.btrfs

[enhancement]
When a disk is formatted as btrfs, it calls btrfs_prepare_device for each device, which takes too much time.

[implementation]
Put each btrfs_prepare_device into a thread, wait for the first thread to complete to mkfs.btrfs, and wait for other threads to complete before adding other devices to the file system.

[test]
Using the btrfs-progs test case mkfs-tests, mkfs.btrfs works fine.

But I don't have an actual zoed device,
so I don't know how much time it saves, If you guys have a way to test it, please let me know.

---
Issue: 496

 mkfs/main.c | 113 +++++++++++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 86 insertions(+), 27 deletions(-)
mkfs/main.c +86 -27
index ce096d3..35fefe2 100644
@@ -31,6 +31,7 @@
 #include <uuid/uuid.h>
 #include <ctype.h>
 #include <blkid/blkid.h>
+#include <pthread.h>
 #include "kernel-shared/ctree.h"
 #include "kernel-shared/disk-io.h"
 #include "kernel-shared/free-space-tree.h"
@@ -60,6 +61,18 @@ struct mkfs_allocation {
 	u64 system;
 };
 
+
+struct prepare_device_progress {
+	char *file;
+	u64 dev_block_count;
+	u64 block_count;
+	bool zero_end;
+	bool discard;
+	bool zoned;
+	int oflags;
+	int ret;
+};
+
 static int create_metadata_block_groups(struct btrfs_root *root, bool mixed,
 				struct mkfs_allocation *allocation)
 {
@@ -969,6 +982,28 @@ fail:
 	return ret;
 }
 
+static void *prepare_one_dev(void *ctx)
+{
+	struct prepare_device_progress *prepare_ctx = ctx;
+	int fd;
+
+	fd = open(prepare_ctx->file, prepare_ctx->oflags);
+	if (fd < 0) {
+		error("unable to open %s: %m", prepare_ctx->file);
+		prepare_ctx->ret = fd;
+		return NULL;
+	}
+	prepare_ctx->ret = btrfs_prepare_device(fd,
+		prepare_ctx->file, &prepare_ctx->dev_block_count,
+		prepare_ctx->block_count,
+		(bconf.verbose ? PREP_DEVICE_VERBOSE : 0) |
+		(prepare_ctx->zero_end ? PREP_DEVICE_ZERO_END : 0) |
+		(prepare_ctx->discard ? PREP_DEVICE_DISCARD : 0) |
+		(prepare_ctx->zoned ? PREP_DEVICE_ZONED : 0));
+	close(fd);
+	return NULL;
+}
+
 int BOX_MAIN(mkfs)(int argc, char **argv)
 {
 	char *file;
@@ -997,7 +1032,6 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 	bool ssd = false;
 	bool zoned = false;
 	bool force_overwrite = false;
-	int oflags;
 	char *source_dir = NULL;
 	bool source_dir_set = false;
 	bool shrink_rootdir = false;
@@ -1006,6 +1040,8 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 	u64 shrink_size;
 	int dev_cnt = 0;
 	int saved_optind;
+	pthread_t *t_prepare = NULL;
+	struct prepare_device_progress *prepare_ctx = NULL;
 	char fs_uuid[BTRFS_UUID_UNPARSED_SIZE] = { 0 };
 	u64 features = BTRFS_MKFS_DEFAULT_FEATURES;
 	u64 runtime_features = BTRFS_MKFS_DEFAULT_RUNTIME_FEATURES;
@@ -1428,29 +1464,45 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 		goto error;
 	}
 
-	dev_cnt--;
-
-	oflags = O_RDWR;
-	if (zoned && zoned_model(file) == ZONED_HOST_MANAGED)
-		oflags |= O_DIRECT;
+	t_prepare = malloc(dev_cnt * sizeof(*t_prepare));
+	prepare_ctx = malloc(dev_cnt * sizeof(*prepare_ctx));
 
-	/*
-	 * Open without O_EXCL so that the problem should not occur by the
-	 * following operation in kernel:
-	 * (btrfs_register_one_device() fails if O_EXCL is on)
-	 */
-	fd = open(file, oflags);
-	if (fd < 0) {
-		error("unable to open %s: %m", file);
+	if (!t_prepare || !prepare_ctx) {
+		error("unable to prepare dev");
 		goto error;
 	}
-	ret = btrfs_prepare_device(fd, file, &dev_block_count, block_count,
-			(zero_end ? PREP_DEVICE_ZERO_END : 0) |
-			(discard ? PREP_DEVICE_DISCARD : 0) |
-			(bconf.verbose ? PREP_DEVICE_VERBOSE : 0) |
-			(zoned ? PREP_DEVICE_ZONED : 0));
+
+	for (i = 0; i < dev_cnt; i++) {
+		prepare_ctx[i].file = argv[optind + i - 1];
+		prepare_ctx[i].block_count = block_count;
+		prepare_ctx[i].dev_block_count = block_count;
+		prepare_ctx[i].zero_end = zero_end;
+		prepare_ctx[i].discard = discard;
+		prepare_ctx[i].zoned = zoned;
+		if (i == 0) {
+			prepare_ctx[i].oflags = O_RDWR;
+			/*
+			 * Open without O_EXCL so that the problem should
+			 * not occur by the following operation in kernel:
+			 * (btrfs_register_one_device() fails if O_EXCL is on)
+			 */
+			if (zoned && zoned_model(file) == ZONED_HOST_MANAGED)
+				prepare_ctx[i].oflags = O_RDWR | O_DIRECT;
+		} else {
+			prepare_ctx[i].oflags = O_RDWR;
+		}
+		ret = pthread_create(&t_prepare[i], NULL,
+			prepare_one_dev, &prepare_ctx[i]);
+	}
+	pthread_join(t_prepare[0], NULL);
+	ret = prepare_ctx[0].ret;
+
 	if (ret)
 		goto error;
+
+	dev_cnt--;
+	fd = open(file, prepare_ctx[0].oflags);
+	dev_block_count = prepare_ctx[0].dev_block_count;
 	if (block_count && block_count > dev_block_count) {
 		error("%s is smaller than requested size, expected %llu, found %llu",
 		      file, (unsigned long long)block_count,
@@ -1459,7 +1511,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 	}
 
 	/* To create the first block group and chunk 0 in make_btrfs */
-	system_group_size = zoned ?  zone_size(file) : BTRFS_MKFS_SYSTEM_GROUP_SIZE;
+	system_group_size = zoned ? zone_size(file) : BTRFS_MKFS_SYSTEM_GROUP_SIZE;
 	if (dev_block_count < system_group_size) {
 		error("device is too small to make filesystem, must be at least %llu",
 				(unsigned long long)system_group_size);
@@ -1557,6 +1609,12 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 	if (dev_cnt == 0)
 		goto raid_groups;
 
+	for (i = 0 ; i < dev_cnt; i++) {
+		pthread_join(t_prepare[i+1], NULL);
+		if (prepare_ctx[i+1].ret) {
+			goto error;
+		}
+	}
 	while (dev_cnt-- > 0) {
 		file = argv[optind++];
 
@@ -1578,12 +1636,9 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 			close(fd);
 			continue;
 		}
-		ret = btrfs_prepare_device(fd, file, &dev_block_count,
-				block_count,
-				(bconf.verbose ? PREP_DEVICE_VERBOSE : 0) |
-				(zero_end ? PREP_DEVICE_ZERO_END : 0) |
-				(discard ? PREP_DEVICE_DISCARD : 0) |
-				(zoned ? PREP_DEVICE_ZONED : 0));
+		dev_block_count = prepare_ctx[argc - saved_optind - dev_cnt - 1]
+			.dev_block_count;
+
 		if (ret) {
 			goto error;
 		}
@@ -1763,12 +1818,16 @@ out:
 
 	btrfs_close_all_devices();
 	free(label);
-
+	free(t_prepare);
+	free(prepare_ctx);
 	return !!ret;
+
 error:
 	if (fd > 0)
 		close(fd);
 
+	free(t_prepare);
+	free(prepare_ctx);
 	free(label);
 	exit(1);
 success:
Signature · 1 line
1.8.3.1
#1 lore raw
Quoted 3 lines · Li Zhang

[enhancement]
When a disk is formatted as btrfs, it calls btrfs_prepare_device for each device, which takes too much time.

The idea is awesome.

Quoted 37 lines

[implementation]
Put each btrfs_prepare_device into a thread, wait for the first thread to complete to mkfs.btrfs, and wait for other threads to complete before adding other devices to the file system.

[test]
Using the btrfs-progs test case mkfs-tests, mkfs.btrfs works fine.

But I don't have an actual zoed device,
so I don't know how much time it saves, If you guys have a way to test it, please let me know.

---
Issue: 496

  mkfs/main.c | 113 +++++++++++++++++++++++++++++++++++++++++++++---------------
  1 file changed, 86 insertions(+), 27 deletions(-)
mkfs/main.c +1 -0
index ce096d3..35fefe2 100644
@@ -31,6 +31,7 @@
  #include <uuid/uuid.h>
  #include <ctype.h>
  #include <blkid/blkid.h>
+#include <pthread.h>
  #include "kernel-shared/ctree.h"
  #include "kernel-shared/disk-io.h"
  #include "kernel-shared/free-space-tree.h"
@@ -60,6 +61,18 @@ struct mkfs_allocation {
  	u64 system;
  };
 

mkfs/main.c struct mkfs_allocation { · line 73

Quoted 10 lines
mkfs/main.c +9 -0
+
+struct prepare_device_progress {
+	char *file;
+	u64 dev_block_count;
+	u64 block_count;
 
+	bool zero_end;
+	bool discard;
+	bool zoned;
+	int oflags;

A small nitpick.

Aren't those 4 values the same shared by all devices?
Thus I'm not sure if they need to be put into prepare_device_progress at all.

IIRC, we may want some shared memory between all the threads:

- A pthread_mutex
Will be explained later

- All the other shared infos like above flags/oflags

It can be global or passed by some pointers.

Quoted 8 lines
mkfs/main.c +3 -0
+	int ret;
+};
+
  static int create_metadata_block_groups(struct btrfs_root *root, bool mixed,
  				struct mkfs_allocation *allocation)
  {
mkfs/main.c +0 -0
@@ -969,6 +982,28 @@ fail:
  	return ret;

mkfs/main.c fail: · line 992

Quoted 10 lines
mkfs/main.c +8 -0
  }
 
+static void *prepare_one_dev(void *ctx)
+{
+	struct prepare_device_progress *prepare_ctx = ctx;
+	int fd;
+
+	fd = open(prepare_ctx->file, prepare_ctx->oflags);
+	if (fd < 0) {
+		error("unable to open %s: %m", prepare_ctx->file);

If we have no permission for all devices (pretty common in fact, e.g. forgot to use sudo), we will have multiple threads printing out the same time.

Without a lock, the output will be a mess.

Thus we may want a mutex, even it's just for synchronizing the output.

Quoted 46 lines
mkfs/main.c +18 -6
+		prepare_ctx->ret = fd;
+		return NULL;
+	}
+	prepare_ctx->ret = btrfs_prepare_device(fd,
+		prepare_ctx->file, &prepare_ctx->dev_block_count,
+		prepare_ctx->block_count,
+		(bconf.verbose ? PREP_DEVICE_VERBOSE : 0) |
+		(prepare_ctx->zero_end ? PREP_DEVICE_ZERO_END : 0) |
+		(prepare_ctx->discard ? PREP_DEVICE_DISCARD : 0) |
+		(prepare_ctx->zoned ? PREP_DEVICE_ZONED : 0));
+	close(fd);
+	return NULL;
+}
+
  int BOX_MAIN(mkfs)(int argc, char **argv)
  {
  	char *file;
@@ -997,7 +1032,6 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
  	bool ssd = false;
  	bool zoned = false;
  	bool force_overwrite = false;
-	int oflags;
  	char *source_dir = NULL;
  	bool source_dir_set = false;
  	bool shrink_rootdir = false;
@@ -1006,6 +1040,8 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
  	u64 shrink_size;
  	int dev_cnt = 0;
  	int saved_optind;
+	pthread_t *t_prepare = NULL;
+	struct prepare_device_progress *prepare_ctx = NULL;
  	char fs_uuid[BTRFS_UUID_UNPARSED_SIZE] = { 0 };
  	u64 features = BTRFS_MKFS_DEFAULT_FEATURES;
  	u64 runtime_features = BTRFS_MKFS_DEFAULT_RUNTIME_FEATURES;
@@ -1428,29 +1464,45 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
  		goto error;
  	}
 
-	dev_cnt--;
-
-	oflags = O_RDWR;
-	if (zoned && zoned_model(file) == ZONED_HOST_MANAGED)
-		oflags |= O_DIRECT;
+	t_prepare = malloc(dev_cnt * sizeof(*t_prepare));
+	prepare_ctx = malloc(dev_cnt * sizeof(*prepare_ctx));
 

mkfs/main.c int BOX_MAIN(mkfs)(int argc, char **argv) · line 1471

Quoted 10 lines
mkfs/main.c +2 -8
-	/*
-	 * Open without O_EXCL so that the problem should not occur by the
-	 * following operation in kernel:
-	 * (btrfs_register_one_device() fails if O_EXCL is on)
-	 */
-	fd = open(file, oflags);
-	if (fd < 0) {
-		error("unable to open %s: %m", file);
+	if (!t_prepare || !prepare_ctx) {
+		error("unable to prepare dev");

Isn't this ENOMEM? The message doesn't seem to match the situation.

Quoted 12 lines
mkfs/main.c +5 -5
  		goto error;
  	}
-	ret = btrfs_prepare_device(fd, file, &dev_block_count, block_count,
-			(zero_end ? PREP_DEVICE_ZERO_END : 0) |
-			(discard ? PREP_DEVICE_DISCARD : 0) |
-			(bconf.verbose ? PREP_DEVICE_VERBOSE : 0) |
-			(zoned ? PREP_DEVICE_ZONED : 0));
+
+	for (i = 0; i < dev_cnt; i++) {
+		prepare_ctx[i].file = argv[optind + i - 1];
+		prepare_ctx[i].block_count = block_count;
+		prepare_ctx[i].dev_block_count = block_count;

mkfs/main.c int BOX_MAIN(mkfs)(int argc, char **argv) · line 1488

Quoted 10 lines
mkfs/main.c +10 -0
+		prepare_ctx[i].zero_end = zero_end;
+		prepare_ctx[i].discard = discard;
+		prepare_ctx[i].zoned = zoned;
+		if (i == 0) {
+			prepare_ctx[i].oflags = O_RDWR;
+			/*
+			 * Open without O_EXCL so that the problem should
+			 * not occur by the following operation in kernel:
+			 * (btrfs_register_one_device() fails if O_EXCL is on)
+			 */

The comment seems out-dated, no O_EXCL involved anywhere.

mkfs/main.c int BOX_MAIN(mkfs)(int argc, char **argv) · line 1490

Quoted 2 lines
mkfs/main.c +2 -0
+			if (zoned && zoned_model(file) == ZONED_HOST_MANAGED)
+				prepare_ctx[i].oflags = O_RDWR | O_DIRECT;

Do we need to treat the initial and other devices differently?

Can't we use the same flags for all devices?

mkfs/main.c int BOX_MAIN(mkfs)(int argc, char **argv)

Quoted 8 lines
mkfs/main.c +8 -0
+		} else {
+			prepare_ctx[i].oflags = O_RDWR;
+		}
+		ret = pthread_create(&t_prepare[i], NULL,
+			prepare_one_dev, &prepare_ctx[i]);
+	}
+	pthread_join(t_prepare[0], NULL);
+	ret = prepare_ctx[0].ret; > +

Can't we just wait for all devices?

I don't think treating them different could have much benefit.

Yes, we can have multiple-devices with different performance characteristics, thus if the first device is the fastest one, it may finish before all the others.

But this also means, the first one can be the slowest.

To me, parallel initialization is already a big enough improvement, and for the most common case, all the devices should have the same or similar performance characteristics, thus waiting for them all shouldn't cause much difference.

Quoted 37 lines
mkfs/main.c +11 -3
  	if (ret)
  		goto error;
+
+	dev_cnt--;
+	fd = open(file, prepare_ctx[0].oflags);
+	dev_block_count = prepare_ctx[0].dev_block_count;
  	if (block_count && block_count > dev_block_count) {
  		error("%s is smaller than requested size, expected %llu, found %llu",
  		      file, (unsigned long long)block_count,
@@ -1459,7 +1511,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
  	}
 
  	/* To create the first block group and chunk 0 in make_btrfs */
-	system_group_size = zoned ?  zone_size(file) : BTRFS_MKFS_SYSTEM_GROUP_SIZE;
+	system_group_size = zoned ? zone_size(file) : BTRFS_MKFS_SYSTEM_GROUP_SIZE;
  	if (dev_block_count < system_group_size) {
  		error("device is too small to make filesystem, must be at least %llu",
  				(unsigned long long)system_group_size);
@@ -1557,6 +1609,12 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
  	if (dev_cnt == 0)
  		goto raid_groups;
 
+	for (i = 0 ; i < dev_cnt; i++) {
+		pthread_join(t_prepare[i+1], NULL);
+		if (prepare_ctx[i+1].ret) {
+			goto error;
+		}
+	}
  	while (dev_cnt-- > 0) {
  		file = argv[optind++];
 
@@ -1578,12 +1636,9 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
  			close(fd);
  			continue;
  		}
-		ret = btrfs_prepare_device(fd, file, &dev_block_count,
-				block_count,

mkfs/main.c int BOX_MAIN(mkfs)(int argc, char **argv) · line 1644

Quoted 10 lines
mkfs/main.c +3 -4
-				(bconf.verbose ? PREP_DEVICE_VERBOSE : 0) |
-				(zero_end ? PREP_DEVICE_ZERO_END : 0) |
-				(discard ? PREP_DEVICE_DISCARD : 0) |
-				(zoned ? PREP_DEVICE_ZONED : 0));
+		dev_block_count = prepare_ctx[argc - saved_optind - dev_cnt - 1]
+			.dev_block_count;
+
  		if (ret) {
  			goto error;
  		}

This goto error is a dead code now.

Thanks for the great idea on reducing the preparation time! Qu

Quoted 18 lines
mkfs/main.c +5 -1
@@ -1763,12 +1818,16 @@ out:
 
  	btrfs_close_all_devices();
  	free(label);
-
+	free(t_prepare);
+	free(prepare_ctx);
  	return !!ret;
+
  error:
  	if (fd > 0)
  		close(fd);
 
+	free(t_prepare);
+	free(prepare_ctx);
  	free(label);
  	exit(1);
  success:
#2 lore raw
Quoted 7 lines · Qu Wenruo

mkfs/main.c int BOX_MAIN(mkfs)(int argc, char **argv) · line 1490

Quoted 2 lines
mkfs/main.c +2 -0
+			if (zoned && zoned_model(file) == ZONED_HOST_MANAGED)
+				prepare_ctx[i].oflags = O_RDWR | O_DIRECT;

Do we need to treat the initial and other devices differently?

Can't we use the same flags for all devices?

Yep we need to have the same flags for all devices. Otherwise only device 0 will be opened with O_DIRECT, in case of a host-managed one and the subsequent will be opened without O_DIRECT causing mkfs to fail.

#10 lore raw
In reply to Li Zhang
Quoted 3 lines · Li Zhang

[enhancement]
When a disk is formatted as btrfs, it calls btrfs_prepare_device for each device, which takes too much time.

That really is awesome. I'll throw it onto my 60 zoned HDD test box, once all devices have the same open flags.

[...]

mkfs/main.c int BOX_MAIN(mkfs)(int argc, char **argv) · line 1468

Quoted 2 lines
mkfs/main.c +2 -0
+	t_prepare = malloc(dev_cnt * sizeof(*t_prepare));
+	prepare_ctx = malloc(dev_cnt * sizeof(*prepare_ctx));

That really should be calloc().

#3 lore raw
In reply to Johannes Thumshirn
Quoted 5 lines · Johannes Thumshirn
Quoted 4 lines · Qu Wenruo

mkfs/main.c int BOX_MAIN(mkfs)(int argc, char **argv) · line 1490

Quoted 2 lines
mkfs/main.c +2 -0
+			if (zoned && zoned_model(file) == ZONED_HOST_MANAGED)
+				prepare_ctx[i].oflags = O_RDWR | O_DIRECT;

Do we need to treat the initial and other devices differently?

Quoted 7 lines
Quoted 3 lines

Can't we use the same flags for all devices?

Yep we need to have the same flags for all devices. Otherwise only device 0 will be opened with O_DIRECT, in case of a host-managed one and the subsequent will be opened without O_DIRECT causing mkfs to fail.

Just a little curious, currently btrfs doesn't support mixed traditional/zoned devices, right?

So that O_DIRECT for all devices are for future mixed zoned mode?

Anyway I'm completely fine if we can use the same oflags for all devices.

Thanks,
Qu

#4 lore raw
Quoted 12 lines · Qu Wenruo
Quoted 9 lines · Johannes Thumshirn
Quoted 7 lines · Qu Wenruo

mkfs/main.c int BOX_MAIN(mkfs)(int argc, char **argv) · line 1490

Quoted 2 lines
mkfs/main.c +2 -0
+			if (zoned && zoned_model(file) == ZONED_HOST_MANAGED)
+				prepare_ctx[i].oflags = O_RDWR | O_DIRECT;

Do we need to treat the initial and other devices differently?

Can't we use the same flags for all devices?

Quoted 8 lines
Quoted 3 lines

Yep we need to have the same flags for all devices. Otherwise only device 0 will be opened with O_DIRECT, in case of a host-managed one and the subsequent will be opened without O_DIRECT causing mkfs to fail.

Just a little curious, currently btrfs doesn't support mixed traditional/zoned devices, right?

So that O_DIRECT for all devices are for future mixed zoned mode?

We need it in case of multiple zoned devices as well. The mixed mode you describe above could actually work thanks to the zoned emulation we have in place. But I've never actually tried to be honest.

author
#5 lore raw
In reply to Johannes Thumshirn

Hi, I'm a bit confused, do you mean if you open a zoned device without O_DIRECT it will fail?

I tested and found that if I open a device with the O_DIRECT flag on a virtual device like a loop device, the device cannot be written to, but with or without O_DIRECT, it works fine on a real device (for me, I only test A normal block device since I don't have any zoned devices)

If we use the same flags for all devices, does that mean we can't use mkfs.btrfs
on both real and virtual devices at the same time.

Below is my test program and test results.

code(main idea):
printf("filename:%s.\n", argv[1]);
int fd = open(argv[1], O_RDWR | O_DIRECT);

if (fd < 0) {
     printf("fd:error.\n");
     return -1;
}

int num = write(fd, "123", 3);
printf("num:%d.\n", num);
close(fd);

result:

$ sudo losetup /dev/loop1 loopDev/loop1
$ sudo ./a.out /dev/loop1
filename:/dev/loop1.
num:-1.
# cannot write to loop1

Thanks,
Li Zhang

Quoted 13 lines · Johannes Thumshirn <[email protected]> 于2022年8月25日周四 16:31写道:
Quoted 7 lines · Qu Wenruo

mkfs/main.c int BOX_MAIN(mkfs)(int argc, char **argv) · line 1490

Quoted 2 lines
mkfs/main.c +2 -0
+                    if (zoned && zoned_model(file) == ZONED_HOST_MANAGED)
+                            prepare_ctx[i].oflags = O_RDWR | O_DIRECT;

Do we need to treat the initial and other devices differently?

Can't we use the same flags for all devices?

Yep we need to have the same flags for all devices. Otherwise only device 0 will be opened with O_DIRECT, in case of a host-managed one and the subsequent will be opened without O_DIRECT causing mkfs to fail.

#6 lore raw
Quoted 2 lines · li zhang

Hi, I'm a bit confused, do you mean if you open a zoned device without O_DIRECT it will fail?

Not a zoned device expert, but to my understanding, if we write into zoned device, without O_DIRECT, there is no guarantee that the data you submitted will end at the same bytenr you specified.

E.g. if you do a pwrite() with a 1M buffer, at device bytenr 4M.

Without O_DIRECT, the zoned code can re-locate the bytenr to any range after the write pointer inside the same zone.

AKA, for zoned device, without O_DIRECT (queue length 1), you can only known the real physical bytenr after the write has fully finished.

(The final physical bytenr is determined by the zoned device, no longer the write initiator).

Quoted 6 lines

I tested and found that if I open a device with the O_DIRECT flag on a virtual device like a loop device, the device cannot be written to, but with or without O_DIRECT, it works fine on a real device (for me, I only test A normal block device since I don't have any zoned devices)

IIRC currently there is no zoned emulation for loop device.

If you want to test zoned device, you can use null block kernel module, with fully memory backed storage:

https://zonedstorage.io/docs/getting-started/nullblk

Or go a little further, using tcmu-runner to create file backed zoned device:

https://zonedstorage.io/docs/tools/tcmu-runner

Quoted 11 lines

If we use the same flags for all devices, does that mean we can't use mkfs.btrfs
on both real and virtual devices at the same time.

Below is my test program and test results.

code(main idea):
printf("filename:%s.\n", argv[1]);
int fd = open(argv[1], O_RDWR | O_DIRECT);

Quoted 6 lines
if (fd < 0) {
      printf("fd:error.\n");
      return -1;
}

int num = write(fd, "123", 3);
printf("num:%d.\n", num);

O_DIRECT requires strict memory alignment, obviously the length 3 is not properly aligned.

Please check open(2p) for the full requirement.

For mkfs usage, all of our write is at least 4K aligned, thus O_DIRECT can work correctly.

Back to btrfs-progs work, I'd say before we do anything, let's check all the devices passed in to determine if we want zoned mode (any zoned device should make it zoned).

Then we can determine the open flags for all devices, and for regular devices, O_DIRECT mostly makes no difference (maybe a little slower, but may not even be observable).

Thanks,
Qu

Quoted 27 lines

close(fd);

result:

$ sudo losetup /dev/loop1 loopDev/loop1
$ sudo ./a.out /dev/loop1
filename:/dev/loop1.
num:-1.
# cannot write to loop1

Thanks,
Li Zhang

Quoted 13 lines · Johannes Thumshirn <[email protected]> 于2022年8月25日周四 16:31写道:
Quoted 7 lines · Qu Wenruo

mkfs/main.c int BOX_MAIN(mkfs)(int argc, char **argv) · line 1490

Quoted 2 lines
mkfs/main.c +2 -0
+                    if (zoned && zoned_model(file) == ZONED_HOST_MANAGED)
+                            prepare_ctx[i].oflags = O_RDWR | O_DIRECT;

Do we need to treat the initial and other devices differently?

Can't we use the same flags for all devices?

Yep we need to have the same flags for all devices. Otherwise only device 0 will be opened with O_DIRECT, in case of a host-managed one and the subsequent will be opened without O_DIRECT causing mkfs to fail.

author
#7 lore raw

Yes, I see what you mean.

There is no doubt that the loop device is not a zone device.
I simulated the zone device with the null_blk module and tested mkfs.btrfs, but an error was reported. In addition, Not only mkfs.btrfs does not work on null_blk zoned devices, mkfs.xfs and mkfs.ext2 also do not work on null_blk zoned devices, here is the test log. My first instinct is
the null_blk problem . But I didn't test tcmu-runner, I'll dig into it later anyway.

#emulate zoned device using null_blk

$ sudo modprobe null_blk nr_devices=4 zoned=1

#mkfs.xfs failed

$ sudo mkfs.xfs -V
mkfs.xfs version 5.18.0
$ sudo mkfs.xfs /dev/nullb0 -f
meta-data=/dev/nullb0            isize=512    agcount=4, agsize=16384000 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1
data     =                       bsize=4096   blocks=65536000, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=32000, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
mkfs.xfs: pwrite failed: Input/output error
libxfs_bwrite: write failed on (unknown) bno 0x1f3fff00/0x100, err=5
mkfs.xfs: Releasing dirty buffer to free list!
found dirty buffer (bulk) on free list!
mkfs.xfs: pwrite failed: Input/output error
libxfs_bwrite: write failed on (unknown) bno 0x0/0x100, err=5
mkfs.xfs: Releasing dirty buffer to free list!
found dirty buffer (bulk) on free list!
mkfs.xfs: pwrite failed: Input/output error
libxfs_bwrite: write failed on xfs_sb bno 0x0/0x1, err=5
mkfs.xfs: Releasing dirty buffer to free list!
mkfs.xfs: libxfs_device_zero write failed: Input/output error

#mkfs.btrfs failed

$ sudo mkfs.btrfs --version
mkfs.btrfs, part of btrfs-progs v5.19
$ sudo mkfs.btrfs -d single -m single -O zoned /dev/nullb0 /dev/nullb1
/dev/nullb2 -f
btrfs-progs v5.19
See http://btrfs.wiki.kernel.org for more information.

Resetting device zones /dev/nullb0 (1000 zones) ...
Resetting device zones /dev/nullb2 (1000 zones) ...
Resetting device zones /dev/nullb1 (1000 zones) ...
NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)

No valid Btrfs found on /dev/nullb0
ERROR: open ctree failed

#mkfs.ext2 failed

$ sudo mke2fs -V
mke2fs 1.46.5 (30-Dec-2021)
Using EXT2FS Library version 1.46.5
$ sudo mke2fs /dev/nullb0
mke2fs 1.46.5 (30-Dec-2021)
Creating filesystem with 65536000 4k blocks and 16384000 inodes
Filesystem UUID: 747350a2-a1d5-4944-9f46-0fe4ca76df9d
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: mke2fs:
Input/output error while writing out and closing file system

thanks,
Li Zhang

Quoted 108 lines · Qu Wenruo <[email protected]> 于2022年8月28日周日 17:54写道:
Quoted 2 lines · li zhang

Hi, I'm a bit confused, do you mean if you open a zoned device without O_DIRECT it will fail?

Not a zoned device expert, but to my understanding, if we write into zoned device, without O_DIRECT, there is no guarantee that the data you submitted will end at the same bytenr you specified.

E.g. if you do a pwrite() with a 1M buffer, at device bytenr 4M.

Without O_DIRECT, the zoned code can re-locate the bytenr to any range after the write pointer inside the same zone.

AKA, for zoned device, without O_DIRECT (queue length 1), you can only known the real physical bytenr after the write has fully finished.

(The final physical bytenr is determined by the zoned device, no longer the write initiator).

Quoted 6 lines

I tested and found that if I open a device with the O_DIRECT flag on a virtual device like a loop device, the device cannot be written to, but with or without O_DIRECT, it works fine on a real device (for me, I only test A normal block device since I don't have any zoned devices)

IIRC currently there is no zoned emulation for loop device.

If you want to test zoned device, you can use null block kernel module, with fully memory backed storage:

https://zonedstorage.io/docs/getting-started/nullblk

Or go a little further, using tcmu-runner to create file backed zoned device:

https://zonedstorage.io/docs/tools/tcmu-runner

Quoted 11 lines

If we use the same flags for all devices, does that mean we can't use mkfs.btrfs
on both real and virtual devices at the same time.

Below is my test program and test results.

code(main idea):
printf("filename:%s.\n", argv[1]);
int fd = open(argv[1], O_RDWR | O_DIRECT);

Quoted 6 lines
if (fd < 0) {
      printf("fd:error.\n");
      return -1;
}

int num = write(fd, "123", 3);
printf("num:%d.\n", num);

O_DIRECT requires strict memory alignment, obviously the length 3 is not properly aligned.

Please check open(2p) for the full requirement.

For mkfs usage, all of our write is at least 4K aligned, thus O_DIRECT can work correctly.

Back to btrfs-progs work, I'd say before we do anything, let's check all the devices passed in to determine if we want zoned mode (any zoned device should make it zoned).

Then we can determine the open flags for all devices, and for regular devices, O_DIRECT mostly makes no difference (maybe a little slower, but may not even be observable).

Thanks,
Qu

Quoted 27 lines

close(fd);

result:

$ sudo losetup /dev/loop1 loopDev/loop1
$ sudo ./a.out /dev/loop1
filename:/dev/loop1.
num:-1.
# cannot write to loop1

Thanks,
Li Zhang

Quoted 13 lines · Johannes Thumshirn <[email protected]> 于2022年8月25日周四 16:31写道:
Quoted 7 lines · Qu Wenruo

mkfs/main.c int BOX_MAIN(mkfs)(int argc, char **argv) · line 1490

Quoted 2 lines
mkfs/main.c +2 -0
+                    if (zoned && zoned_model(file) == ZONED_HOST_MANAGED)
+                            prepare_ctx[i].oflags = O_RDWR | O_DIRECT;

Do we need to treat the initial and other devices differently?

Can't we use the same flags for all devices?

Yep we need to have the same flags for all devices. Otherwise only device 0 will be opened with O_DIRECT, in case of a host-managed one and the subsequent will be opened without O_DIRECT causing mkfs to fail.

author
#8 lore raw

By the way, my kernel version is 5.19.0, and I also tested the 5.0 version (maybe, I only remember that the version starts with 5), the same error output

thanks,
Li Zhang

Quoted 194 lines · li zhang <[email protected]> 于2022年8月28日周日 22:26写道:

Yes, I see what you mean.

There is no doubt that the loop device is not a zone device.
I simulated the zone device with the null_blk module and tested mkfs.btrfs, but an error was reported. In addition, Not only mkfs.btrfs does not work on null_blk zoned devices, mkfs.xfs and mkfs.ext2 also do not work on null_blk zoned devices, here is the test log. My first instinct is
the null_blk problem . But I didn't test tcmu-runner, I'll dig into it later anyway.

#emulate zoned device using null_blk

$ sudo modprobe null_blk nr_devices=4 zoned=1

#mkfs.xfs failed

$ sudo mkfs.xfs -V
mkfs.xfs version 5.18.0
$ sudo mkfs.xfs /dev/nullb0 -f
meta-data=/dev/nullb0            isize=512    agcount=4, agsize=16384000 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1
data     =                       bsize=4096   blocks=65536000, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=32000, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
mkfs.xfs: pwrite failed: Input/output error
libxfs_bwrite: write failed on (unknown) bno 0x1f3fff00/0x100, err=5
mkfs.xfs: Releasing dirty buffer to free list!
found dirty buffer (bulk) on free list!
mkfs.xfs: pwrite failed: Input/output error
libxfs_bwrite: write failed on (unknown) bno 0x0/0x100, err=5
mkfs.xfs: Releasing dirty buffer to free list!
found dirty buffer (bulk) on free list!
mkfs.xfs: pwrite failed: Input/output error
libxfs_bwrite: write failed on xfs_sb bno 0x0/0x1, err=5
mkfs.xfs: Releasing dirty buffer to free list!
mkfs.xfs: libxfs_device_zero write failed: Input/output error

#mkfs.btrfs failed

$ sudo mkfs.btrfs --version
mkfs.btrfs, part of btrfs-progs v5.19
$ sudo mkfs.btrfs -d single -m single -O zoned /dev/nullb0 /dev/nullb1
/dev/nullb2 -f
btrfs-progs v5.19
See http://btrfs.wiki.kernel.org for more information.

Resetting device zones /dev/nullb0 (1000 zones) ...
Resetting device zones /dev/nullb2 (1000 zones) ...
Resetting device zones /dev/nullb1 (1000 zones) ...
NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)

No valid Btrfs found on /dev/nullb0
ERROR: open ctree failed

#mkfs.ext2 failed

$ sudo mke2fs -V
mke2fs 1.46.5 (30-Dec-2021)
Using EXT2FS Library version 1.46.5
$ sudo mke2fs /dev/nullb0
mke2fs 1.46.5 (30-Dec-2021)
Creating filesystem with 65536000 4k blocks and 16384000 inodes
Filesystem UUID: 747350a2-a1d5-4944-9f46-0fe4ca76df9d
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: mke2fs:
Input/output error while writing out and closing file system

thanks,
Li Zhang

Quoted 108 lines · Qu Wenruo <[email protected]> 于2022年8月28日周日 17:54写道:
Quoted 2 lines · li zhang

Hi, I'm a bit confused, do you mean if you open a zoned device without O_DIRECT it will fail?

Not a zoned device expert, but to my understanding, if we write into zoned device, without O_DIRECT, there is no guarantee that the data you submitted will end at the same bytenr you specified.

E.g. if you do a pwrite() with a 1M buffer, at device bytenr 4M.

Without O_DIRECT, the zoned code can re-locate the bytenr to any range after the write pointer inside the same zone.

AKA, for zoned device, without O_DIRECT (queue length 1), you can only known the real physical bytenr after the write has fully finished.

(The final physical bytenr is determined by the zoned device, no longer the write initiator).

Quoted 6 lines

I tested and found that if I open a device with the O_DIRECT flag on a virtual device like a loop device, the device cannot be written to, but with or without O_DIRECT, it works fine on a real device (for me, I only test A normal block device since I don't have any zoned devices)

IIRC currently there is no zoned emulation for loop device.

If you want to test zoned device, you can use null block kernel module, with fully memory backed storage:

https://zonedstorage.io/docs/getting-started/nullblk

Or go a little further, using tcmu-runner to create file backed zoned device:

https://zonedstorage.io/docs/tools/tcmu-runner

Quoted 11 lines

If we use the same flags for all devices, does that mean we can't use mkfs.btrfs
on both real and virtual devices at the same time.

Below is my test program and test results.

code(main idea):
printf("filename:%s.\n", argv[1]);
int fd = open(argv[1], O_RDWR | O_DIRECT);

Quoted 6 lines
if (fd < 0) {
      printf("fd:error.\n");
      return -1;
}

int num = write(fd, "123", 3);
printf("num:%d.\n", num);

O_DIRECT requires strict memory alignment, obviously the length 3 is not properly aligned.

Please check open(2p) for the full requirement.

For mkfs usage, all of our write is at least 4K aligned, thus O_DIRECT can work correctly.

Back to btrfs-progs work, I'd say before we do anything, let's check all the devices passed in to determine if we want zoned mode (any zoned device should make it zoned).

Then we can determine the open flags for all devices, and for regular devices, O_DIRECT mostly makes no difference (maybe a little slower, but may not even be observable).

Thanks,
Qu

Quoted 27 lines

close(fd);

result:

$ sudo losetup /dev/loop1 loopDev/loop1
$ sudo ./a.out /dev/loop1
filename:/dev/loop1.
num:-1.
# cannot write to loop1

Thanks,
Li Zhang

Quoted 13 lines · Johannes Thumshirn <[email protected]> 于2022年8月25日周四 16:31写道:
Quoted 7 lines · Qu Wenruo

mkfs/main.c int BOX_MAIN(mkfs)(int argc, char **argv) · line 1490

Quoted 2 lines
mkfs/main.c +2 -0
+                    if (zoned && zoned_model(file) == ZONED_HOST_MANAGED)
+                            prepare_ctx[i].oflags = O_RDWR | O_DIRECT;

Do we need to treat the initial and other devices differently?

Can't we use the same flags for all devices?

Yep we need to have the same flags for all devices. Otherwise only device 0 will be opened with O_DIRECT, in case of a host-managed one and the subsequent will be opened without O_DIRECT causing mkfs to fail.

#9 lore raw
In reply to li zhang
Quoted 10 lines · li zhang

Yes, I see what you mean.

There is no doubt that the loop device is not a zone device.
I simulated the zone device with the null_blk module and tested mkfs.btrfs, but an error was reported. In addition, Not only mkfs.btrfs does not work on null_blk zoned devices, mkfs.xfs and mkfs.ext2 also do not work on null_blk zoned devices, here is the test log. My first instinct is
the null_blk problem . But I didn't test tcmu-runner, I'll dig into it later anyway.

Please get an overview of what zoned device can and can not in the first place:

https://zonedstorage.io/docs/introduction/zoned-storage

In short, for zoned device it can not do any overwrite.

Johannes, please correct me if I'm wrong, it's only allowed to submit write which bytenr is at (or beyond?) the write pointer inside a zone.

Thus that's why there are only very limited filesystems supporting zoned device for now.

For current btrfs, we have mandatory metadata COW, thus can ensure all our metadata are allocated in ascending bytenr, and uses queue depth 1 to make sure all our metadata can be written exactly where we specify.

For btrfs data, we let the zoned device to decide where the data should be, and record the new bytenr returned by the zoned device into our metadata (and follow above metadata write behavior to write them).

For btrfs super blocks, there are two (?) dedicated zones for superblocks, we write super blocks into one zone like a ring buffer. (Thus at mount we need to read the whole zone to find the newest copy)

So mkfs.xfs is *supposed* to fail, that's nothing new. There are tons of things which can lead to write before the write pointer, like to update the super block.

Quoted 23 lines

#emulate zoned device using null_blk

$ sudo modprobe null_blk nr_devices=4 zoned=1

#mkfs.xfs failed

$ sudo mkfs.xfs -V
mkfs.xfs version 5.18.0
$ sudo mkfs.xfs /dev/nullb0 -f
meta-data=/dev/nullb0            isize=512    agcount=4, agsize=16384000 blks
          =                       sectsz=512   attr=2, projid32bit=1
          =                       crc=1        finobt=1, sparse=1, rmapbt=0
          =                       reflink=1    bigtime=1 inobtcount=1
data     =                       bsize=4096   blocks=65536000, imaxpct=25
          =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=32000, version=2
          =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
mkfs.xfs: pwrite failed: Input/output error
libxfs_bwrite: write failed on (unknown) bno 0x1f3fff00/0x100, err=5
mkfs.xfs: Releasing dirty buffer to free list!
found dirty buffer (bulk) on free list!
Quoted 8 lines
mkfs.xfs: pwrite failed: Input/output error
libxfs_bwrite: write failed on (unknown) bno 0x0/0x100, err=5
mkfs.xfs: Releasing dirty buffer to free list!
found dirty buffer (bulk) on free list!
mkfs.xfs: pwrite failed: Input/output error
libxfs_bwrite: write failed on xfs_sb bno 0x0/0x1, err=5
mkfs.xfs: Releasing dirty buffer to free list!
mkfs.xfs: libxfs_device_zero write failed: Input/output error

That's expected.

Quoted 18 lines

#mkfs.btrfs failed

$ sudo mkfs.btrfs --version
mkfs.btrfs, part of btrfs-progs v5.19
$ sudo mkfs.btrfs -d single -m single -O zoned /dev/nullb0 /dev/nullb1
/dev/nullb2 -f
btrfs-progs v5.19
See http://btrfs.wiki.kernel.org for more information.

Resetting device zones /dev/nullb0 (1000 zones) ...
Resetting device zones /dev/nullb2 (1000 zones) ...
Resetting device zones /dev/nullb1 (1000 zones) ...
NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)

Quoted 1 line

No valid Btrfs found on /dev/nullb0

This looks like you're using null_blk in discard mode (aka, all writes are just discarded).

You need to specify the memory_backed param to let it remember what you have written.

To Johannes, maybe you want to update the null_blk page to specify the memory_backed param?

With that specified, it works fine in my test env:

 # modprobe null_blk nr_devices=1 zoned=1 zone_size=128 gb=1
memory_backed=1
 # mkfs.btrfs  -f /dev/nullb0 -m single -d single
btrfs-progs v5.18.1
See http://btrfs.wiki.kernel.org for more information.

Zoned: /dev/nullb0: host-managed device detected, setting zoned feature Resetting device zones /dev/nullb0 (8 zones) ...
NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)

Label:              (null)
UUID:               d75978cc-cfff-4acd-abb3-5f8023d4f12f
Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single          128.00MiB
  Metadata:         single          128.00MiB
  System:           single          128.00MiB
SSD detected:       yes
Zoned device:       yes
  Zone size:        128.00MiB
Incompat features:  extref, skinny-metadata, no-holes, zoned
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  /dev/nullb0
# mount /dev/nullb0  /mnt/btrfs/

Thanks,
Qu

Quoted 133 lines

ERROR: open ctree failed

#mkfs.ext2 failed

$ sudo mke2fs -V
mke2fs 1.46.5 (30-Dec-2021)
Using EXT2FS Library version 1.46.5
$ sudo mke2fs /dev/nullb0
mke2fs 1.46.5 (30-Dec-2021)
Creating filesystem with 65536000 4k blocks and 16384000 inodes
Filesystem UUID: 747350a2-a1d5-4944-9f46-0fe4ca76df9d
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: mke2fs:
Input/output error while writing out and closing file system

thanks,
Li Zhang

Quoted 108 lines · Qu Wenruo <[email protected]> 于2022年8月28日周日 17:54写道:
Quoted 2 lines · li zhang

Hi, I'm a bit confused, do you mean if you open a zoned device without O_DIRECT it will fail?

Not a zoned device expert, but to my understanding, if we write into zoned device, without O_DIRECT, there is no guarantee that the data you submitted will end at the same bytenr you specified.

E.g. if you do a pwrite() with a 1M buffer, at device bytenr 4M.

Without O_DIRECT, the zoned code can re-locate the bytenr to any range after the write pointer inside the same zone.

AKA, for zoned device, without O_DIRECT (queue length 1), you can only known the real physical bytenr after the write has fully finished.

(The final physical bytenr is determined by the zoned device, no longer the write initiator).

Quoted 6 lines

I tested and found that if I open a device with the O_DIRECT flag on a virtual device like a loop device, the device cannot be written to, but with or without O_DIRECT, it works fine on a real device (for me, I only test A normal block device since I don't have any zoned devices)

IIRC currently there is no zoned emulation for loop device.

If you want to test zoned device, you can use null block kernel module, with fully memory backed storage:

https://zonedstorage.io/docs/getting-started/nullblk

Or go a little further, using tcmu-runner to create file backed zoned device:

https://zonedstorage.io/docs/tools/tcmu-runner

Quoted 11 lines

If we use the same flags for all devices, does that mean we can't use mkfs.btrfs
on both real and virtual devices at the same time.

Below is my test program and test results.

code(main idea):
printf("filename:%s.\n", argv[1]);
int fd = open(argv[1], O_RDWR | O_DIRECT);

Quoted 6 lines
if (fd < 0) {
       printf("fd:error.\n");
       return -1;
}

int num = write(fd, "123", 3);
printf("num:%d.\n", num);

O_DIRECT requires strict memory alignment, obviously the length 3 is not properly aligned.

Please check open(2p) for the full requirement.

For mkfs usage, all of our write is at least 4K aligned, thus O_DIRECT can work correctly.

Back to btrfs-progs work, I'd say before we do anything, let's check all the devices passed in to determine if we want zoned mode (any zoned device should make it zoned).

Then we can determine the open flags for all devices, and for regular devices, O_DIRECT mostly makes no difference (maybe a little slower, but may not even be observable).

Thanks,
Qu

Quoted 27 lines

close(fd);

result:

$ sudo losetup /dev/loop1 loopDev/loop1
$ sudo ./a.out /dev/loop1
filename:/dev/loop1.
num:-1.
# cannot write to loop1

Thanks,
Li Zhang

Quoted 13 lines · Johannes Thumshirn <[email protected]> 于2022年8月25日周四 16:31写道:
Quoted 7 lines · Qu Wenruo

mkfs/main.c int BOX_MAIN(mkfs)(int argc, char **argv) · line 1490

Quoted 2 lines
mkfs/main.c +2 -0
+                    if (zoned && zoned_model(file) == ZONED_HOST_MANAGED)
+                            prepare_ctx[i].oflags = O_RDWR | O_DIRECT;

Do we need to treat the initial and other devices differently?

Can't we use the same flags for all devices?

Yep we need to have the same flags for all devices. Otherwise only device 0 will be opened with O_DIRECT, in case of a host-managed one and the subsequent will be opened without O_DIRECT causing mkfs to fail.