btrfs: add test for encoded reads
PATCH 1/1 btrfs: add test for encoded reads
Add btrfs/333 and its helper programs btrfs_encoded_read and btrfs_encoded_write, in order to test encoded reads.
We use the BTRFS_IOC_ENCODED_WRITE ioctl to write random data into a compressed extent, then use the BTRFS_IOC_ENCODED_READ ioctl to check that it matches what we've written. If the new io_uring interface for encoded reads is supported, we also check that that matches the ioctl.
Note that what we write isn't valid compressed data, so any non-encoded reads on these files will fail.
--- .gitignore | 2 + src/Makefile | 3 +- src/btrfs_encoded_read.c | 175 ++++++++++++++++++++++++++++++ src/btrfs_encoded_write.c | 206 +++++++++++++++++++++++++++++++++++ tests/btrfs/333 | 220 ++++++++++++++++++++++++++++++++++++++ tests/btrfs/333.out | 2 + 6 files changed, 607 insertions(+), 1 deletion(-) create mode 100644 src/btrfs_encoded_read.c create mode 100644 src/btrfs_encoded_write.c create mode 100755 tests/btrfs/333 create mode 100644 tests/btrfs/333.out
.gitignore +2 -0
@@ -62,6 +62,8 @@ tags /src/attr_replace_test /src/attr-list-by-handle-cursor-test /src/bstat +/src/btrfs_encoded_read +/src/btrfs_encoded_write /src/bulkstat_null_ocount /src/bulkstat_unlink_test /src/bulkstat_unlink_test_modified
src/Makefile +2 -1
@@ -34,7 +34,8 @@ LINUX_TARGETS = xfsctl bstat t_mtab getdevicesize preallo_rw_pattern_reader \ attr_replace_test swapon mkswap t_attr_corruption t_open_tmpfiles \ fscrypt-crypt-util bulkstat_null_ocount splice-test chprojid_fail \ detached_mounts_propagation ext4_resize t_readdir_3 splice2pipe \ - uuid_ioctl t_snapshot_deleted_subvolume fiemap-fault min_dio_alignment + uuid_ioctl t_snapshot_deleted_subvolume fiemap-fault min_dio_alignment \ + btrfs_encoded_read btrfs_encoded_write EXTRA_EXECS = dmerror fill2attr fill2fs fill2fs_check scaleread.sh \ btrfs_crc32c_forged_name.py popdir.pl popattr.py \
src/btrfs_encoded_read.c new file +175 -0
175 lines not on this page · raw message
src/btrfs_encoded_write.c new file +206 -0
206 lines not on this page · raw message
tests/btrfs/333 new file +220 -0
220 lines not on this page · raw message
tests/btrfs/333.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 333 +Silence is golden
Signature · 1 line
2.45.2
Quoted 60 lines · Mark Harmstone
Add btrfs/333 and its helper programs
btrfs_encoded_readandbtrfs_encoded_write, in order to test encoded reads.We use the
BTRFS_IOC_ENCODED_WRITEioctl to write random data into a compressed extent, then use theBTRFS_IOC_ENCODED_READioctl to check that it matches what we've written. If the newio_uringinterface for encoded reads is supported, we also check that that matches the ioctl.Note that what we write isn't valid compressed data, so any non-encoded reads on these files will fail.
Signed-off-by Mark Harmstone <[email protected]>--- .gitignore | 2 + src/Makefile | 3 +- src/btrfs_encoded_read.c | 175 ++++++++++++++++++++++++++++++ src/btrfs_encoded_write.c | 206 +++++++++++++++++++++++++++++++++++ tests/btrfs/333 | 220 ++++++++++++++++++++++++++++++++++++++ tests/btrfs/333.out | 2 + 6 files changed, 607 insertions(+), 1 deletion(-)create mode 100644
src/btrfs_encoded_read.c
create mode 100644src/btrfs_encoded_write.c
create mode 100755 tests/btrfs/333
create mode 100644 tests/btrfs/333.out.gitignore +2
-0@@ -62,6 +62,8 @@ tags /src/attr_replace_test /src/attr-list-by-handle-cursor-test /src/bstat +/src/btrfs_encoded_read +/src/btrfs_encoded_write /src/bulkstat_null_ocount /src/bulkstat_unlink_test /src/bulkstat_unlink_test_modifiedsrc/Makefile +2
-1@@ -34,7 +34,8 @@ LINUX_TARGETS = xfsctl bstat t_mtab getdevicesize preallo_rw_pattern_reader \ attr_replace_test swapon mkswap t_attr_corruption t_open_tmpfiles \ fscrypt-crypt-util bulkstat_null_ocount splice-test chprojid_fail \ detached_mounts_propagation ext4_resize t_readdir_3 splice2pipe \ - uuid_ioctl t_snapshot_deleted_subvolume fiemap-fault min_dio_alignment + uuid_ioctl t_snapshot_deleted_subvolume fiemap-fault min_dio_alignment \ + btrfs_encoded_read btrfs_encoded_write EXTRA_EXECS = dmerror fill2attr fill2fs fill2fs_check scaleread.sh \ btrfs_crc32c_forged_name.py popdir.pl popattr.py \src/btrfs_encoded_read.c new file +2
-0@@ -0,0 +1,175 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) Meta Platforms, Inc. and affiliates.
src/btrfs_encoded_read.c line 12
Quoted 10 lines
src/btrfs_encoded_read.c new file +10
-0+ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <errno.h> +#include <fcntl.h> +#include <unistd.h> +#include <sys/uio.h> +#include <sys/ioctl.h> +#include <linux/btrfs.h>
For this I need
+#include <linux/io_uring.h>
otherwise I get:
[CC] btrfs_encoded_read/bin/sh ../libtool --quiet --tag=CC --mode=link /usr/bin/gcc-13 btrfs_encoded_read.c -o btrfs_encoded_read -g -O2 -g -O2 -DDEBUG -I../include -DVERSION=\"1.1.1\" -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -funsigned-char -fno-strict-aliasing -Wall -DHAVE_FALLOCATE
-DNEED_INTERNAL_XFS_IOC_EXCHANGE_RANGE -lhandle -lacl -lpthread -lrt
-luuid -lgdbm_compat -lgdbm -laio
-luring ../lib/libtest.labtrfs_encoded_read.c: In function 'encoded_read_io_uring':
btrfs_encoded_read.c:100:26: error: 'IORING_OP_URING_CMD' undeclared (first use in this function); did you mean 'IORING_OP_LINKAT'?
100 | io_uring_prep_rw(IORING_OP_URING_CMD, sqe, fd, &enc,
sizeof(enc), 0);
| ^~~~~~~~~~~~~~~~~~~
| IORING_OP_LINKATbtrfs_encoded_read.c:100:26: note: each undeclared identifier is reported only once for each function it appears in btrfs_encoded_read.c:101:12: error: 'struct io_uring_sqe' has no member named 'cmd_op'
101 | sqe->cmd_op = BTRFS_IOC_ENCODED_READ;
| ^~
during compilation.
Not sure if a ./configure macro thingy should/would solve this.
Quoted 98 lines · Johannes Thumshirn
Quoted 1 line
Quoted 60 lines · Mark Harmstone
Add btrfs/333 and its helper programs
btrfs_encoded_readandbtrfs_encoded_write, in order to test encoded reads.We use the
BTRFS_IOC_ENCODED_WRITEioctl to write random data into a compressed extent, then use theBTRFS_IOC_ENCODED_READioctl to check that it matches what we've written. If the newio_uringinterface for encoded reads is supported, we also check that that matches the ioctl.Note that what we write isn't valid compressed data, so any non-encoded reads on these files will fail.
Signed-off-by Mark Harmstone <[email protected]>--- .gitignore | 2 + src/Makefile | 3 +- src/btrfs_encoded_read.c | 175 ++++++++++++++++++++++++++++++ src/btrfs_encoded_write.c | 206 +++++++++++++++++++++++++++++++++++ tests/btrfs/333 | 220 ++++++++++++++++++++++++++++++++++++++ tests/btrfs/333.out | 2 + 6 files changed, 607 insertions(+), 1 deletion(-)create mode 100644
src/btrfs_encoded_read.c
create mode 100644src/btrfs_encoded_write.c
create mode 100755 tests/btrfs/333
create mode 100644 tests/btrfs/333.out.gitignore +2
-0@@ -62,6 +62,8 @@ tags /src/attr_replace_test /src/attr-list-by-handle-cursor-test /src/bstat +/src/btrfs_encoded_read +/src/btrfs_encoded_write /src/bulkstat_null_ocount /src/bulkstat_unlink_test /src/bulkstat_unlink_test_modifiedsrc/Makefile +2
-1@@ -34,7 +34,8 @@ LINUX_TARGETS = xfsctl bstat t_mtab getdevicesize preallo_rw_pattern_reader \ attr_replace_test swapon mkswap t_attr_corruption t_open_tmpfiles \ fscrypt-crypt-util bulkstat_null_ocount splice-test chprojid_fail \ detached_mounts_propagation ext4_resize t_readdir_3 splice2pipe \ - uuid_ioctl t_snapshot_deleted_subvolume fiemap-fault min_dio_alignment + uuid_ioctl t_snapshot_deleted_subvolume fiemap-fault min_dio_alignment \ + btrfs_encoded_read btrfs_encoded_write EXTRA_EXECS = dmerror fill2attr fill2fs fill2fs_check scaleread.sh \ btrfs_crc32c_forged_name.py popdir.pl popattr.py \src/btrfs_encoded_read.c new file +2
-0@@ -0,0 +1,175 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) Meta Platforms, Inc. and affiliates. src/btrfs_encoded_read.c line 12
Quoted 10 lines
src/btrfs_encoded_read.c new file +10
-0+ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <errno.h> +#include <fcntl.h> +#include <unistd.h> +#include <sys/uio.h> +#include <sys/ioctl.h> +#include <linux/btrfs.h> For this I need
+#include <
linux/io_uring.h>otherwise I get:
[CC]
btrfs_encoded_read/bin/sh../libtool --quiet --tag=CC --mode=link/usr/bin/gcc-13btrfs_encoded_read.c-obtrfs_encoded_read-g -O2 -g -O2 -DDEBUG(quoted diff) +0
-4-I../include -DVERSION=\"1.1.1\" -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -funsigned-char -fno-strict-aliasing -Wall -DHAVE_FALLOCATE -DNEED_INTERNAL_XFS_IOC_EXCHANGE_RANGE -lhandle -lacl -lpthread -lrt -luuid -lgdbm_compat -lgdbm -laio -luring ../lib/libtest.la
btrfs_encoded_read.c: In function 'encoded_read_io_uring':
btrfs_encoded_read.c:100:26: error: 'IORING_OP_URING_CMD' undeclared (first use in this function); did you mean 'IORING_OP_LINKAT'?
100 | io_uring_prep_rw(IORING_OP_URING_CMD, sqe, fd, &enc,sizeof(enc), 0); | ^~~~~~~~~~~~~~~~~~~ | IORING_OP_LINKATbtrfs_encoded_read.c:100:26: note: each undeclared identifier is reported only once for each function it appears in btrfs_encoded_read.c:101:12: error: 'struct
io_uring_sqe' has no member named 'cmd_op'
Quoted 6 lines
101 | sqe->cmd_op = BTRFS_IOC_ENCODED_READ; | ^~during compilation.
Not sure if a ./configure macro thingy should/would solve this.
We could do that, but elsewhere we're using liburing.h rather than the kernel version.
It looks like IORING_OP_URING_CMD was added to liburing with version 2.2, which came out in June 2022. I don't know whether that's old enough that we can just declare it as our minimum version, whether we should be probing for the liburing version, whether we should be working round this somehow, or what.
Zorro, what do you think?
Mark
Quoted 109 lines · Mark Harmstone
Quoted 98 lines · Johannes Thumshirn
Quoted 1 line
Quoted 60 lines · Mark Harmstone
Add btrfs/333 and its helper programs
btrfs_encoded_readandbtrfs_encoded_write, in order to test encoded reads.We use the
BTRFS_IOC_ENCODED_WRITEioctl to write random data into a compressed extent, then use theBTRFS_IOC_ENCODED_READioctl to check that it matches what we've written. If the newio_uringinterface for encoded reads is supported, we also check that that matches the ioctl.Note that what we write isn't valid compressed data, so any non-encoded reads on these files will fail.
Signed-off-by Mark Harmstone <[email protected]>--- .gitignore | 2 + src/Makefile | 3 +- src/btrfs_encoded_read.c | 175 ++++++++++++++++++++++++++++++ src/btrfs_encoded_write.c | 206 +++++++++++++++++++++++++++++++++++ tests/btrfs/333 | 220 ++++++++++++++++++++++++++++++++++++++ tests/btrfs/333.out | 2 + 6 files changed, 607 insertions(+), 1 deletion(-)create mode 100644
src/btrfs_encoded_read.c
create mode 100644src/btrfs_encoded_write.c
create mode 100755 tests/btrfs/333
create mode 100644 tests/btrfs/333.out.gitignore +2
-0@@ -62,6 +62,8 @@ tags /src/attr_replace_test /src/attr-list-by-handle-cursor-test /src/bstat +/src/btrfs_encoded_read +/src/btrfs_encoded_write /src/bulkstat_null_ocount /src/bulkstat_unlink_test /src/bulkstat_unlink_test_modifiedsrc/Makefile +2
-1@@ -34,7 +34,8 @@ LINUX_TARGETS = xfsctl bstat t_mtab getdevicesize preallo_rw_pattern_reader \ attr_replace_test swapon mkswap t_attr_corruption t_open_tmpfiles \ fscrypt-crypt-util bulkstat_null_ocount splice-test chprojid_fail \ detached_mounts_propagation ext4_resize t_readdir_3 splice2pipe \ - uuid_ioctl t_snapshot_deleted_subvolume fiemap-fault min_dio_alignment + uuid_ioctl t_snapshot_deleted_subvolume fiemap-fault min_dio_alignment \ + btrfs_encoded_read btrfs_encoded_write EXTRA_EXECS = dmerror fill2attr fill2fs fill2fs_check scaleread.sh \ btrfs_crc32c_forged_name.py popdir.pl popattr.py \src/btrfs_encoded_read.c new file +2
-0@@ -0,0 +1,175 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) Meta Platforms, Inc. and affiliates. src/btrfs_encoded_read.c line 12
Quoted 10 lines
src/btrfs_encoded_read.c new file +10
-0+ +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <errno.h> +#include <fcntl.h> +#include <unistd.h> +#include <sys/uio.h> +#include <sys/ioctl.h> +#include <linux/btrfs.h> For this I need
+#include <
linux/io_uring.h>otherwise I get:
[CC]
btrfs_encoded_read/bin/sh../libtool --quiet --tag=CC --mode=link/usr/bin/gcc-13btrfs_encoded_read.c-obtrfs_encoded_read-g -O2 -g -O2 -DDEBUG(quoted diff) +0
-4-I../include -DVERSION=\"1.1.1\" -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -funsigned-char -fno-strict-aliasing -Wall -DHAVE_FALLOCATE -DNEED_INTERNAL_XFS_IOC_EXCHANGE_RANGE -lhandle -lacl -lpthread -lrt -luuid -lgdbm_compat -lgdbm -laio -luring ../lib/libtest.la
btrfs_encoded_read.c: In function 'encoded_read_io_uring':
btrfs_encoded_read.c:100:26: error: 'IORING_OP_URING_CMD' undeclared (first use in this function); did you mean 'IORING_OP_LINKAT'?
100 | io_uring_prep_rw(IORING_OP_URING_CMD, sqe, fd, &enc,sizeof(enc), 0); | ^~~~~~~~~~~~~~~~~~~ | IORING_OP_LINKATbtrfs_encoded_read.c:100:26: note: each undeclared identifier is reported only once for each function it appears in btrfs_encoded_read.c:101:12: error: 'struct
io_uring_sqe' has no member named 'cmd_op'Quoted 6 lines
101 | sqe->cmd_op = BTRFS_IOC_ENCODED_READ; | ^~during compilation.
Not sure if a ./configure macro thingy should/would solve this.
We could do that, but elsewhere we're using
liburing.hrather than the kernel version.
Quoted 7 lines
It looks like
IORING_OP_URING_CMDwas added to liburing with version 2.2, which came out in June 2022. I don't know whether that's old enough that we can just declare it as our minimum version, whether we should be probing for the liburing version, whether we should be working round this somehow, or what.Zorro, what do you think?
2022 was just 2 years ago, some downstream distributions might use old version. I think that might be too early to leave a "2 years ago" system out of the using of latest xfstests :)
Thanks,
Zorro
Quoted 3 lines
Mark
On 14/11/24 05:06, Zorro Lang wrote:
...
Quoted 6 lines
Quoted 6 lines
It looks like
IORING_OP_URING_CMDwas added to liburing with version 2.2, which came out in June 2022. I don't know whether that's old enough that we can just declare it as our minimum version, whether we should be probing for the liburing version, whether we should be working round this somehow, or what.
Quoted 8 lines
Quoted 1 line
Zorro, what do you think?
2022 was just 2 years ago, some downstream distributions might use old version. I think that might be too early to leave a "2 years ago" system out of the using of latest xfstests :)
Thanks,
Zorro
Okay, no worries. I can change it so that it uses the raw syscalls rather than the liburing helpers. It'll be a lot uglier, but at least it'll work.
Johannes, what distro and version are you on? I'll make sure it works on that.
Mark
Quoted 18 lines · Mark Harmstone
On 14/11/24 05:06, Zorro Lang wrote:
...Quoted 6 lines
Quoted 6 lines
It looks like
IORING_OP_URING_CMDwas added to liburing with version 2.2, which came out in June 2022. I don't know whether that's old enough that we can just declare it as our minimum version, whether we should be probing for the liburing version, whether we should be working round this somehow, or what.Quoted 8 lines
Quoted 1 line
Zorro, what do you think?
2022 was just 2 years ago, some downstream distributions might use old version. I think that might be too early to leave a "2 years ago" system out of the using of latest xfstests :)
Thanks,
Zorro
Quoted 6 lines
Okay, no worries. I can change it so that it uses the raw syscalls rather than the liburing helpers. It'll be a lot uglier, but at least it'll work.
Johannes, what distro and version are you on? I'll make sure it works on that.
I'm on openSUSE 15.6 atm.
Thanks,
Johannes
Quoted 19 lines · Johannes Thumshirn
Quoted 18 lines · Mark Harmstone
On 14/11/24 05:06, Zorro Lang wrote:
...Quoted 14 lines
Quoted 7 lines
It looks like
IORING_OP_URING_CMDwas added to liburing with version 2.2, which came out in June 2022. I don't know whether that's old enough that we can just declare it as our minimum version, whether we should be probing for the liburing version, whether we should be working round this somehow, or what.Zorro, what do you think?
2022 was just 2 years ago, some downstream distributions might use old version. I think that might be too early to leave a "2 years ago" system out of the using of latest xfstests :)
Thanks,
Zorro
Quoted 9 lines
Quoted 6 lines
Okay, no worries. I can change it so that it uses the raw syscalls rather than the liburing helpers. It'll be a lot uglier, but at least it'll work.
Johannes, what distro and version are you on? I'll make sure it works on that.
I'm on openSUSE 15.6 atm.
Sorry for the delayed comment. Systems without the latest headers files (missing BTRFS_IOC_ENCODED_READ) will fail to compile, e.g., OL9 with UEK7.
27 | struct btrfs_ioctl_encoded_io_args enc;
btrfs_encoded_read.c:27:44: error: storage size of 'enc' isn't known
btrfs_encoded_read.c:43:25: error: 'BTRFS_IOC_ENCODED_READ' undeclared (first use in this function); did you mean 'BTRFS_IOC_DEVICES_READY'?
43 | ret = ioctl(fd, BTRFS_IOC_ENCODED_READ, &enc);
fstests ./configure may need to manage feature enable?.
Thanks, Anand
Quoted 2 lines
Thanks,
Johannes
And another thing (sorry for spamming you)
When running the test on for-next (HEAD == e82c936293a) with lockdep enabled I get the following:
johannes@nuc:ci$ cat ../fstests/results/btrfs/333.dmesg [ 5.826816] run fstests btrfs/333 at 2024-11-12 09:43:20 [ 6.992664] BTRFS: device fsid c92bd0ac-9334-40e5-8c01-75a45093c706 devid 1 transid 6 /dev/nvme1n1 (259:2) scanned by mount (659) [ 6.994951] BTRFS info (device nvme1n1): first mount of filesystem c92bd0ac-9334-40e5-8c01-75a45093c706 [ 6.996139] BTRFS info (device nvme1n1): using crc32c (crc32c-intel) checksum algorithm [ 6.997099] BTRFS info (device nvme1n1): using free-space-tree [ 7.000920] BTRFS info (device nvme1n1): checking UUID tree
[ 7.465790] ================================================
[ 7.466415] WARNING: lock held when returning to user space!
[ 7.467024] 6.12.0-rc7+ #1044 Not tainted
[ 7.467470] ------------------------------------------------
[ 7.468135] btrfs_encoded_r/703 is leaving the kernel with locks
still held!
[ 7.468955] 1 lock held by btrfs_encoded_r/703:
[ 7.469486] #0: ffff8881163ca4e0
(&sb->s_type->i_mutex_key#14){....}-{3:3}, at: btrfs_inode_lock+0x2a/0x70
[ 11.346356] BTRFS info (device nvme1n1): last unmount of filesystem
c92bd0ac-9334-40e5-8c01-75a45093c706
[ 11.375284] BTRFS info (device nvme0n1): last unmount of filesystem 5c716421-ae86-49ee-b283-13cc4758d395
Byte,
Johannes
Quoted 30 lines · Johannes Thumshirn
Quoted 1 line
And another thing (sorry for spamming you)
When running the test on for-next (HEAD ==
e82c936293a) with lockdep enabled I get the following:johannes@nuc:ci$ cat ../fstests/results/btrfs/333.dmesg [ 5.826816] run fstests btrfs/333 at 2024-11-12 09:43:20 [ 6.992664] BTRFS: device fsid c92bd0ac-9334-40e5-8c01-75a45093c706 devid 1 transid 6 /dev/nvme1n1 (259:2) scanned by mount (659) [ 6.994951] BTRFS info (device nvme1n1): first mount of filesystem c92bd0ac-9334-40e5-8c01-75a45093c706 [ 6.996139] BTRFS info (device nvme1n1): using crc32c (crc32c-intel) checksum algorithm [ 6.997099] BTRFS info (device nvme1n1): using free-space-tree [ 7.000920] BTRFS info (device nvme1n1): checking UUID tree[ 7.465790] ================================================ [ 7.466415] WARNING: lock held when returning to user space! [ 7.467024] 6.12.0-rc7+ #1044 Not tainted [ 7.467470] ------------------------------------------------ [ 7.468135] btrfs_encoded_r/703 is leaving the kernel with locks still held! [ 7.468955] 1 lock held by btrfs_encoded_r/703: [ 7.469486] #0: ffff8881163ca4e0 (&sb->s_type->i_mutex_key#14){....}-{3:3}, at: btrfs_inode_lock+0x2a/0x70 [ 11.346356] BTRFS info (device nvme1n1): last unmount of filesystem c92bd0ac-9334-40e5-8c01-75a45093c706 [ 11.375284] BTRFS info (device nvme0n1): last unmount of filesystem
Quoted 4 lines
5c716421-ae86-49ee-b283-13cc4758d395
Byte,
Johannes
Thanks Johannes, I'll send a patch for this.
Mark