Skip to contentSkip to conversation

btrfs-progs: Fix the issues btrfs-convert don't recognition ext4 i_{a,c,a}time_extra

PATCH In review Li Zhang 2 participants 5 messages Aug 24, 2021 – Aug 30, 2021 Btrfs file system

lore
Share & tools
lore

Li Zhang

author
#0 lore raw

PATCH 1/1 btrfs-progs: Fix the issues btrfs-convert don't recognition ext4 i_{a,c,a}time_extra

Hi, I ran convert-tests.sh, and it reported that the 019-ext4-copy-timestamps test failed. The log is as follows

...
mount -o loop -t ext4 btrfs-progs/tests/test.img btrfs-progs/tests/mnt ====== RUN CHECK touch btrfs-progs/tests/mnt/file
====== RUN CHECK stat btrfs-progs/tests/mnt/file
File: 'btrfs-progs/tests/mnt/file'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 700h/1792d Inode: 13 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: unconfined_u:object_r:unlabeled_t:s0
Access: 2021-08-24 22:10:21.999209679 +0800
Modify: 2021-08-24 22:10:21.999209679 +0800
Change: 2021-08-24 22:10:21.999209679 +0800
...
btrfs-progs/btrfs-convert btrfs-progs/tests/test.img
...
====== RUN CHECK mount -t btrfs -o loop btrfs-progs/tests/test.img btrfs-progs/tests/mnt ====== RUN CHECK stat btrfs-progs/tests/mnt/file
File: 'btrfs-progs/tests/mnt/file'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 2ch/44d Inode: 267 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: unconfined_u:object_r:unlabeled_t:s0
Access: 2021-08-24 22:10:21.000000000 +0800
Modify: 2021-08-24 22:10:21.000000000 +0800
Change: 2021-08-24 22:10:21.000000000 +0800
...
atime on converted inode does not match
test failed for case 019-ext4-copy-timestamps

Obviously, the log says that btrfs-convert does not support nanoseconds. I looked at the source code and found that only if ext2_fs.h defines EXT4_EPOCH_MASK btrfs-convert to support nanoseconds. But in e2fsprogs, EXT4_EPOCH_MASK was introduced in v1.43, but in some older versions, such as v1.40, e2fsprogs actually supports nanoseconds. It seems that if struct ext2_inode_large contains the i_atime_extra member, ext4 is supports nanoseconds, so I updated the logic to determine whether the current ext4 file system supports nanosecond precision. In addition, I imported some definitions to encode and decode tv_nsec (copied from e2fsprogs source code).

---
 configure.ac | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
configure.ac +15 -1
index c4fa461..20297c5 100644
@@ -253,7 +253,21 @@ AX_CHECK_DEFINE([linux/fiemap.h], [FIEMAP_EXTENT_SHARED], [],
 AX_CHECK_DEFINE([ext2fs/ext2_fs.h], [EXT4_EPOCH_MASK],
 		[AC_DEFINE([HAVE_EXT4_EPOCH_MASK_DEFINE], [1],
 			   [Define to 1 if e2fsprogs defines EXT4_EPOCH_MASK])],
-		[AC_MSG_WARN([no definition of EXT4_EPOCH_MASK found, probably old e2fsprogs, no 64bit time precision of converted images])])
+        [have_ext4_epoch_mask_define=no])
+
+AS_IF([test "x$have_ext4_epoch_mask_define" = xno], [
+    AC_CHECK_MEMBERS([struct ext2_inode_large.i_atime_extra],
+        [
+            AC_DEFINE([HAVE_EXT4_EPOCH_MASK_DEFINE], [1], [Define to 1 if ext2_inode_large includes i_atime_extra]),
+            AC_DEFINE([EXT4_EPOCH_BITS], [2],[for encode and decode tv_nsec in ext2 inode]),
+            AC_DEFINE([EXT4_EPOCH_MASK], [((1 << EXT4_EPOCH_BITS) - 1)], [For encode and decode tv_nsec info in ext2 inode]),
+            AC_DEFINE([EXT4_NSEC_MASK],  [(~0UL << EXT4_EPOCH_BITS)], [For encode and decode tv_nsec info in ext2 inode]),
+            AC_DEFINE([inode_includes(size, field)],[m4_normalize[(size >= (sizeof(((struct ext2_inode_large *)0)->field) + offsetof(struct ext2_inode_large, field)))]],
+                [For encode and decode tv_nsec info in ext2 inode])
+        ],
+        [AC_MSG_WARN([It seems that ext2_inode_large don't includes tv_nsec related info, probably old e2fsprogs, no 64bit time precision of converted images])],
+        [[#include <ext2fs/ext2_fs.h>]])
+])
 
 AC_CHECK_HEADER(linux/blkzoned.h, [blkzoned_found=yes], [blkzoned_found=no])
 AC_CHECK_MEMBER([struct blk_zone.capacity], [blkzoned_capacity=yes], [blkzoned_capacity=no], [[#include <linux/blkzoned.h>]])
Signature · 5 lines
1.8.3.1


Best regards,
Li
#1 lore raw
Quoted 33 lines · Li Zhang

Hi, I ran convert-tests.sh, and it reported that the 019-ext4-copy-timestamps test failed. The log is as follows

...
mount -o loop -t ext4 btrfs-progs/tests/test.img btrfs-progs/tests/mnt ====== RUN CHECK touch btrfs-progs/tests/mnt/file
====== RUN CHECK stat btrfs-progs/tests/mnt/file
File: 'btrfs-progs/tests/mnt/file'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 700h/1792d Inode: 13 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: unconfined_u:object_r:unlabeled_t:s0
Access: 2021-08-24 22:10:21.999209679 +0800
Modify: 2021-08-24 22:10:21.999209679 +0800
Change: 2021-08-24 22:10:21.999209679 +0800
...
btrfs-progs/btrfs-convert btrfs-progs/tests/test.img
...
====== RUN CHECK mount -t btrfs -o loop btrfs-progs/tests/test.img btrfs-progs/tests/mnt ====== RUN CHECK stat btrfs-progs/tests/mnt/file
File: 'btrfs-progs/tests/mnt/file'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 2ch/44d Inode: 267 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: unconfined_u:object_r:unlabeled_t:s0
Access: 2021-08-24 22:10:21.000000000 +0800
Modify: 2021-08-24 22:10:21.000000000 +0800
Change: 2021-08-24 22:10:21.000000000 +0800
...
atime on converted inode does not match
test failed for case 019-ext4-copy-timestamps

Quoted 12 lines

Obviously, the log says that btrfs-convert does not support nanoseconds. I looked at the source code and found that only if ext2_fs.h defines EXT4_EPOCH_MASK btrfs-convert to support nanoseconds. But in e2fsprogs, EXT4_EPOCH_MASK was introduced in v1.43, but in some older versions, such as v1.40, e2fsprogs actually supports nanoseconds. It seems that if struct ext2_inode_large contains the i_atime_extra member, ext4 is supports nanoseconds, so I updated the logic to determine whether the current ext4 file system supports nanosecond precision. In addition, I imported some definitions to encode and decode tv_nsec (copied from e2fsprogs source code).

So it's supportable even up to the old versions (1.40 was released in 2007) with the updated detection, nice.

Quoted 13 lines
---
 configure.ac | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
configure.ac +0 -1
index c4fa461..20297c5 100644
@@ -253,7 +253,21 @@ AX_CHECK_DEFINE([linux/fiemap.h], [FIEMAP_EXTENT_SHARED], [],
 AX_CHECK_DEFINE([ext2fs/ext2_fs.h], [EXT4_EPOCH_MASK],
 		[AC_DEFINE([HAVE_EXT4_EPOCH_MASK_DEFINE], [1],
 			   [Define to 1 if e2fsprogs defines EXT4_EPOCH_MASK])],
-		[AC_MSG_WARN([no definition of EXT4_EPOCH_MASK found, probably old e2fsprogs, no 64bit time precision of converted images])])

configure.ac AX_CHECK_DEFINE([linux/fiemap.h], [FIEMAP_EXTENT_SHARED], [], · line 265

Quoted 10 lines
configure.ac +10 -0
+        [have_ext4_epoch_mask_define=no])
+
+AS_IF([test "x$have_ext4_epoch_mask_define" = xno], [
+    AC_CHECK_MEMBERS([struct ext2_inode_large.i_atime_extra],
+        [
+            AC_DEFINE([HAVE_EXT4_EPOCH_MASK_DEFINE], [1], [Define to 1 if ext2_inode_large includes i_atime_extra]),
+            AC_DEFINE([EXT4_EPOCH_BITS], [2],[for encode and decode tv_nsec in ext2 inode]),
+            AC_DEFINE([EXT4_EPOCH_MASK], [((1 << EXT4_EPOCH_BITS) - 1)], [For encode and decode tv_nsec info in ext2 inode]),
+            AC_DEFINE([EXT4_NSEC_MASK],  [(~0UL << EXT4_EPOCH_BITS)], [For encode and decode tv_nsec info in ext2 inode]),
+            AC_DEFINE([inode_includes(size, field)],[m4_normalize[(size >= (sizeof(((struct ext2_inode_large *)0)->field) + offsetof(struct ext2_inode_large, field)))]],

The "," can't be at the end of the AC_DEFINE lines, this does not produce valid ./configure and fails with

checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... yes checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... yes checking for struct ext2_inode_large.i_atime_extra... yes ./configure: line 6487: ,: command not found ./configure: line 6490: ,: command not found ./configure: line 6493: ,: command not found ./configure: line 6496: ,: command not found

because the "," appear in the final file as separate commands. Removing them produces valid script and the detection works.

Added to devel, thanks.

author
#2 lore raw
Quoted 85 lines · David Sterba
Quoted 33 lines · Li Zhang

Hi, I ran convert-tests.sh, and it reported that the 019-ext4-copy-timestamps test failed. The log is as follows

...
mount -o loop -t ext4 btrfs-progs/tests/test.img btrfs-progs/tests/mnt ====== RUN CHECK touch btrfs-progs/tests/mnt/file
====== RUN CHECK stat btrfs-progs/tests/mnt/file
File: 'btrfs-progs/tests/mnt/file'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 700h/1792d Inode: 13 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: unconfined_u:object_r:unlabeled_t:s0
Access: 2021-08-24 22:10:21.999209679 +0800
Modify: 2021-08-24 22:10:21.999209679 +0800
Change: 2021-08-24 22:10:21.999209679 +0800
...
btrfs-progs/btrfs-convert btrfs-progs/tests/test.img
...
====== RUN CHECK mount -t btrfs -o loop btrfs-progs/tests/test.img btrfs-progs/tests/mnt ====== RUN CHECK stat btrfs-progs/tests/mnt/file
File: 'btrfs-progs/tests/mnt/file'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 2ch/44d Inode: 267 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: unconfined_u:object_r:unlabeled_t:s0
Access: 2021-08-24 22:10:21.000000000 +0800
Modify: 2021-08-24 22:10:21.000000000 +0800
Change: 2021-08-24 22:10:21.000000000 +0800
...
atime on converted inode does not match
test failed for case 019-ext4-copy-timestamps

Quoted 12 lines

Obviously, the log says that btrfs-convert does not support nanoseconds. I looked at the source code and found that only if ext2_fs.h defines EXT4_EPOCH_MASK btrfs-convert to support nanoseconds. But in e2fsprogs, EXT4_EPOCH_MASK was introduced in v1.43, but in some older versions, such as v1.40, e2fsprogs actually supports nanoseconds. It seems that if struct ext2_inode_large contains the i_atime_extra member, ext4 is supports nanoseconds, so I updated the logic to determine whether the current ext4 file system supports nanosecond precision. In addition, I imported some definitions to encode and decode tv_nsec (copied from e2fsprogs source code).

So it's supportable even up to the old versions (1.40 was released in 2007) with the updated detection, nice.

Quoted 13 lines
---
 configure.ac | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
configure.ac +0 -1
index c4fa461..20297c5 100644
@@ -253,7 +253,21 @@ AX_CHECK_DEFINE([linux/fiemap.h], [FIEMAP_EXTENT_SHARED], [],
 AX_CHECK_DEFINE([ext2fs/ext2_fs.h], [EXT4_EPOCH_MASK],
 		[AC_DEFINE([HAVE_EXT4_EPOCH_MASK_DEFINE], [1],
 			   [Define to 1 if e2fsprogs defines EXT4_EPOCH_MASK])],
-		[AC_MSG_WARN([no definition of EXT4_EPOCH_MASK found, probably old e2fsprogs, no 64bit time precision of converted images])])

configure.ac AX_CHECK_DEFINE([linux/fiemap.h], [FIEMAP_EXTENT_SHARED], [], · line 265

Quoted 10 lines
configure.ac +10 -0
+        [have_ext4_epoch_mask_define=no])
+
+AS_IF([test "x$have_ext4_epoch_mask_define" = xno], [
+    AC_CHECK_MEMBERS([struct ext2_inode_large.i_atime_extra],
+        [
+            AC_DEFINE([HAVE_EXT4_EPOCH_MASK_DEFINE], [1], [Define to 1 if ext2_inode_large includes i_atime_extra]),
+            AC_DEFINE([EXT4_EPOCH_BITS], [2],[for encode and decode tv_nsec in ext2 inode]),
+            AC_DEFINE([EXT4_EPOCH_MASK], [((1 << EXT4_EPOCH_BITS) - 1)], [For encode and decode tv_nsec info in ext2 inode]),
+            AC_DEFINE([EXT4_NSEC_MASK],  [(~0UL << EXT4_EPOCH_BITS)], [For encode and decode tv_nsec info in ext2 inode]),
+            AC_DEFINE([inode_includes(size, field)],[m4_normalize[(size >= (sizeof(((struct ext2_inode_large *)0)->field) + offsetof(struct ext2_inode_large, field)))]],

The "," can't be at the end of the AC_DEFINE lines, this does not produce valid ./configure and fails with

checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... yes checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... yes checking for struct ext2_inode_large.i_atime_extra... yes ./configure: line 6487: ,: command not found ./configure: line 6490: ,: command not found ./configure: line 6493: ,: command not found ./configure: line 6496: ,: command not found

Quoted 4 lines

because the "," appear in the final file as separate commands. Removing them produces valid script and the detection works.

Added to devel, thanks.

Cool, thanks!

Is there any possibility that the version of the GNU build system caused the ./configure error
to be generated? On my machine, I produced a valid ./configure, and my compilation
environment is as follows:

aclocal:    aclocal (GNU automake) 1.13.4
autoconf:   autoconf (GNU Autoconf) 2.69
autoheader: autoheader (GNU Autoconf) 2.69
automake:   automake (GNU automake) 1.13.4
OS: centos 7.6
#3 lore raw
Quoted 10 lines · li zhang

Is there any possibility that the version of the GNU build system caused the ./configure error
to be generated? On my machine, I produced a valid ./configure, and my compilation
environment is as follows:

aclocal:    aclocal (GNU automake) 1.13.4
autoconf:   autoconf (GNU Autoconf) 2.69
autoheader: autoheader (GNU Autoconf) 2.69
automake:   automake (GNU automake) 1.13.4
OS: centos 7.6

You'd have to run the ./configure script or inspect it manually though it could be hard to locate the lines in the generated script.

I get the following diff, the commas are interprted as shell commands.

configure-good 2021-08-30 11:34:19.130281370 +0200 renamed from configure-bad 2021-08-30 11:35:22.034446343 +0200 +6 -5
@@ -6469,6 +6469,8 @@ else
   have_ext4_epoch_mask_define=no
 fi
 
+have_ext4_epoch_mask_define=no
+
 if test "x$have_ext4_epoch_mask_define" = xno; then :
 
     ac_fn_c_check_member "$LINENO" "struct ext2_inode_large" "i_atime_extra" "ac_cv_member_struct_ext2_inode_large_i_atime_extra" "#include <ext2fs/ext2_fs.h>
@@ -6482,16 +6484,16 @@ _ACEOF
 
 
 $as_echo "#define HAVE_EXT4_EPOCH_MASK_DEFINE 1" >>confdefs.h
-,
+
 
 $as_echo "#define EXT4_EPOCH_BITS 2" >>confdefs.h
-,
+
 
 $as_echo "#define EXT4_EPOCH_MASK ((1 << EXT4_EPOCH_BITS) - 1)" >>confdefs.h
-,
+
 
 $as_echo "#define EXT4_NSEC_MASK (~0UL << EXT4_EPOCH_BITS)" >>confdefs.h
-,
+
 
 $as_echo "#define inode_includes(size, field) (size >= (sizeof(((struct ext2_inode_large *)0)->field) + offsetof(struct ext2_inode_large, field)))" >>confdefs.h
--- 
author
#4 lore raw
In reply to David Sterba

Cool, thanks!

Is there any possibility that the version of the GNU build system caused the ./configure error
to be generated? On my machine, I produced a valid ./configure, and my compilation
environment is as follows:

aclocal:    aclocal (GNU automake) 1.13.4
autoconf:   autoconf (GNU Autoconf) 2.69
autoheader: autoheader (GNU Autoconf) 2.69
automake:   automake (GNU automake) 1.13.4
OS: centos 7.6
Quoted 90 lines · David Sterba <[email protected]> 于2021年8月27日周五 上午2:36写道:
Quoted 33 lines · Li Zhang

Hi, I ran convert-tests.sh, and it reported that the 019-ext4-copy-timestamps test failed. The log is as follows

...
mount -o loop -t ext4 btrfs-progs/tests/test.img btrfs-progs/tests/mnt ====== RUN CHECK touch btrfs-progs/tests/mnt/file
====== RUN CHECK stat btrfs-progs/tests/mnt/file
File: 'btrfs-progs/tests/mnt/file'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 700h/1792d Inode: 13 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: unconfined_u:object_r:unlabeled_t:s0
Access: 2021-08-24 22:10:21.999209679 +0800
Modify: 2021-08-24 22:10:21.999209679 +0800
Change: 2021-08-24 22:10:21.999209679 +0800
...
btrfs-progs/btrfs-convert btrfs-progs/tests/test.img
...
====== RUN CHECK mount -t btrfs -o loop btrfs-progs/tests/test.img btrfs-progs/tests/mnt ====== RUN CHECK stat btrfs-progs/tests/mnt/file
File: 'btrfs-progs/tests/mnt/file'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 2ch/44d Inode: 267 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: unconfined_u:object_r:unlabeled_t:s0
Access: 2021-08-24 22:10:21.000000000 +0800
Modify: 2021-08-24 22:10:21.000000000 +0800
Change: 2021-08-24 22:10:21.000000000 +0800
...
atime on converted inode does not match
test failed for case 019-ext4-copy-timestamps

Quoted 12 lines

Obviously, the log says that btrfs-convert does not support nanoseconds. I looked at the source code and found that only if ext2_fs.h defines EXT4_EPOCH_MASK btrfs-convert to support nanoseconds. But in e2fsprogs, EXT4_EPOCH_MASK was introduced in v1.43, but in some older versions, such as v1.40, e2fsprogs actually supports nanoseconds. It seems that if struct ext2_inode_large contains the i_atime_extra member, ext4 is supports nanoseconds, so I updated the logic to determine whether the current ext4 file system supports nanosecond precision. In addition, I imported some definitions to encode and decode tv_nsec (copied from e2fsprogs source code).

So it's supportable even up to the old versions (1.40 was released in 2007) with the updated detection, nice.

Quoted 13 lines
---
 configure.ac | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
configure.ac +0 -1
index c4fa461..20297c5 100644
@@ -253,7 +253,21 @@ AX_CHECK_DEFINE([linux/fiemap.h], [FIEMAP_EXTENT_SHARED], [],
 AX_CHECK_DEFINE([ext2fs/ext2_fs.h], [EXT4_EPOCH_MASK],
              [AC_DEFINE([HAVE_EXT4_EPOCH_MASK_DEFINE], [1],
                         [Define to 1 if e2fsprogs defines EXT4_EPOCH_MASK])],
-             [AC_MSG_WARN([no definition of EXT4_EPOCH_MASK found, probably old e2fsprogs, no 64bit time precision of converted images])])

configure.ac AX_CHECK_DEFINE([linux/fiemap.h], [FIEMAP_EXTENT_SHARED], [], · line 265

Quoted 10 lines
configure.ac +10 -0
+        [have_ext4_epoch_mask_define=no])
+
+AS_IF([test "x$have_ext4_epoch_mask_define" = xno], [
+    AC_CHECK_MEMBERS([struct ext2_inode_large.i_atime_extra],
+        [
+            AC_DEFINE([HAVE_EXT4_EPOCH_MASK_DEFINE], [1], [Define to 1 if ext2_inode_large includes i_atime_extra]),
+            AC_DEFINE([EXT4_EPOCH_BITS], [2],[for encode and decode tv_nsec in ext2 inode]),
+            AC_DEFINE([EXT4_EPOCH_MASK], [((1 << EXT4_EPOCH_BITS) - 1)], [For encode and decode tv_nsec info in ext2 inode]),
+            AC_DEFINE([EXT4_NSEC_MASK],  [(~0UL << EXT4_EPOCH_BITS)], [For encode and decode tv_nsec info in ext2 inode]),
+            AC_DEFINE([inode_includes(size, field)],[m4_normalize[(size >= (sizeof(((struct ext2_inode_large *)0)->field) + offsetof(struct ext2_inode_large, field)))]],

The "," can't be at the end of the AC_DEFINE lines, this does not produce valid ./configure and fails with

checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... yes checking for EXT4_EPOCH_MASK defined in ext2fs/ext2_fs.h... yes checking for struct ext2_inode_large.i_atime_extra... yes ./configure: line 6487: ,: command not found ./configure: line 6490: ,: command not found ./configure: line 6493: ,: command not found ./configure: line 6496: ,: command not found

because the "," appear in the final file as separate commands. Removing them produces valid script and the detection works.

Added to devel, thanks.