Skip to contentSkip to conversation

libfrog: move crc32c selftest buffer into a separate file

PATCH In review Darrick J. Wong 2 participants 4 messages Mar 16, 2023 – Mar 18, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 libfrog: move crc32c selftest buffer into a separate file

From: Darrick J. Wong <[email protected]>

Move the random buffer used for selftests into a separate file so that we can link to it from multiple places.

---
 libfrog/Makefile         |    6 -
 libfrog/crc32cselftest.h |  526 +---------------------------------------------
 libfrog/randbytes.c      |  527 ++++++++++++++++++++++++++++++++++++++++++++++
 libfrog/randbytes.h      |   11 +
 4 files changed, 548 insertions(+), 522 deletions(-)
 create mode 100644 libfrog/randbytes.c
 create mode 100644 libfrog/randbytes.h
libfrog/Makefile +4 -2
index 01107082..b89818ed 100644
@@ -26,6 +26,7 @@ paths.c \
 projects.c \
 ptvar.c \
 radix-tree.c \
+randbytes.c \
 scrub.c \
 util.c \
 workqueue.c
@@ -45,6 +46,7 @@ paths.h \
 projects.h \
 ptvar.h \
 radix-tree.h \
+randbytes.h \
 scrub.h \
 workqueue.h
 
@@ -69,9 +71,9 @@ crc32table.h: gen_crc32table.c crc32defs.h
 # systems/architectures. Hence we make sure that xfsprogs will never use a
 # busted CRC calculation at build time and hence avoid putting bad CRCs down on
 # disk.
-crc32selftest: gen_crc32table.c crc32table.h crc32.c crc32defs.h
+crc32selftest: gen_crc32table.c crc32table.h crc32.c crc32defs.h randbytes.c
 	@echo "    [TEST]    CRC32"
-	$(Q) $(BUILD_CC) $(BUILD_CFLAGS) -D CRC32_SELFTEST=1 crc32.c -o $@
+	$(Q) $(BUILD_CC) $(BUILD_CFLAGS) -D CRC32_SELFTEST=1 randbytes.c crc32.c -o $@
 	$(Q) ./$@
 
 include $(BUILDRULES)
libfrog/crc32cselftest.h +6 -520

547 lines not on this page · raw message

libfrog/randbytes.c new file +527 -0

527 lines not on this page · raw message

libfrog/randbytes.h new file +11 -0
new file mode 100644
index 00000000..00fd7c4c
@@ -0,0 +1,11 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2023 Oracle.  All Rights Reserved.
+ * Author: Darrick J. Wong <[email protected]>
+ */
+#ifndef __LIBFROG_RANDBYTES_H__
+#define __LIBFROG_RANDBYTES_H__
+
+extern uint8_t randbytes_test_buf[];
+
+#endif /* __LIBFROG_RANDBYTES_H__ */
#1 lore raw

PATCH 2/2 misc: test the dir/attr hash before formatting or repairing fs

From: Darrick J. Wong <[email protected]>

Back in the 6.2-rc1 days, Eric Whitney reported a fstests regression in ext4 against generic/454. The cause of this test failure was the unfortunate combination of setting an xattr name containing UTF8 encoded emoji, an xattr hash function that accepted a char pointer with no explicit signedness, signed type extension of those chars to an int, and the 6.2 build tools maintainers deciding to mandate -funsigned-char across the board. As a result, the ondisk extended attribute structure written out by 6.1 and 6.2 were not the same.

This discrepancy, in fact, had been noticeable if a filesystem with such an xattr were moved between any two architectures that don't employ the same signedness of a raw "char" declaration. The only reason anyone noticed is that x86 gcc defaults to signed, and no such -funsigned-char update was made to e2fsprogs, so e2fsck immediately started reporting data corruption.

After a day and a half of discussing how to handle this use case (xattrs with bit 7 set anywhere in the name) without breaking existing users, Linus merged his own patch and didn't tell the mailing list. None of the developers noticed until AUTOSEL made an announcement.

In the end, this problem could have been detected much earlier if there had been any useful tests of hash function(s) in use inside ext4 to make sure that they always produce the same outputs given the same inputs.

The XFS dirent/xattr name hash takes a uint8_t*, so I don't think it's vulnerable to this problem. However, let's avoid all this drama by adding our own self test to check that the da hash produces the same outputs for a static pile of inputs on various platforms. This corresponds to the similar patch for the kernel.

---
 libfrog/Makefile         |    1 
 libfrog/crc32cselftest.h |   17 ++---
 libfrog/dahashselftest.h |  172 ++++++++++++++++++++++++++++++++++++++++++++++
 mkfs/xfs_mkfs.c          |    8 ++
 repair/init.c            |    5 +
 5 files changed, 195 insertions(+), 8 deletions(-)
 create mode 100644 libfrog/dahashselftest.h
libfrog/Makefile +1 -0
index b89818ed..f292afe3 100644
@@ -40,6 +40,7 @@ crc32c.h \
 crc32cselftest.h \
 crc32defs.h \
 crc32table.h \
+dahashselftest.h \
 fsgeom.h \
 logging.h \
 paths.h \
libfrog/crc32cselftest.h +9 -8
index 757a9cf1..ad9c74c7 100644
@@ -41,7 +41,7 @@ static struct crc_test {
 	uint32_t start;	/* random 6 bit offset in buf */
 	uint32_t length;	/* random 11 bit length of test */
 	uint32_t crc32c_le;	/* expected crc32c_le result */
-} test[] =
+} crc_tests[] =
 {
 	{0x674bf11d, 0x00000038, 0x00000542, 0xf6e93d6c},
 	{0x35c672c6, 0x0000003a, 0x000001aa, 0x0fe92aca},
@@ -164,18 +164,19 @@ crc32c_test(
 
 	/* pre-warm the cache */
 	for (i = 0; i < 100; i++) {
-		bytes += 2*test[i].length;
+		bytes += 2 * crc_tests[i].length;
 
-		crc ^= crc32c_le(test[i].crc, randbytes_test_buf +
-		    test[i].start, test[i].length);
+		crc ^= crc32c_le(crc_tests[i].crc,
+				randbytes_test_buf + crc_tests[i].start,
+				crc_tests[i].length);
 	}
 
 	gettimeofday(&start, NULL);
 	for (i = 0; i < 100; i++) {
-		crc = crc32c_le(test[i].crc,
-				randbytes_test_buf + test[i].start,
-				test[i].length);
-		if (crc != test[i].crc32c_le)
+		crc = crc32c_le(crc_tests[i].crc,
+				randbytes_test_buf + crc_tests[i].start,
+				crc_tests[i].length);
+		if (crc != crc_tests[i].crc32c_le)
 			errors++;
 	}
 	gettimeofday(&stop, NULL);
libfrog/dahashselftest.h new file +172 -0

172 lines not on this page · raw message

mkfs/xfs_mkfs.c +8 -0
index 54730497..4320cd9e 100644
@@ -11,6 +11,7 @@
 #include "libfrog/fsgeom.h"
 #include "libfrog/convert.h"
 #include "libfrog/crc32cselftest.h"
+#include "libfrog/dahashselftest.h"
 #include "proto.h"
 #include <ini.h>
 
@@ -4264,6 +4265,13 @@ main(
 		return 1;
 	}
 
+	/* Make sure our dir/attr hash algorithm really works. */
+	if (dahash_test(DAHASHTEST_QUIET) != 0) {
+		fprintf(stderr,
+ _("xfs dir/attr self-test failed, will not create a filesystem here.\n"));
+		return 1;
+	}
+
 	/*
 	 * All values have been validated, discard the old device layout.
 	 */
repair/init.c +5 -0
index 3a320b4f..0d5bfabc 100644
@@ -15,6 +15,7 @@
 #include "incore.h"
 #include "prefetch.h"
 #include "libfrog/crc32cselftest.h"
+#include "libfrog/dahashselftest.h"
 #include <sys/resource.h>
 
 static void
@@ -105,4 +106,8 @@ _("Unmount or use the dangerous (-d) option to repair a read-only mounted filesy
 	if (crc32c_test(CRC32CTEST_QUIET) != 0)
 		do_error(
  _("crc32c self-test failed, will not examine filesystem.\n"));
+
+	if (dahash_test(DAHASHTEST_QUIET) != 0)
+		do_error(
+ _("xfs dir/attr hash self-test failed, will not examine filesystem.\n"));
 }
Reviewed-by on v1 2/2
#2 lore raw

Re: [PATCH 2/2] misc: test the dir/attr hash before formatting or repairing fs

Quoted 35 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Back in the 6.2-rc1 days, Eric Whitney reported a fstests regression in ext4 against generic/454. The cause of this test failure was the unfortunate combination of setting an xattr name containing UTF8 encoded emoji, an xattr hash function that accepted a char pointer with no explicit signedness, signed type extension of those chars to an int, and the 6.2 build tools maintainers deciding to mandate -funsigned-char across the board. As a result, the ondisk extended attribute structure written out by 6.1 and 6.2 were not the same.

This discrepancy, in fact, had been noticeable if a filesystem with such an xattr were moved between any two architectures that don't employ the same signedness of a raw "char" declaration. The only reason anyone noticed is that x86 gcc defaults to signed, and no such -funsigned-char update was made to e2fsprogs, so e2fsck immediately started reporting data corruption.

After a day and a half of discussing how to handle this use case (xattrs with bit 7 set anywhere in the name) without breaking existing users, Linus merged his own patch and didn't tell the mailing list. None of the developers noticed until AUTOSEL made an announcement.

In the end, this problem could have been detected much earlier if there had been any useful tests of hash function(s) in use inside ext4 to make sure that they always produce the same outputs given the same inputs.

The XFS dirent/xattr name hash takes a uint8_t*, so I don't think it's vulnerable to this problem. However, let's avoid all this drama by adding our own self test to check that the da hash produces the same outputs for a static pile of inputs on various platforms. This corresponds to the similar patch for the kernel.

Quoted 8 lines
---
 libfrog/Makefile         |    1 
 libfrog/crc32cselftest.h |   17 ++---
 libfrog/dahashselftest.h |  172 ++++++++++++++++++++++++++++++++++++++++++++++
 mkfs/xfs_mkfs.c          |    8 ++
 repair/init.c            |    5 +
 5 files changed, 195 insertions(+), 8 deletions(-)
 create mode 100644 libfrog/dahashselftest.h

Looks good.

Signature · 2 lines
Dave Chinner
[email protected]
Reviewed-by on v1 1/2
#3 lore raw
In reply to Darrick J. Wong
Quoted 6 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Move the random buffer used for selftests into a separate file so that we can link to it from multiple places.

Quoted 8 lines
---
 libfrog/Makefile         |    6 -
 libfrog/crc32cselftest.h |  526 +---------------------------------------------
 libfrog/randbytes.c      |  527 ++++++++++++++++++++++++++++++++++++++++++++++
 libfrog/randbytes.h      |   11 +
 4 files changed, 548 insertions(+), 522 deletions(-)
 create mode 100644 libfrog/randbytes.c
 create mode 100644 libfrog/randbytes.h

looks fine.

Signature · 2 lines
Dave Chinner
[email protected]