Skip to contentSkip to conversation

erofs: deprecate superblock checksum feature

PATCH In review Jingbo Xu 4 participants 10 messages Jul 17, 2023 – Aug 1, 2023 EROFS file system

lore
Share & tools
lore

Jingbo Xu

author
#0 lore raw

PATCH v2 1/1 erofs: deprecate superblock checksum feature

Later we're going to try the self-contained image verification. The current superblock checksum feature has quite limited functionality, instead, merkle trees can provide better protection for image integrity.

xxhash is also used in the following xattr name filter feature. It is redundant for one filesystem to rely on two hashing algorithms at the same time.

Since the superblock checksum is a compatible feature, just deprecate it now.

---
changes since v1:
- improve commit message (Gao Xiang)

v1: https://lore.kernel.org/all/[email protected]/

---
 fs/erofs/Kconfig |  1 -
 fs/erofs/super.c | 44 +++++---------------------------------------
 2 files changed, 5 insertions(+), 40 deletions(-)
fs/erofs/Kconfig +0 -1
index f259d92c9720..ebcb1f6a426a 100644
@@ -4,7 +4,6 @@ config EROFS_FS
 	tristate "EROFS filesystem support"
 	depends on BLOCK
 	select FS_IOMAP
-	select LIBCRC32C
 	help
 	  EROFS (Enhanced Read-Only File System) is a lightweight read-only
 	  file system with modern designs (e.g. no buffer heads, inline
fs/erofs/super.c +5 -39

73 lines not on this page · raw message

Signature · 1 line
2.19.1.6.gb485710b
Reviewed-by on v2 1/1
#1 lore raw
Quoted 5 lines · Jingbo Xu

Later we're going to try the self-contained image verification. The current superblock checksum feature has quite limited functionality, instead, merkle trees can provide better protection for image integrity.

Quoted 8 lines

xxhash is also used in the following xattr name filter feature. It is redundant for one filesystem to rely on two hashing algorithms at the same time.

Since the superblock checksum is a compatible feature, just deprecate it now.

I tend to agree on this since it slightly impacts mount time too:

Thanks,
Gao Xiang

maintainer Reviewed-by on v2 1/1
#2 lore raw
In reply to Jingbo Xu
Quoted 5 lines · Jingbo Xu

Later we're going to try the self-contained image verification. The current superblock checksum feature has quite limited functionality, instead, merkle trees can provide better protection for image integrity.

Quoted 8 lines

xxhash is also used in the following xattr name filter feature. It is redundant for one filesystem to rely on two hashing algorithms at the same time.

Since the superblock checksum is a compatible feature, just deprecate it now.

Thanks,

#3 lore raw
In reply to Jingbo Xu
Quoted 4 lines · Jingbo Xu

Later we're going to try the self-contained image verification. The current superblock checksum feature has quite limited functionality, instead, merkle trees can provide better protection for image integrity.

The crc32c checksum is also used by libblkid to gain more confidence in its filesystem detection.
I guess a merkle tree would be much harder to implement.

This is for example used by the mount(8) cli program to allow mounting of devices without explicitly needing to specify a filesystem.

Note: libblkid tests for EROFS_FEATURE_SB_CSUM so at least it won't break when the checksum is removed.

Quoted 19 lines

xxhash is also used in the following xattr name filter feature. It is redundant for one filesystem to rely on two hashing algorithms at the same time.

Since the superblock checksum is a compatible feature, just deprecate it now.

---
changes since v1:
- improve commit message (Gao Xiang)

v1: https://lore.kernel.org/all/[email protected]/

---
 fs/erofs/Kconfig |  1 -
 fs/erofs/super.c | 44 +++++---------------------------------------
 2 files changed, 5 insertions(+), 40 deletions(-)

[..]

#4 lore raw

Hi Thomas,

Quoted 10 lines · Thomas Weißschuh
Quoted 4 lines · Jingbo Xu

Later we're going to try the self-contained image verification. The current superblock checksum feature has quite limited functionality, instead, merkle trees can provide better protection for image integrity.

The crc32c checksum is also used by libblkid to gain more confidence in its filesystem detection.
I guess a merkle tree would be much harder to implement.

Quoted 5 lines

This is for example used by the mount(8) cli program to allow mounting of devices without explicitly needing to specify a filesystem.

Note: libblkid tests for EROFS_FEATURE_SB_CSUM so at least it won't break when the checksum is removed.

I'm not sure if we could switch EROFS_FEATURE_SB_CSUM to a simpler checksum instead (e.g. just sum each byte up if both EROFS_FEATURE_SB_CSUM and COMPAT_XATTR_FILTER bits are set, or ignore checksums completely at least in the kernel) if the better filesystem detection by using sb chksum is needed (not sure if other filesystems have sb chksum or just do magic comparsion)?

The main problem here is after xattr name filter feature is added (xxhash is generally faster than crc32c), there could be two hard-depended hashing algorithms, this increases more dependency especially for embededed devices.

Thanks,
Gao Xiang

#5 lore raw

Hi Gao!

Quoted 17 lines · Gao Xiang
Quoted 10 lines · Thomas Weißschuh
Quoted 4 lines · Jingbo Xu

Later we're going to try the self-contained image verification. The current superblock checksum feature has quite limited functionality, instead, merkle trees can provide better protection for image integrity.

The crc32c checksum is also used by libblkid to gain more confidence in its filesystem detection.
I guess a merkle tree would be much harder to implement.

Quoted 5 lines

This is for example used by the mount(8) cli program to allow mounting of devices without explicitly needing to specify a filesystem.

Note: libblkid tests for EROFS_FEATURE_SB_CSUM so at least it won't break when the checksum is removed.

Quoted 6 lines

I'm not sure if we could switch EROFS_FEATURE_SB_CSUM to a simpler checksum instead (e.g. just sum each byte up if both EROFS_FEATURE_SB_CSUM and COMPAT_XATTR_FILTER bits are set, or ignore checksums completely at least in the kernel) if the better filesystem detection by using sb chksum is needed (not sure if other filesystems have sb chksum or just do magic comparsion)?

Overloading EROFS_FEATURE_SB_CSUM in combination with COMPAT_XATTR_FILTER would break all existing deployments of libblkid, so it's not an option.

All other serious and halfway modern filesystems do have superblock checksums which are also checked by libblkid.

Quoted 4 lines

The main problem here is after xattr name filter feature is added (xxhash is generally faster than crc32c), there could be two hard-depended hashing algorithms, this increases more dependency especially for embededed devices.

From libblkid side nothing really speaks against a simpler checksum. XOR is easy to implement and xxhash is already part of libblkid for other filesystems.

The drawbacks are:
* It would need a completely new feature bit in erofs.
* Old versions of libblkid could not validate checksums on newer filesystems.

#6 lore raw
Quoted 27 lines · Thomas Weißschuh

Hi Gao!

Quoted 17 lines · Gao Xiang
Quoted 10 lines · Thomas Weißschuh
Quoted 4 lines · Jingbo Xu

Later we're going to try the self-contained image verification. The current superblock checksum feature has quite limited functionality, instead, merkle trees can provide better protection for image integrity.

The crc32c checksum is also used by libblkid to gain more confidence in its filesystem detection.
I guess a merkle tree would be much harder to implement.

Quoted 5 lines

This is for example used by the mount(8) cli program to allow mounting of devices without explicitly needing to specify a filesystem.

Note: libblkid tests for EROFS_FEATURE_SB_CSUM so at least it won't break when the checksum is removed.

Quoted 6 lines

I'm not sure if we could switch EROFS_FEATURE_SB_CSUM to a simpler checksum instead (e.g. just sum each byte up if both EROFS_FEATURE_SB_CSUM and COMPAT_XATTR_FILTER bits are set, or ignore checksums completely at least in the kernel) if the better filesystem detection by using sb chksum is needed (not sure if other filesystems have sb chksum or just do magic comparsion)?

Quoted 3 lines

Overloading EROFS_FEATURE_SB_CSUM in combination with COMPAT_XATTR_FILTER would break all existing deployments of libblkid, so it's not an option.

I think for libblkid, you could still use:
EROFS_FEATURE_SB_CSUM is not set, don't check anything; EROFS_FEATURE_SB_CSUM only is set, check with crc32c; EROFS_FEATURE_SB_CSUM | COMPAT_XATTR_FILTER (or some other bit) is
set, check with a simpler hash?

Quoted 9 lines

All other serious and halfway modern filesystems do have superblock checksums which are also checked by libblkid.

Quoted 4 lines

The main problem here is after xattr name filter feature is added (xxhash is generally faster than crc32c), there could be two hard-depended hashing algorithms, this increases more dependency especially for embededed devices.

Quoted 8 lines

From libblkid side nothing really speaks against a simpler checksum.
XOR is easy to implement and xxhash is already part of libblkid for other filesystems.

The drawbacks are:
* It would need a completely new feature bit in erofs.
* Old versions of libblkid could not validate checksums on newer filesystems.

just checking magic for Old versions of libblkid will cause false positive in which extent?

Thanks,
Gao Xiang

#7 lore raw
Quoted 30 lines · Gao Xiang
Quoted 25 lines · Thomas Weißschuh
Quoted 17 lines · Gao Xiang
Quoted 10 lines · Thomas Weißschuh
Quoted 4 lines · Jingbo Xu

Later we're going to try the self-contained image verification. The current superblock checksum feature has quite limited functionality, instead, merkle trees can provide better protection for image integrity.

The crc32c checksum is also used by libblkid to gain more confidence in its filesystem detection.
I guess a merkle tree would be much harder to implement.

Quoted 5 lines

This is for example used by the mount(8) cli program to allow mounting of devices without explicitly needing to specify a filesystem.

Note: libblkid tests for EROFS_FEATURE_SB_CSUM so at least it won't break when the checksum is removed.

Quoted 6 lines

I'm not sure if we could switch EROFS_FEATURE_SB_CSUM to a simpler checksum instead (e.g. just sum each byte up if both EROFS_FEATURE_SB_CSUM and COMPAT_XATTR_FILTER bits are set, or ignore checksums completely at least in the kernel) if the better filesystem detection by using sb chksum is needed (not sure if other filesystems have sb chksum or just do magic comparsion)?

Quoted 3 lines

Overloading EROFS_FEATURE_SB_CSUM in combination with COMPAT_XATTR_FILTER would break all existing deployments of libblkid, so it's not an option.

Quoted 5 lines

I think for libblkid, you could still use:
EROFS_FEATURE_SB_CSUM is not set, don't check anything; EROFS_FEATURE_SB_CSUM only is set, check with crc32c; EROFS_FEATURE_SB_CSUM | COMPAT_XATTR_FILTER (or some other bit) is
set, check with a simpler hash?

We could change this in newer versions of libblkid.
But we can't change the older versions that are already deployed today.

And the current code does

if (EROFS_FEATURE_SB_CSUM)
validate_crc32c();

So to stay compatible we need to keep the relationship of EROFS_FEATURE_SB_CSUM -> crc32c.

Quoted 12 lines
Quoted 12 lines

All other serious and halfway modern filesystems do have superblock checksums which are also checked by libblkid.

Quoted 4 lines

The main problem here is after xattr name filter feature is added (xxhash is generally faster than crc32c), there could be two hard-depended hashing algorithms, this increases more dependency especially for embededed devices.

From libblkid side nothing really speaks against a simpler checksum.
XOR is easy to implement and xxhash is already part of libblkid for other filesystems.

Quoted 7 lines
Quoted 4 lines

The drawbacks are:
* It would need a completely new feature bit in erofs.
* Old versions of libblkid could not validate checksums on newer filesystems.

just checking magic for Old versions of libblkid will cause false positive in which extent?

Hard to tell for sure. But it would not surprise me if it would indeed affect users as experience has shown.

Imagine for example erofs filesystems that have then been overwritten with another filesystem but still have a valid erofs magic. With the checksum validation the new filesystem is detected correctly, without it will find the old erofs.

Sometimes the files inside some filesystem look like the superblock of another filesystem and break the detection.

etc.

Having some sort of checksum makes this much easier to handle.

#8 lore raw
Quoted 40 lines · Thomas Weißschuh
Quoted 30 lines · Gao Xiang
Quoted 25 lines · Thomas Weißschuh
Quoted 17 lines · Gao Xiang
Quoted 10 lines · Thomas Weißschuh
Quoted 4 lines · Jingbo Xu

Later we're going to try the self-contained image verification. The current superblock checksum feature has quite limited functionality, instead, merkle trees can provide better protection for image integrity.

The crc32c checksum is also used by libblkid to gain more confidence in its filesystem detection.
I guess a merkle tree would be much harder to implement.

Quoted 5 lines

This is for example used by the mount(8) cli program to allow mounting of devices without explicitly needing to specify a filesystem.

Note: libblkid tests for EROFS_FEATURE_SB_CSUM so at least it won't break when the checksum is removed.

Quoted 6 lines

I'm not sure if we could switch EROFS_FEATURE_SB_CSUM to a simpler checksum instead (e.g. just sum each byte up if both EROFS_FEATURE_SB_CSUM and COMPAT_XATTR_FILTER bits are set, or ignore checksums completely at least in the kernel) if the better filesystem detection by using sb chksum is needed (not sure if other filesystems have sb chksum or just do magic comparsion)?

Quoted 3 lines

Overloading EROFS_FEATURE_SB_CSUM in combination with COMPAT_XATTR_FILTER would break all existing deployments of libblkid, so it's not an option.

Quoted 5 lines

I think for libblkid, you could still use:

   EROFS_FEATURE_SB_CSUM is not set, don't check anything;
   EROFS_FEATURE_SB_CSUM only is set, check with crc32c;
   EROFS_FEATURE_SB_CSUM | COMPAT_XATTR_FILTER (or some other bit) is

set, check with a simpler hash?

We could change this in newer versions of libblkid.
But we can't change the older versions that are already deployed today.

Quoted 7 lines

And the current code does

if (EROFS_FEATURE_SB_CSUM)
validate_crc32c();

So to stay compatible we need to keep the relationship of EROFS_FEATURE_SB_CSUM -> crc32c.

Yes, you are right, thanks for reminder. We really need a new bit for this.

Quoted 29 lines
Quoted 13 lines
Quoted 13 lines

All other serious and halfway modern filesystems do have superblock checksums which are also checked by libblkid.

Quoted 4 lines

The main problem here is after xattr name filter feature is added (xxhash is generally faster than crc32c), there could be two hard-depended hashing algorithms, this increases more dependency especially for embededed devices.

From libblkid side nothing really speaks against a simpler checksum.
XOR is easy to implement and xxhash is already part of libblkid for other filesystems.

Quoted 7 lines
Quoted 4 lines

The drawbacks are:
* It would need a completely new feature bit in erofs.
* Old versions of libblkid could not validate checksums on newer filesystems.

just checking magic for Old versions of libblkid will cause false positive in which extent?

Hard to tell for sure. But it would not surprise me if it would indeed affect users as experience has shown.

Imagine for example erofs filesystems that have then been overwritten with another filesystem but still have a valid erofs magic. With the checksum validation the new filesystem is detected correctly, without it will find the old erofs.

Quoted 6 lines

Sometimes the files inside some filesystem look like the superblock of another filesystem and break the detection.

etc.

Having some sort of checksum makes this much easier to handle.

Yes, but just checking magic for old versions of libblkid for new generated images only.

I'm not sure about this all (I just suggest that we might need a simpler algorithm like XOR instead for sb_chksum otherwise it seems too heavy), let me just drop this commit from -next for further discussion.

Thanks,
Gao Xiang

author
#9 lore raw
In reply to Thomas Weißschuh

Hi, Thomas,

Quoted 39 lines · Thomas Weißschuh

Hi Gao!

Quoted 17 lines · Gao Xiang
Quoted 10 lines · Thomas Weißschuh
Quoted 4 lines · Jingbo Xu

Later we're going to try the self-contained image verification. The current superblock checksum feature has quite limited functionality, instead, merkle trees can provide better protection for image integrity.

The crc32c checksum is also used by libblkid to gain more confidence in its filesystem detection.
I guess a merkle tree would be much harder to implement.

Quoted 5 lines

This is for example used by the mount(8) cli program to allow mounting of devices without explicitly needing to specify a filesystem.

Note: libblkid tests for EROFS_FEATURE_SB_CSUM so at least it won't break when the checksum is removed.

Quoted 6 lines

I'm not sure if we could switch EROFS_FEATURE_SB_CSUM to a simpler checksum instead (e.g. just sum each byte up if both EROFS_FEATURE_SB_CSUM and COMPAT_XATTR_FILTER bits are set, or ignore checksums completely at least in the kernel) if the better filesystem detection by using sb chksum is needed (not sure if other filesystems have sb chksum or just do magic comparsion)?

Overloading EROFS_FEATURE_SB_CSUM in combination with COMPAT_XATTR_FILTER would break all existing deployments of libblkid, so it's not an option.

All other serious and halfway modern filesystems do have superblock checksums which are also checked by libblkid.

Quoted 4 lines

The main problem here is after xattr name filter feature is added (xxhash is generally faster than crc32c), there could be two hard-depended hashing algorithms, this increases more dependency especially for embededed devices.

Quoted 8 lines

From libblkid side nothing really speaks against a simpler checksum. XOR is easy to implement and xxhash is already part of libblkid for other filesystems.

The drawbacks are:
* It would need a completely new feature bit in erofs.
* Old versions of libblkid could not validate checksums on newer filesystems.

Thanks for pointing this out. we indeed need further discussion for a
better solution.

As mentioned previously, we don't want two hashing algorithms dependency for erofs. The best idea as far as I can come up with is that, introduce a new feature bit indicating XOR hashing algorithm for the sb checksum, while the original EROFS_FEATURE_SB_CSUM is not set. As for the old version libblkid, only fs magic is available for the fs type detection, not perfect but in a best-effort way.

Signature · 2 lines
Thanks,
Jingbo