Skip to contentSkip to conversation

erofs: Question on unused fields in on-disk structs

DISCUSSION Richard Weinberger 5 participants 11 messages Aug 19, 2019 – Aug 22, 2019 VFS

lore
Share & tools
lore
#0 lore raw

Hi!

struct erofs_super_block has "checksum" and "features" fields, but they are not used in the source.
What is the plan for these?

Same for i_checksum in erofs_inode_v1 and erofs_inode_v2.

At least the "features" field in the super block is something I'd expect to be used.
...such that you can have new filesystem features in future.

Thanks,
//richard

#1 lore raw

Hi Richard,

Quoted 5 lines · Richard Weinberger

Hi!

struct erofs_super_block has "checksum" and "features" fields, but they are not used in the source.
What is the plan for these?

Yes, both will be used laterly (features is used for compatible features, we already have some incompatible features in 5.3).

Quoted 2 lines

Same for i_checksum in erofs_inode_v1 and erofs_inode_v2.

checksum field apart from super_block has been reserved again for linux-next. checksum in the super_block still exists and will be used sooner.

The reason I discussed with Chao is
since EROFS is a read-only filesystem, we will develop block-based checksum and integrity check to EROFS in the future version. It's more effectively than adding such fields to all metadata (some metadata is too large, we cannot calculate a checksum for the whole metadata and compare at runtime, but we can do block-based metadata chksum since EROFS is a read-only fs).

Quoted 4 lines

At least the "features" field in the super block is something I'd expect to be used.
...such that you can have new filesystem features in future.

Of course, "features" is for compatible features, "requirements" is for incompatible features. Both will be used in the future.

Thanks,
Gao Xiang

Quoted 3 lines

Thanks,
//richard

#2 lore raw

Gao Xiang,

Quoted 6 lines · Gao Xiang via Linux-erofs
Quoted 3 lines

struct erofs_super_block has "checksum" and "features" fields, but they are not used in the source.
What is the plan for these?

Yes, both will be used laterly (features is used for compatible features, we already have some incompatible features in 5.3).

Good. :-)
I suggest to check the fields being 0 right now.
Otherwise you are in danger that they get burned if an mkfs.erofs does not initialize the fields.

Signature · 2 lines
Thanks,
//richard
#3 lore raw

Hi Richard,

Quoted 8 lines · Richard Weinberger

Gao Xiang,

Quoted 4 lines · Gao Xiang via Linux-erofs
Quoted 3 lines

struct erofs_super_block has "checksum" and "features" fields, but they are not used in the source.
What is the plan for these?

Quoted 7 lines
Quoted 2 lines

Yes, both will be used laterly (features is used for compatible features, we already have some incompatible features in 5.3).

Good. :-)
I suggest to check the fields being 0 right now.
Otherwise you are in danger that they get burned if an mkfs.erofs does not initialize the fields.

Sorry... I cannot get the point...

super block chksum could be a compatible feature right? which means new kernel can support it (maybe we can add a warning if such image doesn't have a chksum then when mounting) but old kernel doesn't care it.

Or maybe you mean these reserved fields? I have no idea all other filesystems check these fields to 0 or not... But I think it should be used with some other flag is set rather than directly use, right?

Thanks,
Gao Xiang

Quoted 4 lines
Signature · 3 lines
-- 
Thanks,
//richard
#4 lore raw
Quoted 15 lines · Gao Xiang

Hi Richard,

Quoted 11 lines · Richard Weinberger

Gao Xiang,

Quoted 6 lines · Gao Xiang via Linux-erofs
Quoted 3 lines

struct erofs_super_block has "checksum" and "features" fields, but they are not used in the source.
What is the plan for these?

Yes, both will be used laterly (features is used for compatible features, we already have some incompatible features in 5.3).

Quoted 6 lines
Quoted 4 lines

Good. :-)
I suggest to check the fields being 0 right now.
Otherwise you are in danger that they get burned if an mkfs.erofs does not initialize the fields.

Sorry... I cannot get the point...

Sorry for being unclear, let me explain in more detail.

Quoted 4 lines

super block chksum could be a compatible feature right? which means new kernel can support it (maybe we can add a warning if such image doesn't have a chksum then when mounting) but old kernel doesn't care it.

Yes. But you need some why to indicate that the chksum field is now valid and must be used.

The features field can be used for that, but you don't use it right now. I recommend to check it for being 0, 0 means then "no features". If somebody creates in future a erofs with more features this code can refuse to mount because it does not support these features.

But be very sure that existing erofs filesystems actually have this field set to 0 or something other which is always the same. Otherwise you cannot use the field anymore because it could be anything. A common bug is that the mkfs program keeps such unused fields uninitialized and then it can be a more or less random value without notice.

Quoted 3 lines

Or maybe you mean these reserved fields? I have no idea all other filesystems check these fields to 0 or not... But I think it should be used with some other flag is set rather than directly use, right?

Basically you want a way to know when a field shall be used and when not. Most filesystems have version/feature fields. Often multiple to denote different levels of compatibility.

Signature · 2 lines
Thanks,
//richard
#5 lore raw

Hi Richard,

Quoted 16 lines · Richard Weinberger
Quoted 15 lines · Gao Xiang

Hi Richard,

Quoted 11 lines · Richard Weinberger

Gao Xiang,

Quoted 6 lines · Gao Xiang via Linux-erofs
Quoted 3 lines

struct erofs_super_block has "checksum" and "features" fields, but they are not used in the source.
What is the plan for these?

Yes, both will be used laterly (features is used for compatible features, we already have some incompatible features in 5.3).

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Good. :-)
I suggest to check the fields being 0 right now.
Otherwise you are in danger that they get burned if an mkfs.erofs does not initialize the fields.

Sorry... I cannot get the point...

Sorry for being unclear, let me explain in more detail.

Thank you!

Quoted 8 lines
Quoted 5 lines

super block chksum could be a compatible feature right? which means new kernel can support it (maybe we can add a warning if such image doesn't have a chksum then when mounting) but old kernel doesn't care it.

Yes. But you need some why to indicate that the chksum field is now valid and must be used.

We can add a compat "feature" as my following saying... (If I missed something, please kindly point out...)

Quoted 5 lines

The features field can be used for that, but you don't use it right now. I recommend to check it for being 0, 0 means then "no features". If somebody creates in future a erofs with more features this code can refuse to mount because it does not support these features.

"requirements" field is for that, it means incompat features as the following code shown:
69 static bool check_layout_compatibility(struct super_block *sb,
70 struct erofs_super_block *layout)
71 {
72 const unsigned int requirements = le32_to_cpu(layout->requirements);
73
74 EROFS_SB(sb)->requirements = requirements;
75
76 /* check if current kernel meets all mandatory requirements */
77 if (requirements & (~EROFS_ALL_REQUIREMENTS)) {
78 errln("unidentified requirements %x, please upgrade kernel version",
79 requirements & ~EROFS_ALL_REQUIREMENTS);
80 return false;
81 }
82 return true;
83 }

if some "requirements" don't be recognized by the current kernel, it will refuse to mount but "features" not.

Quoted 7 lines

But be very sure that existing erofs filesystems actually have this field set to 0 or something other which is always the same. Otherwise you cannot use the field anymore because it could be anything. A common bug is that the mkfs program keeps such unused fields uninitialized and then it can be a more or less random value without notice.

Why? In my thought, the logic is that
- v4.3, "features" that kernel can handle is 0, so chksum is unused (DONTCARE field) and chksum field could be anything, but the kernel doesn't care.

- later version, add an extra compat feature to "features" to indicate SB_CHKSUM

    is now valid, such as EROFS_FEATURE_SB_CHKSUM (rather than requirements, it's
    incompat), so the kernel can check the checksum like that:

    if (feature & EROFS_FEATURE_SB_CHKSUM) {	/* chksum is set */
        if (chk crc32c and no match) {
             return -EFSBADCRC;
	}
        go ahead
    } else {
        /* still don't care chksum field but print the following warning to kmsg */
        warnln("You are mounting a image without super_block chksum, please take care!!!!");

        or maybe we can even refuse mount these images, except for some mount option
        such as "force-mount".
    }

That is also what F2FS did recently, refer the following commit
commit d440c52d3151("f2fs: support superblock checksum")

Quoted 8 lines
Quoted 4 lines

Or maybe you mean these reserved fields? I have no idea all other filesystems check these fields to 0 or not... But I think it should be used with some other flag is set rather than directly use, right?

Basically you want a way to know when a field shall be used and when not. Most filesystems have version/feature fields. Often multiple to denote different levels of compatibility.

On-disk inode has i_advise field, and super_block has "features" and "requirements" fields. we can use some of them or any combinations.

Thanks,
Gao Xiang

Quoted 4 lines
Signature · 3 lines
-- 
Thanks,
//richard
#6 lore raw

On Thu, Aug 22, 2019 at 05:05:41PM +0800, Gao Xiang wrote:

[]

Quoted 11 lines
Quoted 7 lines

But be very sure that existing erofs filesystems actually have this field set to 0 or something other which is always the same. Otherwise you cannot use the field anymore because it could be anything. A common bug is that the mkfs program keeps such unused fields uninitialized and then it can be a more or less random value without notice.

Why? In my thought, the logic is that
- v4.3, "features" that kernel can handle is 0, so chksum is unused (DONTCARE field) and chksum field could be anything, but the kernel doesn't care.

- sorry, I meant linux <= v5.3. add a word....

Thanks,
Gao Xiang

Quoted 41 lines

- later version, add an extra compat feature to "features" to indicate SB_CHKSUM

    is now valid, such as EROFS_FEATURE_SB_CHKSUM (rather than requirements, it's
    incompat), so the kernel can check the checksum like that:

    if (feature & EROFS_FEATURE_SB_CHKSUM) {	/* chksum is set */
        if (chk crc32c and no match) {
             return -EFSBADCRC;
	}
        go ahead
    } else {
        /* still don't care chksum field but print the following warning to kmsg */
        warnln("You are mounting a image without super_block chksum, please take care!!!!");

        or maybe we can even refuse mount these images, except for some mount option
        such as "force-mount".
    }

That is also what F2FS did recently, refer the following commit
commit d440c52d3151("f2fs: support superblock checksum")

Quoted 8 lines
Quoted 4 lines

Or maybe you mean these reserved fields? I have no idea all other filesystems check these fields to 0 or not... But I think it should be used with some other flag is set rather than directly use, right?

Basically you want a way to know when a field shall be used and when not. Most filesystems have version/feature fields. Often multiple to denote different levels of compatibility.

On-disk inode has i_advise field, and super_block has "features" and "requirements" fields. we can use some of them or any combinations.

Thanks,
Gao Xiang

Quoted 4 lines
Signature · 3 lines
-- 
Thanks,
//richard
#7 lore raw
In reply to Richard Weinberger
Quoted 5 lines · Richard Weinberger
Quoted 4 lines

super block chksum could be a compatible feature right? which means new kernel can support it (maybe we can add a warning if such image doesn't have a chksum then when mounting) but old kernel doesn't care it.

Quoted 7 lines

Yes. But you need some why to indicate that the chksum field is now valid and must be used.

The features field can be used for that, but you don't use it right now. I recommend to check it for being 0, 0 means then "no features". If somebody creates in future a erofs with more features this code can refuse to mount because it does not support these features.

The whole point of "compat" features is that the kernel can go ahead and mount the file system even if there is some new "compat" feature which it doesn't understand. So the fact that right now erofs doesn't have any "compat" features means it's not surprising, and perfectly OK, if it's not referenced by the kernel.

For ext4, we have some more complex feature bitmasks, "compat", "ro_compat" (OK to mount read-only if there are features you don't understand) and "incompat" (if there are any bits you don't understand, fail the mount). But since erofs is a read-only file system, things are much simpler.

It might make life easier for other kernel developers if "features" was named "compat_features" and "requirements" were named "incompat_features", just because of the long-standing use of that in ext2, ext3, ext4, ocfs2, etc. But that naming scheme really is a legacy of ext2 and its descendents, and there's no real reason why it has to be that way on other file systems.

Cheers,

- Ted

#8 lore raw
Quoted 6 lines · Theodore Y. Ts'o

It might make life easier for other kernel developers if "features" was named "compat_features" and "requirements" were named "incompat_features", just because of the long-standing use of that in ext2, ext3, ext4, ocfs2, etc. But that naming scheme really is a legacy of ext2 and its descendents, and there's no real reason why it has to be that way on other file systems.

Yes, the naming confused me a little. :-)

Signature · 2 lines
Thanks,
//richard
#9 lore raw

Hi Richard,

Quoted 9 lines · Richard Weinberger
Quoted 6 lines · Theodore Y. Ts'o

It might make life easier for other kernel developers if "features" was named "compat_features" and "requirements" were named "incompat_features", just because of the long-standing use of that in ext2, ext3, ext4, ocfs2, etc. But that naming scheme really is a legacy of ext2 and its descendents, and there's no real reason why it has to be that way on other file systems.

Yes, the naming confused me a little. :-)

Sorry for confusing... And thanks, I'm happy that you give us those reports. and sorry about my poor English...

Thanks,
Gao Xiang

Quoted 4 lines
Signature · 3 lines
-- 
Thanks,
//richard
#10 lore raw
In reply to Theodore Y. Ts'o

Hi Ted,

Quoted 26 lines · Theodore Y. Ts'o
Quoted 5 lines · Richard Weinberger
Quoted 4 lines

super block chksum could be a compatible feature right? which means new kernel can support it (maybe we can add a warning if such image doesn't have a chksum then when mounting) but old kernel doesn't care it.

Quoted 7 lines

Yes. But you need some why to indicate that the chksum field is now valid and must be used.

The features field can be used for that, but you don't use it right now. I recommend to check it for being 0, 0 means then "no features". If somebody creates in future a erofs with more features this code can refuse to mount because it does not support these features.

The whole point of "compat" features is that the kernel can go ahead and mount the file system even if there is some new "compat" feature which it doesn't understand. So the fact that right now erofs doesn't have any "compat" features means it's not surprising, and perfectly OK, if it's not referenced by the kernel.

For ext4, we have some more complex feature bitmasks, "compat", "ro_compat" (OK to mount read-only if there are features you don't understand) and "incompat" (if there are any bits you don't understand, fail the mount). But since erofs is a read-only file system, things are much simpler.

Quoted 6 lines

It might make life easier for other kernel developers if "features" was named "compat_features" and "requirements" were named "incompat_features", just because of the long-standing use of that in ext2, ext3, ext4, ocfs2, etc. But that naming scheme really is a legacy of ext2 and its descendents, and there's no real reason why it has to be that way on other file systems.

Thanks for your detailed explanation, thanks a lot!

Thanks,
Gao Xiang

Quoted 4 lines

Cheers,

- Ted