New: Incorrect fix by e2fsck for blocks_count corruption

https://bugzilla.kernel.org/show_bug.cgi?id=211971

            Bug ID: 211971
           Summary: Incorrect fix by e2fsck for blocks_count corruption
           Product: File System
           Version: 2.5
    Kernel Version: Linux 5.4.0-65-generic
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: ext4
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 295497
--> https://bugzilla.kernel.org/attachment.cgi?id=295497&action=edit
log files from mke2fs, dumpe2fs and e2fsck

For an ext4 file system image with only one superblock, if the blocks_count field in superblock is corrupted, e2fsck fixed it incorrectly. In the fixed image, the corrupted blocks_count is unchanged and other fields (e.g., free blocks count) are changed accordingly.
This issue also occurs in images with multiple superblocks too. For example, For an ext4 image with primary and backup superblock (backup superblocks are not located in default locations, e.g., it is located on 513rd block), if the blocks_count field in superblock is corrupted, e2fsck fixed it incorrectly. In the fixed image, the corrupted blocks_count is unchanged and other fields (e.g., free blocks count) are changed accordingly.

e2fsprogs_version_used: e2fsprogs 1.45.6 (20-Mar-2020) The commands that I ran to recreate the scenario are:
For image with only one superblock:

dd if=/dev/zero bs=1024 count=8193 of=/home/hdd/image mke2fs -b 1024 image 8193
debugfs -w image

debugfs:  ssv blocks_count 4000
debugfs:  q

e2fsck -yf image
e2fsck -yf image

# e2fsck fixes the blocks_count corruption in correctly
# In the clean image the blocks_count was 8193, in the fixed image the
blocks_count is 4000
#The second run of e2fsck is consistent with the first run, it doesn't fix
anything, but blocks_count is still 4000
# Expected that e2fsck would fix the blocks count corruption instead of
changing other fields (e.g.,free blocks_count)

For image with multiple superblocks:
dd if=/dev/zero bs=1024 count=8193 of=/home/hdd/image1 mke2fs -b 1024 -g 512 image1 8193
debugfs -w image1

debugfs:  ssv blocks_count 4000
debugfs:  q

e2fsck -yf image1
e2fsck -yf image1

# e2fsck fixes the blocks_count corruption in correctly
# In the clean image the blocks_count was 8193, in the fixed image the
blocks_count is 4000
# The second run of e2fsck is consistent with the first run, it doesn't fix
anything, but blocks_count is still 4000
#There were 16 block groups in the clean image, but there are only 7 block
groups in the fixed image
# Expected that e2fsck would fix the blocks count corruption instead of
changing other fields (e.g.,free blocks_count) and removing the block groups.  

I attached the images and also the logs from mke2fs, dumpe2fs and e2fsck.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

Can you replicate this on modern 5.4 from kernel.org? -generic kernels are from Canonical and are sometimes broken compared to upstream. If you can't replicate this on mainline, you'll need to contact Canonical. We can't do anything if the problem only persists on distribution kernels.

https://bugzilla.kernel.org/show_bug.cgi?id=211971

--- Comment #1 from Amy ([email protected]) ---
Can you replicate this on modern 5.4 from kernel.org? -generic kernels are from Canonical and are sometimes broken compared to upstream. If you can't replicate this on mainline, you'll need to contact Canonical. We can't do anything if the problem only persists on distribution kernels.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

This has nothing to do with the kernel. What the user is complaining about is that e2fsck trusts the blocks count field in the superblock as to be a source of truth. If that field is artificially changed to be a smaller value, e2fsck will assume the file system size indicated by that changed size.

That's an intentional design choice of e2fsck. Given that with modern ext4 file systems, we have metadata checksums, if the superblock has been accidentally corrupted, the checksum will fail, and then e2fsck will try using the backup superblock instead.

For older file systems that don't have metadata checksums enabled, we could check to see if certain "fundamental constants" in the primary superblock is different from the secondary superblock, but...

This will update the blocks_count in the primary and all secondary backups. So that's not going to really help the user. Effectively, the complaint is "I pointed the gun at my foot, and pulled the triggered, and now my foot hurts!"

The problem is that e2fsck can't really determine that the blocks count field has been corrupted. We could warn the user if the blocks_count is smaller than the reported size of the device, but.... that's actually something that can happen in real life, and it's not necessarily a file system "corruption", but rather an intentional choice by the system administrator. If we were to give a warning, or worse, assume that blocks count should be adjusted to be the size of the deivce, we'd be getting complaints from users who deliberately chose to set the file system size to be something smaller than the block device.

So this is a case of e2fsck is working as intended.

Cheers,

- Ted

https://bugzilla.kernel.org/show_bug.cgi?id=211971

--- Comment #2 from Theodore Tso ([email protected]) ---

This has nothing to do with the kernel. What the user is complaining about is that e2fsck trusts the blocks count field in the superblock as to be a source of truth. If that field is artificially changed to be a smaller value, e2fsck will assume the file system size indicated by that changed size.

That's an intentional design choice of e2fsck. Given that with modern ext4 file systems, we have metadata checksums, if the superblock has been accidentally corrupted, the checksum will fail, and then e2fsck will try using the backup superblock instead.

For older file systems that don't have metadata checksums enabled, we could check to see if certain "fundamental constants" in the primary superblock is different from the secondary superblock, but...

This will update the blocks_count in the primary and all secondary backups. So that's not going to really help the user. Effectively, the complaint is "I pointed the gun at my foot, and pulled the triggered, and now my foot hurts!"

The problem is that e2fsck can't really determine that the blocks count field has been corrupted. We could warn the user if the blocks_count is smaller than the reported size of the device, but.... that's actually something that can happen in real life, and it's not necessarily a file system "corruption", but rather an intentional choice by the system administrator. If we were to give a warning, or worse, assume that blocks count should be adjusted to be the size of the deivce, we'd be getting complaints from users who deliberately chose to set the file system size to be something smaller than the block device.

So this is a case of e2fsck is working as intended.

Cheers,

- Ted

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=211971

--- Comment #3 from [email protected] ---
Hello Ted,

Thank you very much for the detailed clarification! It mostly makes sense to me. But I still have two questions regarding the debugfs/e2fsck behavior.

(1)

This is different from what I observed. In my experiment, “debugfs: ssv blocks_count 4000” only updated the blocks_count (and the checksum) in the primary superblock. All secondary backups were not updated (neither the blocks_count nor the checksum). Does this imply that there is a potential bug in debugfs (because it didn’t update all backups as you suggested)? I’m attaching two images before and after “debugfs: ssv blocks_count 4000” for reference (“image1_before”, “image1_after”). I have verified backups are not updated by dumping the backup superblocks information with dumpe2fs.

(2)

In my experiment, I observed that e2fsck was able to fix the debugfs-modified primary superblock using secondary superblocks when the secondary superblocks are located in default locations (ex. 8193rd block). However, in an image where secondary superblocks are not in their default locations (ex:513rd block), I found that e2fsck cannot fix the primary superblock using secondary superblocks. So e2fsck’s behavior is inconsistent depending on the location of the secondary superblocks. Could you please comment on this?

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=211971

--- Comment #4 from [email protected] ---
Created attachment 295609
--> https://bugzilla.kernel.org/attachment.cgi?id=295609&action=edit
The before and after image after using debugfs

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.