e2fsck and backup superblocks

DISCUSSION Phillip Susi 4 participants 8 messages Feb 18, 2012 – Feb 20, 2012

Share & tools

Phillip Susi

author

2012-02-18 03:00 UTC #0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have noticed that e2fsck always complains about block group descriptor checksums being invalid when run on a backup superblock, even when run again immediately after being told to fix them. Is there some odd intentional behavior here I'm not aware of?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPPxQ5AAoJEJrBOlT6nu75dgUH/19Sv2UASFGCLTfsvJcBBFjC z6w0LgI/GB37mlan7g06mT3UYSEN5LlD7VYAqWUEOgchu3Y+t735w1MRZy7WmVrc 9PopsJVTUZnc8vCcGAoh/T4OcJkf5zU7EAoL+BQUInTOkermLL/Bp9yGVIVsDXN1 gXnf6BdozDkm4LRCIbN1RStg6IqUClIxbvLwItDYhgdwmMFIVsd0dUPTPhb8s8c3 tMQclqLTx+9lYJXC7ZBKVC5BZhaMmuT4ITrKd66h2goRY4wEWz/v1iwhuLyNfgUo enXjG65PrxiiqirGiLPS5UtEvoqMQ8PsyaSILnYqpgJKd9AdP9NQjvjnG6PxA/Y= =TfhG
-----END PGP SIGNATURE-----

Eric Sandeen

2012-02-18 03:05 UTC #1 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 1 line · Phillip Susi

I have noticed that e2fsck always complains about block group descriptor checksums being invalid when run on a backup superblock, even when run again immediately after being told to fix them. Is there some odd intentional behavior here I'm not aware of?

Check the list for:

"[PATCH] libext2fs: quiet spurious group checksum errors"

- -Eric
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJPPxV/AAoJECCuFpLhPd7gul4P/3OOyVnLqAapkkApwJf1cRq0 SDNH/1d8cbldwLMizgAuE3omoJRbzgpy6jghOtq573e1IOIskbQFDxG8HBSg6GGn E1Yw8UcaCqKERRiemTESmC6Idtj6EAusZJWeTlC9XRx0Cbx8bGJenlU6eR/vpxH0 AvOPmgJhNMcRyMdsHLsvXXZNER5j9St1AEEVa2Wzpo2PGyJjVP8jDnJiuhvz2bK3 nv5u28WSuF5y8zzTs0j6DT40XQlpT/OVNflqUGelhxKHBB+sT2c0VvaVhVy63hw/ yuaIMmnwggJSs6p//9iOtKJhTSmR6qSGuuT65ajMgixy66efRLI0C0e5Y/U4stS4 cpT8m8+FGILhIe+Q+sJ2nOxC1JUvZW3AQvi0/gLZDCpzBid9RrmigfU/FTxs4MQB CD9QDkmhz7A7rUvRulC4NSSXcgErMtwC5zAXG9MKiFAtvMBqYXWhWoKfxslaZXOU 5sOIvoB+TOwlWL2StQgwnJjJMCZ9/Eju2VhgEFbZxEgpInaFpk/FUm2k4hzQpHY9 M1ItllWFrR/XMXxozmN1/KmynOLdBEf4rW/BouC+ZCGZLeUOg8M+obJFNICFQEXq VLSvUfIfsLaLb2ApBUyq75/mvIQIo5mCjRYqi1Ev7Rx8XrFWHfNR62Q/VwAVU6X+ vqDKt1dPO06sBPpzOdiP
=jdg1
-----END PGP SIGNATURE-----

Ted Ts'o

2012-02-18 22:31 UTC #2 lore raw

Quoted 9 lines · Eric Sandeen

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 1 line · Phillip Susi
I have noticed that e2fsck always complains about block group descriptor checksums being invalid when run on a backup superblock, even when run again immediately after being told to fix them. Is there some odd intentional behavior here I'm not aware of?
Check the list for:
"[PATCH] libext2fs: quiet spurious group checksum errors"

Which is in e2fsprogs 1.42.1....

- Ted

Eric Sandeen

2012-02-20 03:11 UTC #3 lore raw

Quoted 4 lines · Ted Ts'o

Quoted 3 lines · Eric Sandeen
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 9 lines

Quoted 6 lines · Phillip Susi
Quoted 1 line
I have noticed that e2fsck always complains about block group descriptor checksums being invalid when run on a backup superblock, even when run again immediately after being told to fix them. Is there some odd intentional behavior here I'm not aware of?
Check the list for:
"[PATCH] libext2fs: quiet spurious group checksum errors"
Which is in e2fsprogs 1.42.1....

Was that the release announcement? :)

- Eric

Quoted 5 lines

- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Ted Ts'o

2012-02-20 03:19 UTC #4 lore raw

E2fsprogs 1.42.1 release

Quoted 3 lines · Eric Sandeen

Quoted 2 lines
Which is in e2fsprogs 1.42.1....
Was that the release announcement? :)

I knew I forgot something. :-)

Over the weekend I released e2fsprogs 1.42.1. It can be found in the usual places: on sourceforge, a signed git tag, and on ftp.kernel.org:/pub/linux/kernel/people/tytso/e2fsprogs/v1.42.1.

- Ted

E2fsprogs 1.42.1 (February 17, 2012)
===================================

The mke2fs and e2fsck now use significantly less memory when creating or checking very large file systems. This was enabled by adding extent-based bitmaps which are stored using a red-block tree, since block and inode allocations tend to be contiguous.

The command mke2fs -S is used as a last ditch recovery command to write new superblock and block group descriptors, but _not_ to destroy the inode table in hopes of recovering from a badly corrupted file system. So if the uninit_bg feature is enabled, mke2fs -S will now set the unused inodes count field to zero. Otherwise, e2fsck -fy after using mke2fs -S would leave the file system completely empty.

Since mke2fs recognizes mke3fs in argv[0] to mean "mkfs.ext3", also honor "mke4fs" to work the same as "mke2fs.ext4", since RHEL5 has installed an mke2fs binary using that name.

The usage and help messages for the -G, -t and -T options in mke2fs have been fixed.

If e2fsck needs to use the backup group descriptors, the ext2fs_open2() function clears the UNINIT bits to ensure all of the inodes in the file systems get scanned. However, the code which reset the UNINIT flags did not also recalculate the checksum, which produced many spurious (and scary) e2fsck messages. This has been fixed by resetting cheksums when the UNINIT bits are cleared.

Relax a check in e2fsck which required that the block bitmap to be initialized when the inode bitmap is in use. This will allow us to eventually eliminate code from the kernel which forcibly initialized the block bitmap when the inode bitmap is first used, requiring an extra journal credit and disk write. (Addresses Google Bug: #5944440)

Make sure rdebugfs (which may be installed setuid or setgid disk) does not honor environment variables if euid != uid or egid != gid.

Debugfs's ncheck command has been optimized and now is much more robust with faced with corrupted file systems. The ncheck command also now has a -c option which will verify the file type information in the directory entry to see if matches the inode's mode bits. This is extremely useful when trying to use debugsfs to determine which parts of the file system metadata can be trusted.

E2image will try to use ftruncate64() to set the i_size for raw images, instead of writing a single null byte. This avoid allocating an extra block to the raw image, for those file systems and/or operating systems that support this. (Linux does.) In addition, fix a logic bug that caused the file to not be properly extended if the size of the last hole was exactly an multiple of a megabyte.

Fixed a bug in resize2fs where for 1k and 2k file systems, where s_first_data_block is non-zero, this wasn't taken into account when calculate the minimum file system size for use with the -M option.

Fixed the badblocks program to honor the -s flag when in read-only -t mode. (Addresses Debian Bug #646629)

Update Czech, Dutch, French, Polish, and Sweedish translation from the Translation Project.

Fixed various Debian Packaging issues so that dpkg-buildflags is used if present, which allows e2fsprogs to be built with security hardening flags. (Addresses Debian Bugs: #654457)

Programmer's Notes
------------------

Fix a bug in ext2fs_clear_generic_bmap() when used for 32-bit bitmaps. This was only an issue for programs compiled against e2fsprogs 1.41 that manipulate bitmaps directly. (Addresses Sourceforge Bugs:
#3451486)

The libext2fs library now uses sysconf() to fetch the page size, instead of the deprecated getpagesize().

The ext2fs_get_pathname() function will return a partial path if an a directory in the path is not a directory, displaying it as an inode number in angle brackets instead of giving up and displaying an error. This is much more helpful when a user is trying to debug a corrupted file system.

Codepoints for the RO_COMPAT_REPLICA feature has been reserved.

Added a new library function, ext2fs_file_get_inode_num(), for use by fuse2fs.

Fixed a bug in ext2fs_file_set_size2() so that when it is truncating a file, it actually works.

The block iterator now properly honors the BLOCK_ABORT flag for extent-based flags. Previously, it didn't, which generally made code be less efficient, but it could cause bugs in ext2fs_link(), for example, by causing it to insert multiple directory entries.

Fixed an (harmless other than causing a compiler warning) use of an uninitialized variable in e2fsck's MMP code.

Eric Sandeen

2012-02-20 18:59 UTC #5 lore raw

Re: E2fsprogs 1.42.1 release

Quoted 10 lines · Ted Ts'o

Quoted 3 lines · Eric Sandeen
Quoted 2 lines
Which is in e2fsprogs 1.42.1....
Was that the release announcement? :)
I knew I forgot something. :-)
Over the weekend I released e2fsprogs 1.42.1. It can be found in the usual places: on sourceforge, a signed git tag, and on ftp.kernel.org:/pub/linux/kernel/people/tytso/e2fsprogs/v1.42.1.

The tarball is a little messed up:

[esandeen@neon e2fsprogs]$ tar xvzf e2fsprogs-1.42.1.tar.gz e2fsprogs-libs-1.42.1/ e2fsprogs-libs-1.42.1/.gitignore e2fsprogs-libs-1.42.1/.release-checklist e2fsprogs-libs-1.42.1/lib/

why does it unpack into e2fsprogs-libs-1.42.1?

                                  ^^^^

I could work around that but it's unexpected.

-Eric

Quoted 104 lines

- Ted
E2fsprogs 1.42.1 (February 17, 2012)
===================================
The mke2fs and e2fsck now use significantly less memory when creating or checking very large file systems. This was enabled by adding extent-based bitmaps which are stored using a red-block tree, since block and inode allocations tend to be contiguous.
The command mke2fs -S is used as a last ditch recovery command to write new superblock and block group descriptors, but _not_ to destroy the inode table in hopes of recovering from a badly corrupted file system. So if the uninit_bg feature is enabled, mke2fs -S will now set the unused inodes count field to zero. Otherwise, e2fsck -fy after using mke2fs -S would leave the file system completely empty.
Since mke2fs recognizes mke3fs in argv[0] to mean "mkfs.ext3", also honor "mke4fs" to work the same as "mke2fs.ext4", since RHEL5 has installed an mke2fs binary using that name.
The usage and help messages for the -G, -t and -T options in mke2fs have been fixed.
If e2fsck needs to use the backup group descriptors, the ext2fs_open2() function clears the UNINIT bits to ensure all of the inodes in the file systems get scanned. However, the code which reset the UNINIT flags did not also recalculate the checksum, which produced many spurious (and scary) e2fsck messages. This has been fixed by resetting cheksums when the UNINIT bits are cleared.
Relax a check in e2fsck which required that the block bitmap to be initialized when the inode bitmap is in use. This will allow us to eventually eliminate code from the kernel which forcibly initialized the block bitmap when the inode bitmap is first used, requiring an extra journal credit and disk write. (Addresses Google Bug: #5944440)
Make sure rdebugfs (which may be installed setuid or setgid disk) does not honor environment variables if euid != uid or egid != gid.
Debugfs's ncheck command has been optimized and now is much more robust with faced with corrupted file systems. The ncheck command also now has a -c option which will verify the file type information in the directory entry to see if matches the inode's mode bits. This is extremely useful when trying to use debugsfs to determine which parts of the file system metadata can be trusted.
E2image will try to use ftruncate64() to set the i_size for raw images, instead of writing a single null byte. This avoid allocating an extra block to the raw image, for those file systems and/or operating systems that support this. (Linux does.) In addition, fix a logic bug that caused the file to not be properly extended if the size of the last hole was exactly an multiple of a megabyte.
Fixed a bug in resize2fs where for 1k and 2k file systems, where s_first_data_block is non-zero, this wasn't taken into account when calculate the minimum file system size for use with the -M option.
Fixed the badblocks program to honor the -s flag when in read-only -t mode. (Addresses Debian Bug #646629)
Update Czech, Dutch, French, Polish, and Sweedish translation from the Translation Project.
Fixed various Debian Packaging issues so that dpkg-buildflags is used if present, which allows e2fsprogs to be built with security hardening flags. (Addresses Debian Bugs: #654457)
Programmer's Notes
------------------
Fix a bug in ext2fs_clear_generic_bmap() when used for 32-bit bitmaps. This was only an issue for programs compiled against e2fsprogs 1.41 that manipulate bitmaps directly. (Addresses Sourceforge Bugs:
#3451486)
The libext2fs library now uses sysconf() to fetch the page size, instead of the deprecated getpagesize().
The ext2fs_get_pathname() function will return a partial path if an a directory in the path is not a directory, displaying it as an inode number in angle brackets instead of giving up and displaying an error. This is much more helpful when a user is trying to debug a corrupted file system.
Codepoints for the RO_COMPAT_REPLICA feature has been reserved.
Added a new library function, ext2fs_file_get_inode_num(), for use by fuse2fs.
Fixed a bug in ext2fs_file_set_size2() so that when it is truncating a file, it actually works.
The block iterator now properly honors the BLOCK_ABORT flag for extent-based flags. Previously, it didn't, which generally made code be less efficient, but it could cause bugs in ext2fs_link(), for example, by causing it to insert multiple directory entries.
Fixed an (harmless other than causing a compiler warning) use of an uninitialized variable in e2fsck's MMP code.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Eric Sandeen

2012-02-20 19:01 UTC #6 lore raw

Re: E2fsprogs 1.42.1 release

Quoted 20 lines · Eric Sandeen

Quoted 10 lines · Ted Ts'o
Quoted 3 lines · Eric Sandeen
Quoted 2 lines
Which is in e2fsprogs 1.42.1....
Was that the release announcement? :)
I knew I forgot something. :-)
Over the weekend I released e2fsprogs 1.42.1. It can be found in the usual places: on sourceforge, a signed git tag, and on ftp.kernel.org:/pub/linux/kernel/people/tytso/e2fsprogs/v1.42.1.
The tarball is a little messed up:
[esandeen@neon e2fsprogs]$ tar xvzf e2fsprogs-1.42.1.tar.gz e2fsprogs-libs-1.42.1/ e2fsprogs-libs-1.42.1/.gitignore e2fsprogs-libs-1.42.1/.release-checklist e2fsprogs-libs-1.42.1/lib/

Quoted 4 lines

why does it unpack into e2fsprogs-libs-1.42.1?
                                  ^^^^
I could work around that but it's unexpected.

Oh actually I can't work around it, it really is just the libs.

-Eric

Ted Ts'o

2012-02-20 19:18 UTC #7 lore raw

Re: E2fsprogs 1.42.1 release

Quoted 8 lines · Eric Sandeen

Quoted 8 lines
The tarball is a little messed up:
[esandeen@neon e2fsprogs]$ tar xvzf e2fsprogs-1.42.1.tar.gz e2fsprogs-libs-1.42.1/ e2fsprogs-libs-1.42.1/.gitignore e2fsprogs-libs-1.42.1/.release-checklist e2fsprogs-libs-1.42.1/lib/

Quoted 6 lines

Quoted 4 lines
why does it unpack into e2fsprogs-libs-1.42.1?
                                  ^^^^
I could work around that but it's unexpected.
Oh actually I can't work around it, it really is just the libs.

Whoops, kup screwup. I uploaded e2fsprogs-1.42.tar.gz and e2fsprogs-libs-1.42.tar.gz.... into the same destination file. I just fixed it on master.kernel.org; it should propagate over in 15 minutes or so.

Or you can grab the files from sourceforge. (Note that the sourceforge tarballs are actually signed by me. In contrast the kernel.org tarballs are signed by the ftp.kernel.org machine key, and the signature covers the uncompressed tar file. That way they can use the same signature file for the tar.gz and tar.bz2 tarball.)

- Ted