EXT4-fs: group descriptors corrupted!

Hello,

I have an EXT4 filesystem on a raid6 that I created with 2.6.28. After the initial creation of the partition I later did 3 filesystem expands after adding drives. I didn't have a problem until I recently had a kernel panic and rebooted. I think the kernel panic was related to my nvidia driver. After rebooting the raid no longer mounted, and dmesg reported:

EXT4-fs: ext4_check_descriptors: Inode bitmap for group 0 not in group (block 3245938880)!
EXT4-fs: group descriptors corrupted!

After reading several threads online, I attempted a fsck pointing to a backup superblock and now I receive this error:

EXT4-fs: ext4_check_descriptors: Checksum for group 0 failed (7390!=34008)
EXT4-fs: group descriptors corrupted!

using dd if=/dev/md1|strings I can see a number of the files on the disk.

Running Gentoo:

Linux server1 2.6.28.5 #5 SMP Mon Feb 23 00:52:10 EST 2009 x86_64 Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz GenuineIntel GNU/Linux

e2fsprogs 1.41.3

6GB memory
6GB swap

# dumpe2fs /dev/md1
dumpe2fs 1.41.3 (12-Oct-2008)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          1b9e0aec-79b4-48e1-b801-54a2792ef9b3
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index
filetype extent flex_bg sparse_super large_file huge_file uninit_bg
dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              549429248
Block count:              2197703904
Reserved block count:     0
Free blocks:              1027127451
Free inodes:              545908446
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      500
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Wed Feb 18 16:40:04 2009
Last mount time:          Wed Feb 18 17:07:50 2009
Last write time:          Fri Mar  6 22:11:15 2009
Mount count:              1
Maximum mount count:      35
Last checked:             Wed Feb 18 16:40:04 2009
Check interval:           15552000 (6 months)
Next check after:         Mon Aug 17 17:40:04 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      27ba6512-da53-49ba-abdf-f79299a6eba2
Journal backup:           inode blocks
Journal size:             128M

 # dumpe2fs -o superblock=32768 /dev/md1
 dumpe2fs 1.41.3 (12-Oct-2008)
 Filesystem volume name:   <none>
 Last mounted on:          <not available>
 Filesystem UUID:          1b9e0aec-79b4-48e1-b801-54a2792ef9b3
 Filesystem magic number:  0xEF53
 Filesystem revision #:    1 (dynamic)
 Filesystem features:      has_journal ext_attr resize_inode dir_index
 filetype needs_recovery extent flex_bg sparse_super large_file
 huge_file uninit_bg dir_nlink extra_isize
 Filesystem flags:         signed_directory_hash 
 Default mount options:    (none)
 Filesystem state:         clean
 Errors behavior:          Continue
 Filesystem OS type:       Linux
 Inode count:              549429248
 Block count:              2197703904
 Reserved block count:     0
 Free blocks:              1027127451
 Free inodes:              545908446
 First block:              0
 Block size:               4096
 Fragment size:            4096
 Reserved GDT blocks:      500
 Blocks per group:         32768
 Fragments per group:      32768
 Inodes per group:         8192
 Inode blocks per group:   512
 Flex block group size:    16
 Filesystem created:       Wed Feb 18 16:40:04 2009
 Last mount time:          Wed Feb 18 17:07:50 2009
 Last write time:          Wed Feb 18 17:07:50 2009
 Mount count:              1
 Maximum mount count:      35
 Last checked:             Wed Feb 18 16:40:04 2009
 Check interval:           15552000 (6 months)
 Next check after:         Mon Aug 17 17:40:04 2009
 Reserved blocks uid:      0 (user root)
 Reserved blocks gid:      0 (group root)
 First inode:              11
 Inode size:               256
 Required extra isize:     28
 Desired extra isize:      28
 Journal inode:            8
 Default directory hash:   half_md4
 Directory Hash Seed:      27ba6512-da53-49ba-abdf-f79299a6eba2
 Journal backup:           inode blocks
 Journal size:             128M

I'm stuck right now and hopefully I can recover the filesystem.

Any help would be appreciated.

Thanks

I fixed this problem using fsck, but I will detail what occured incase someone else comes across the same problem.

Running fsck -n on the disk gave me the following:

fsck 1.41.4 (27-Jan-2009)

Group descriptor 0 checksum is invalid.  Fix? no
Group descriptor 1 checksum is invalid.  Fix? no
Group descriptor 2 checksum is invalid.  Fix? no

[33,000+ of these..]
Group descriptor 33534 checksum is invalid. Fix? no Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

Free blocks count wrong for group #0 (23513, counted=1880). Fix? no
Free blocks count wrong for group #1 (31743, counted=684). Fix? no
[58,000+ of these..]
Free blocks count wrong for group #58287 (32254, counted=5095). Fix? no

Free inodes count wrong for group #0 (8181, counted=0). Fix? no
Directories count wrong for group #0 (2, counted=110). Fix? no
[751 of these..]
Free inodes count wrong for group #913 (8192, counted=3376). Fix? no
Directories count wrong for group #913 (0, counted=160). Fix? no
Free inodes count wrong for group #928 (8192, counted=4034). Fix? no
Directories count wrong for group #928 (0, counted=548). Fix? no
Free inodes count wrong (545908446, counted=546390043). Fix? no

/dev/md1: ********** WARNING: Filesystem still has errors **********

/dev/md1: 3520802/549429248 files (5.2% non-contiguous), 1170576453/2197703904 blocks

Next was running fsck -y with fingers crossed..

fsck 1.41.4 (27-Jan-2009)
Group descriptor 0 checksum is invalid. Fix? yes

[same thing as above but yes..]

Directories count wrong for group #928 (0, counted=548). Fix? yes
Free inodes count wrong (549429237, counted=546390043). Fix? yes
/dev/md1: ***** FILE SYSTEM WAS MODIFIED ***** /dev/md1: 3039205/549429248 files (6.0% non-contiguous), 1252924969/2197703904 blocks

After this the file system was back to normal, with no data loss. I'm thinking that something screwy occured during the 3 times that I ran resize2fs (live) to expand the ext4 filesystem. I initially started with 5 1.5TB drives, and then added three more 1.5TB drives one-by-one running resize2fs each time. During all of this I never unmounted the device or rebooted the system.

-Christian