Filesystem corrupted: "Sorry, could not find valid secondary superblock"
Hi,
I'm using Debian Lenny with xfsprogs 2.9.8-1lenny1 and a kernel 2.6.26-2-amd64. Recently I had some disk failures on a 1.5TB RAID 5 array, which is now running in a degraded mode but the xfs filesystem on this array is inconsistent. I can mount it with -o ro,norecovery and read some files, but I can't read most of them.
Using xfs_repair, I tried to repair the filesystem. It went for 12 hours, displaying these messages:
"found candidate secondary superblock...
unable to verify superblock, continuing..."
and finally:
"Sorry, could not find valid secondary superblock
Exiting now."
Is there anything else I could do to repair the filesystem? Upgrade to a newer version of xfsprogs? Is it possible that there is no valid secondary superblock on the whole disk or are they just not found?
Thanks a lot for any help,
NS.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 15 lines · Nicolas STRANSKY
Hi,
I'm using Debian Lenny with xfsprogs 2.9.8-1lenny1 and a kernel 2.6.26-2-amd64. Recently I had some disk failures on a 1.5TB RAID 5 array, which is now running in a degraded mode but the xfs filesystem on this array is inconsistent. I can mount it with -o ro,norecovery and read some files, but I can't read most of them.
Usingxfs_repair, I tried to repair the filesystem. It went for 12 hours, displaying these messages:
"found candidate secondary superblock...
unable to verify superblock, continuing..."
and finally:
"Sorry, could not find valid secondary superblock
Exiting now."
Quoted 6 lines
Is there anything else I could do to repair the filesystem? Upgrade to a newer version of xfsprogs? Is it possible that there is no valid secondary superblock on the whole disk or are they just not found?
Thanks a lot for any help,
NS.
xfs_repair needs to be smarter about this; I at least wish it'd say -why- the candidate was not valid.
Can you include the whole repair output? Then maybe we can direct you to some xfs_db jujitsu to examine the fs.
Trying newer xfsprogs would not be a -bad- idea...
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Eric Sandeen
xfs_repairneeds to be smarter about this; I at least wish it'd say -why- the candidate was not valid.Can you include the whole repair output? Then maybe we can direct you to some
xfs_dbjujitsu to examine the fs.Trying newer xfsprogs would not be a -bad- idea...
-Eric
The output is as follows:
# xfs_repair /dev/sda Phase 1 - find and verify superblock... bad primary superblock - bad magic number !!!
attempting to find secondary superblock...
......................
found candidate secondary superblock...
unable to verify superblock, continuing...
......................
found candidate secondary superblock...
unable to verify superblock, continuing...
......................
[The previous one repeats hundreds of times over many hours]
......................
Sorry, could not find valid secondary superblock Exiting now.
I am trying right now xfsprogs 3.1.0 from Debian testing, will see what it gives...
Thanks!
NS
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Nicolas Stransky
Quoted 8 lines · Eric Sandeen
xfs_repairneeds to be smarter about this; I at least wish it'd say -why- the candidate was not valid.Can you include the whole repair output? Then maybe we can direct you to some
xfs_dbjujitsu to examine the fs.Trying newer xfsprogs would not be a -bad- idea...
Quoted 7 lines
Quoted 1 line
-Eric
The output is as follows:
# xfs_repair /dev/sda Phase 1 - find and verify superblock... bad primary superblock - bad magic number !!!
xfs_db /dev/sdaxfs_db> sb 0xfs_db> p
might be interesting to see what's there.
-Eric
Quoted 17 lines
attempting to find secondary superblock...
......................
found candidate secondary superblock...
unable to verify superblock, continuing...
......................
found candidate secondary superblock...
unable to verify superblock, continuing...
......................
[The previous one repeats hundreds of times over many hours]
......................
Sorry, could not find valid secondary superblock Exiting now.I am trying right now xfsprogs 3.1.0 from Debian testing, will see what it gives...
Quoted 8 lines
Thanks!
NS_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
OK here is a bunch of new stuff:
# xfs_repair /dev/sda1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs
to be replayed. Mount the filesystem to replay the log, and unmount it
before re-running xfs_repair. If you are unable to mount the
filesystem, then use the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a
mount of the filesystem before doing this.# mount/ dev/sda1 mount: Structure needs cleaning
Is the use of -L the only way to go? I wonder, since it will cause data loss...
Also here is the output of xfs_db:
# xfs_db /dev/sda1 xfs_db> sb 0 xfs_db> p magicnum = 0x58465342 blocksize = 4096 dblocks = 366197650 rblocks = 0 rextents = 0 uuid = dab73302-1744-4fc0-89db-5c6670a02aa0 logstart = 134217732 rootino = 128 rbmino = 129 rsumino = 130 rextsize = 16 agblocks = 6866155 agcount = 54 rbmblocks = 0 logblocks = 32768 versionnum = 0x3484 sectsize = 512 inodesize = 256 inopblock = 16 fname = "\000\000\000\000\000\000\000\000\000\000\000\000" blocklog = 12 sectlog = 9 inodelog = 8 inopblog = 4 agblklog = 23 rextslog = 0 inprogress = 0 imax_pct = 25 icount = 4115008 ifree = 954636 fdblocks = 85242778 frextents = 0 uquotino = 0 gquotino = 0 qflags = 0 flags = 0 shared_vn = 0 inoalignmt = 2 unit = 0 width = 0 dirblklog = 0 logsectlog = 0 logsectsize = 0 logsunit = 1 features2 = 0 bad_features2 = 0
Thanks!
Quoted 52 lines · Eric Sandeen
Quoted 9 lines · Nicolas Stransky
Quoted 8 lines · Eric Sandeen
xfs_repairneeds to be smarter about this; I at least wish it'd say -why- the candidate was not valid.Can you include the whole repair output? Then maybe we can direct you to some
xfs_dbjujitsu to examine the fs.Trying newer xfsprogs would not be a -bad- idea...
Quoted 7 lines
Quoted 1 line
-Eric
The output is as follows:
# xfs_repair /dev/sda Phase 1 - find and verify superblock... bad primary superblock - bad magic number !!!
xfs_db/dev/sdaxfs_db> sb 0xfs_db> pmight be interesting to see what's there.
-Eric
Quoted 17 lines
attempting to find secondary superblock...
......................
found candidate secondary superblock...
unable to verify superblock, continuing...
......................
found candidate secondary superblock...
unable to verify superblock, continuing...
......................
[The previous one repeats hundreds of times over many hours]
......................
Sorry, could not find valid secondary superblock Exiting now.I am trying right now xfsprogs 3.1.0 from Debian testing, will see what it gives...
Quoted 8 lines
Thanks!
NS_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 5 lines
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 13 lines · Nicolas Stransky
OK here is a bunch of new stuff:
# xfs_repair /dev/sda1 Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.
Quoted 2 lines
# mount/ dev/sda1 mount: Structure needs cleaning
dmesg at this point would be good
Quoted 3 lines
Is the use of -L the only way to go? I wonder, since it will cause data loss...
if you can't mount it, yeah
Quoted 7 lines
Also here is the output of
xfs_db:# xfs_db /dev/sda1 xfs_db> sb 0 xfs_db> p magicnum = 0x58465342
ok, odd; you got:
Quoted 3 lines
Quoted 3 lines
# xfs_repair /dev/sda Phase 1 - find and verify superblock... bad primary superblock - bad magic number !!!
but this magic number is fine ...
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Hi Eric,
Thanks so much for your help,
Quoted 3 lines · Eric Sandeen
Quoted 1 line
# mount/ dev/sda1 mount: Structure needs cleaningdmesg at this point would be good
This particular command does not produce any output in dmesg, but previous commands did, especially when I mounted the filesystem with no-recovery.
[205411.967412] XFS: log mount/recovery failed: error 117 [205411.972696] XFS: log mount failed [205469.205586] Mounting filesystem "sda1" in no-recovery mode. Filesystem will be inconsistent. [205483.946417] XFS mounting filesystem sda1 [205484.136159] Starting XFS recovery on filesystem: sda1 (logdev: internal) [205484.488176] Filesystem "sda1": xfs_inode_recover: Bad inode magic number, dino ptr = 0xffff81004d109f00, dino bp = 0xffff81002e85be80, ino = 2954558655 [205484.493476] Filesystem "sda1": XFS internal error xlog_recover_do_inode_trans(1) at line 2326 of file fs/xfs/xfs_log_recover.c. Caller 0xffffffffa018553a [205484.498909] Pid: 27800, comm: mount Not tainted 2.6.26-2-amd64 #1 [205484.506907] [205484.506908] Call Trace: [205484.516417] [<ffffffffa018553a>] :xfs:xlog_recover_do_trans+0x64/0xfd [205484.523162] [<ffffffffa0184e33>] :xfs:xlog_recover_do_inode_trans+0x23c/0x766 [205484.528729] [<ffffffffa01966b0>] :xfs:xfs_buf_iostart+0x29/0x86 [205484.534304] [<ffffffffa018553a>] :xfs:xlog_recover_do_trans+0x64/0xfd [205484.539907] [<ffffffffa0185605>] :xfs:xlog_recover_commit_trans+0x32/0x4b [205484.545489] [<ffffffffa018576b>] :xfs:xlog_recover_process_data+0x14d/0x1c9 [205484.551117] [<ffffffffa0185a2f>] :xfs:xlog_do_recovery_pass+0x248/0x661 [205484.556736] [<ffffffffa0185e9a>] :xfs:xlog_do_log_recovery+0x52/0x75 [205484.564000] [<ffffffffa0185ec9>] :xfs:xlog_do_recover+0xc/0xf3 [205484.568026] [<ffffffffa018602a>] :xfs:xlog_recover+0x7a/0x84 [205484.573631] [<ffffffffa0180460>] :xfs:xfs_log_mount+0xb1/0x105 [205484.579270] [<ffffffffa01880dc>] :xfs:xfs_mountfs+0x25a/0x5ac [205484.584866] [<ffffffffa0193311>] :xfs:kmem_alloc+0x60/0xc4 [205484.590483] [<ffffffffa01933d1>] :xfs:kmem_zalloc+0x9/0x21 [205484.596066] [<ffffffffa018deb4>] :xfs:xfs_mount+0x29b/0x347 [205484.601625] [<ffffffffa019c3ea>] :xfs:xfs_fs_fill_super+0x0/0x1ee [205484.607231] [<ffffffffa019c49f>] :xfs:xfs_fs_fill_super+0xb5/0x1ee [205484.612727] [<ffffffff8029d290>] get_sb_bdev+0xf8/0x145 [205484.618174] [<ffffffff8029ccb4>] vfs_kern_mount+0x93/0x11b [205484.623745] [<ffffffff8029cd8f>] do_kern_mount+0x43/0xe3 [205484.629183] [<ffffffff802b1805>] do_new_mount+0x5b/0x95 [205484.634627] [<ffffffff802b19fc>] do_mount+0x1bd/0x1e7 [205484.639990] [<ffffffff80276909>] __alloc_pages_internal+0xd6/0x3bf [205484.645219] [<ffffffff802b1ab0>] sys_mount+0x8a/0xce [205484.650275] [<ffffffff8020beca>] system_call_after_swapgs+0x8a/0x8f
Quoted 4 lines
Quoted 2 lines
Is the use of -L the only way to go? I wonder, since it will cause data loss...
if you can't mount it, yeah
I'm going ahead and doing that.
There is an awful lot of output but at least it's doing something.
Quoted 10 lines
Quoted 3 lines
Also here is the output of
xfs_db:# xfs_db /dev/sda1 xfs_db> sb 0 xfs_db> p magicnum = 0x58465342ok, odd; you got:
Quoted 2 lines
Quoted 2 lines
# xfs_repair /dev/sda Phase 1 - find and verify superblock... bad primary superblock - bad magic number !!!but this magic number is fine ...
Yes, I figured that I had used /dev/sda instead of /dev/sda1, my bad :( So now the magic number is ok...
Signature · 6 lines
Nico _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Hello,
Unfortunately I am still having big problems with this filesystem. I can mount it OK but there are problems reading files:
du: cannot access `./DCIS 2 AB': Structure needs cleaning
The previous goes together with a lot of these in dmesg:
[ 431.976517] 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 431.988072] Filesystem "sda1": XFS internal error xfs_da_do_buf(2) at line 2085 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffffa016749f [ 431.993185] Pid: 4246, comm: du Not tainted 2.6.26-2-amd64 #1 [ 431.998308] [ 431.998309] Call Trace: [ 432.010348] [<ffffffffa016749f>] :xfs:xfs_da_read_buf+0x24/0x29 [ 432.015530] [<ffffffffa016734d>] :xfs:xfs_da_do_buf+0x54e/0x636 [ 432.020696] [<ffffffffa016749f>] :xfs:xfs_da_read_buf+0x24/0x29 [ 432.025860] [<ffffffffa019580c>] :xfs:_xfs_buf_lookup_pages+0x298/0x2ca [ 432.031974] [<ffffffffa016749f>] :xfs:xfs_da_read_buf+0x24/0x29 [ 432.037173] [<ffffffffa016c66f>] :xfs:xfs_dir2_leaf_getdents+0x381/0x61a [ 432.042391] [<ffffffffa016c66f>] :xfs:xfs_dir2_leaf_getdents+0x381/0x61a [ 432.047617] [<ffffffffa0196f68>] :xfs:xfs_hack_filldir+0x0/0x5b [ 432.052673] [<ffffffffa0196f68>] :xfs:xfs_hack_filldir+0x0/0x5b [ 432.057517] [<ffffffffa01695d8>] :xfs:xfs_readdir+0xa6/0xb5 [ 432.062197] [<ffffffffa0196f1b>] :xfs:xfs_file_readdir+0xff/0x14c [ 432.066835] [<ffffffff802a6bec>] filldir+0x0/0xb7 [ 432.071987] [<ffffffff802a6bec>] filldir+0x0/0xb7 [ 432.076557] [<ffffffff802a6d18>] vfs_readdir+0x75/0xa7 [ 432.081137] [<ffffffff802a6f68>] sys_getdents+0x75/0xbd [ 432.085716] [<ffffffff802a65da>] sys_fcntl+0x2eb/0x2f7 [ 432.090267] [<ffffffff8020beca>] system_call_after_swapgs+0x8a/0x8f
The server has 2GB of RAM and 4GB of swap. I have tried several times to xfs_repair but it always fails because of memory problems, even if I use 'xfs_repair -m1500 /dev/sda1' to limit the memory usage. xfs_repair produces an awful lot of output. I could of course send it but it's 130k lines long. Here is what it is currently printing, I have started "xfs_repair -t15 -m500 /dev/sda1" again a few minutes ago. It is currently in Phase 3.
bp=(bno 1211109312, len 8192 bytes) key=(bno 1211109312, len 4096 bytes) 429f7950: Badness in key lookup (length)
bp=(bno 1211109328, len 8192 bytes) key=(bno 1211109328, len 4096 bytes) bad directory block magic # 0x2f49cdbc in block 2 for directory inode 2952790372
corrupt block 2 in directory inode 2952790372
will junk block
bad directory block magic # 0 in block 5 for directory inode 2952790372 corrupt block 5 in directory inode 2952790372
will junk block
no . entry for directory 2952790372
no .. entry for directory 2952790372
bad directory block magic # 0 in block 0 for directory inode 2952790373 corrupt block 0 in directory inode 2952790373
will junk block
bad directory block magic # 0 in block 1 for directory inode 2952790373 corrupt block 1 in directory inode 2952790373
will junk block
doubling cache size to 66768
doubling cache size to 133536
doubling cache size to 267072
doubling cache size to 534144
doubling cache size to 1068288
Any ideas on how I could repair this filesystem?
Thanks a lot!
Signature · 6 lines
Nico _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs