Skip to contentSkip to conversation

Filesystem corrupted: "Sorry, could not find valid secondary superblock"

DISCUSSION Nicolas STRANSKY 2 participants 8 messages Jan 29, 2010 – Feb 1, 2010

lore
Share & tools
lore
#0 lore raw

Hi,

I'm using Debian Lenny with xfsprogs 2.9.8-1lenny1 and a kernel 2.6.26-2-amd64. Recently I had some disk failures on a 1.5TB RAID 5 array, which is now running in a degraded mode but the xfs filesystem on this array is inconsistent. I can mount it with -o ro,norecovery and read some files, but I can't read most of them.
Using xfs_repair, I tried to repair the filesystem. It went for 12 hours, displaying these messages:
"found candidate secondary superblock...
unable to verify superblock, continuing..."
and finally:
"Sorry, could not find valid secondary superblock
Exiting now."

Is there anything else I could do to repair the filesystem? Upgrade to a newer version of xfsprogs? Is it possible that there is no valid secondary superblock on the whole disk or are they just not found?

Thanks a lot for any help,
NS.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 15 lines · Nicolas STRANSKY

Hi,

I'm using Debian Lenny with xfsprogs 2.9.8-1lenny1 and a kernel 2.6.26-2-amd64. Recently I had some disk failures on a 1.5TB RAID 5 array, which is now running in a degraded mode but the xfs filesystem on this array is inconsistent. I can mount it with -o ro,norecovery and read some files, but I can't read most of them.
Using xfs_repair, I tried to repair the filesystem. It went for 12 hours, displaying these messages:
"found candidate secondary superblock...
unable to verify superblock, continuing..."
and finally:
"Sorry, could not find valid secondary superblock
Exiting now."

Quoted 6 lines

Is there anything else I could do to repair the filesystem? Upgrade to a newer version of xfsprogs? Is it possible that there is no valid secondary superblock on the whole disk or are they just not found?

Thanks a lot for any help,
NS.

xfs_repair needs to be smarter about this; I at least wish it'd say -why- the candidate was not valid.

Can you include the whole repair output? Then maybe we can direct you to some xfs_db jujitsu to examine the fs.

Trying newer xfsprogs would not be a -bad- idea...

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 9 lines · Eric Sandeen

xfs_repair needs to be smarter about this; I at least wish it'd say -why- the candidate was not valid.

Can you include the whole repair output? Then maybe we can direct you to some xfs_db jujitsu to examine the fs.

Trying newer xfsprogs would not be a -bad- idea...

-Eric

The output is as follows:

# xfs_repair /dev/sda
Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

attempting to find secondary superblock...
......................
found candidate secondary superblock...
unable to verify superblock, continuing...
......................
found candidate secondary superblock...
unable to verify superblock, continuing...
......................
[The previous one repeats hundreds of times over many hours]
......................
Sorry, could not find valid secondary superblock Exiting now.

I am trying right now xfsprogs 3.1.0 from Debian testing, will see what it gives...

Thanks!
NS

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
Quoted 9 lines · Nicolas Stransky
Quoted 8 lines · Eric Sandeen

xfs_repair needs to be smarter about this; I at least wish it'd say -why- the candidate was not valid.

Can you include the whole repair output? Then maybe we can direct you to some xfs_db jujitsu to examine the fs.

Trying newer xfsprogs would not be a -bad- idea...

Quoted 7 lines
Quoted 1 line

-Eric

The output is as follows:

# xfs_repair /dev/sda
Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

xfs_db /dev/sda
xfs_db> sb 0
xfs_db> p

might be interesting to see what's there.

-Eric

Quoted 17 lines

attempting to find secondary superblock...
......................
found candidate secondary superblock...
unable to verify superblock, continuing...
......................
found candidate secondary superblock...
unable to verify superblock, continuing...
......................
[The previous one repeats hundreds of times over many hours]
......................
Sorry, could not find valid secondary superblock Exiting now.

I am trying right now xfsprogs 3.1.0 from Debian testing, will see what it gives...

Quoted 8 lines

Thanks!
NS

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw

OK here is a bunch of new stuff:

# xfs_repair /dev/sda1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs
to be replayed.  Mount the filesystem to replay the log, and unmount it
before re-running xfs_repair.  If you are unable to mount the
filesystem, then use the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a
mount of the filesystem before doing this.
# mount/ dev/sda1
mount: Structure needs cleaning

Is the use of -L the only way to go? I wonder, since it will cause data loss...

Also here is the output of xfs_db:

# xfs_db /dev/sda1
xfs_db> sb 0
xfs_db> p
magicnum = 0x58465342
blocksize = 4096
dblocks = 366197650
rblocks = 0
rextents = 0
uuid = dab73302-1744-4fc0-89db-5c6670a02aa0
logstart = 134217732
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 16
agblocks = 6866155
agcount = 54
rbmblocks = 0
logblocks = 32768
versionnum = 0x3484
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 23
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 4115008
ifree = 954636
fdblocks = 85242778
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 1
features2 = 0
bad_features2 = 0

Thanks!

Quoted 52 lines · Eric Sandeen
Quoted 9 lines · Nicolas Stransky
Quoted 8 lines · Eric Sandeen

xfs_repair needs to be smarter about this; I at least wish it'd say -why- the candidate was not valid.

Can you include the whole repair output? Then maybe we can direct you to some xfs_db jujitsu to examine the fs.

Trying newer xfsprogs would not be a -bad- idea...

Quoted 7 lines
Quoted 1 line

-Eric

The output is as follows:

# xfs_repair /dev/sda
Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

xfs_db /dev/sda
xfs_db> sb 0
xfs_db> p

might be interesting to see what's there.

-Eric

Quoted 17 lines

attempting to find secondary superblock...
......................
found candidate secondary superblock...
unable to verify superblock, continuing...
......................
found candidate secondary superblock...
unable to verify superblock, continuing...
......................
[The previous one repeats hundreds of times over many hours]
......................
Sorry, could not find valid secondary superblock Exiting now.

I am trying right now xfsprogs 3.1.0 from Debian testing, will see what it gives...

Quoted 8 lines

Thanks!
NS

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoted 5 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw
Quoted 13 lines · Nicolas Stransky

OK here is a bunch of new stuff:

# xfs_repair /dev/sda1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs
to be replayed.  Mount the filesystem to replay the log, and unmount it
before re-running xfs_repair.  If you are unable to mount the
filesystem, then use the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a
mount of the filesystem before doing this.
Quoted 2 lines
# mount/ dev/sda1
mount: Structure needs cleaning

dmesg at this point would be good

Quoted 3 lines

Is the use of -L the only way to go? I wonder, since it will cause data loss...

if you can't mount it, yeah

Quoted 7 lines

Also here is the output of xfs_db:

# xfs_db /dev/sda1
xfs_db> sb 0
xfs_db> p
magicnum = 0x58465342

ok, odd; you got:

Quoted 3 lines
Quoted 3 lines
# xfs_repair /dev/sda
Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

but this magic number is fine ...

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw

Hi Eric,

Thanks so much for your help,

Quoted 3 lines · Eric Sandeen
Quoted 1 line
# mount/ dev/sda1 mount: Structure needs cleaning

dmesg at this point would be good

This particular command does not produce any output in dmesg, but previous commands did, especially when I mounted the filesystem with no-recovery.

[205411.967412] XFS: log mount/recovery failed: error 117
[205411.972696] XFS: log mount failed
[205469.205586] Mounting filesystem "sda1" in no-recovery mode.
Filesystem will be inconsistent.
[205483.946417] XFS mounting filesystem sda1
[205484.136159] Starting XFS recovery on filesystem: sda1 (logdev: internal)
[205484.488176] Filesystem "sda1": xfs_inode_recover: Bad inode magic
number, dino ptr = 0xffff81004d109f00, dino bp = 0xffff81002e85be80, ino
= 2954558655
[205484.493476] Filesystem "sda1": XFS internal error
xlog_recover_do_inode_trans(1) at line 2326 of file
fs/xfs/xfs_log_recover.c.  Caller 0xffffffffa018553a
[205484.498909] Pid: 27800, comm: mount Not tainted 2.6.26-2-amd64 #1
[205484.506907]
[205484.506908] Call Trace:
[205484.516417]  [<ffffffffa018553a>] :xfs:xlog_recover_do_trans+0x64/0xfd
[205484.523162]  [<ffffffffa0184e33>]
:xfs:xlog_recover_do_inode_trans+0x23c/0x766
[205484.528729]  [<ffffffffa01966b0>] :xfs:xfs_buf_iostart+0x29/0x86
[205484.534304]  [<ffffffffa018553a>] :xfs:xlog_recover_do_trans+0x64/0xfd
[205484.539907]  [<ffffffffa0185605>]
:xfs:xlog_recover_commit_trans+0x32/0x4b
[205484.545489]  [<ffffffffa018576b>]
:xfs:xlog_recover_process_data+0x14d/0x1c9
[205484.551117]  [<ffffffffa0185a2f>] :xfs:xlog_do_recovery_pass+0x248/0x661
[205484.556736]  [<ffffffffa0185e9a>] :xfs:xlog_do_log_recovery+0x52/0x75
[205484.564000]  [<ffffffffa0185ec9>] :xfs:xlog_do_recover+0xc/0xf3
[205484.568026]  [<ffffffffa018602a>] :xfs:xlog_recover+0x7a/0x84
[205484.573631]  [<ffffffffa0180460>] :xfs:xfs_log_mount+0xb1/0x105
[205484.579270]  [<ffffffffa01880dc>] :xfs:xfs_mountfs+0x25a/0x5ac
[205484.584866]  [<ffffffffa0193311>] :xfs:kmem_alloc+0x60/0xc4
[205484.590483]  [<ffffffffa01933d1>] :xfs:kmem_zalloc+0x9/0x21
[205484.596066]  [<ffffffffa018deb4>] :xfs:xfs_mount+0x29b/0x347
[205484.601625]  [<ffffffffa019c3ea>] :xfs:xfs_fs_fill_super+0x0/0x1ee
[205484.607231]  [<ffffffffa019c49f>] :xfs:xfs_fs_fill_super+0xb5/0x1ee
[205484.612727]  [<ffffffff8029d290>] get_sb_bdev+0xf8/0x145
[205484.618174]  [<ffffffff8029ccb4>] vfs_kern_mount+0x93/0x11b
[205484.623745]  [<ffffffff8029cd8f>] do_kern_mount+0x43/0xe3
[205484.629183]  [<ffffffff802b1805>] do_new_mount+0x5b/0x95
[205484.634627]  [<ffffffff802b19fc>] do_mount+0x1bd/0x1e7
[205484.639990]  [<ffffffff80276909>] __alloc_pages_internal+0xd6/0x3bf
[205484.645219]  [<ffffffff802b1ab0>] sys_mount+0x8a/0xce
[205484.650275]  [<ffffffff8020beca>] system_call_after_swapgs+0x8a/0x8f
Quoted 4 lines
Quoted 2 lines

Is the use of -L the only way to go? I wonder, since it will cause data loss...

if you can't mount it, yeah

I'm going ahead and doing that.
There is an awful lot of output but at least it's doing something.

Quoted 10 lines
Quoted 3 lines

Also here is the output of xfs_db:

# xfs_db /dev/sda1 xfs_db> sb 0 xfs_db> p magicnum = 0x58465342

ok, odd; you got:

Quoted 2 lines
Quoted 2 lines
# xfs_repair /dev/sda Phase 1 - find and verify superblock... bad
primary superblock - bad magic number !!!

but this magic number is fine ...

Yes, I figured that I had used /dev/sda instead of /dev/sda1, my bad :( So now the magic number is ok...

Signature · 6 lines
Nico

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#7 lore raw
In reply to Eric Sandeen

Hello,

Unfortunately I am still having big problems with this filesystem. I can mount it OK but there are problems reading files:

du: cannot access `./DCIS 2 AB': Structure needs cleaning

The previous goes together with a lot of these in dmesg:

[  431.976517] 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 ................
[  431.988072] Filesystem "sda1": XFS internal error xfs_da_do_buf(2) at
line 2085 of file fs/xfs/xfs_da_btree.c.  Caller 0xffffffffa016749f
[  431.993185] Pid: 4246, comm: du Not tainted 2.6.26-2-amd64 #1
[  431.998308]
[  431.998309] Call Trace:
[  432.010348]  [<ffffffffa016749f>] :xfs:xfs_da_read_buf+0x24/0x29
[  432.015530]  [<ffffffffa016734d>] :xfs:xfs_da_do_buf+0x54e/0x636
[  432.020696]  [<ffffffffa016749f>] :xfs:xfs_da_read_buf+0x24/0x29
[  432.025860]  [<ffffffffa019580c>] :xfs:_xfs_buf_lookup_pages+0x298/0x2ca
[  432.031974]  [<ffffffffa016749f>] :xfs:xfs_da_read_buf+0x24/0x29
[  432.037173]  [<ffffffffa016c66f>] :xfs:xfs_dir2_leaf_getdents+0x381/0x61a
[  432.042391]  [<ffffffffa016c66f>] :xfs:xfs_dir2_leaf_getdents+0x381/0x61a
[  432.047617]  [<ffffffffa0196f68>] :xfs:xfs_hack_filldir+0x0/0x5b
[  432.052673]  [<ffffffffa0196f68>] :xfs:xfs_hack_filldir+0x0/0x5b
[  432.057517]  [<ffffffffa01695d8>] :xfs:xfs_readdir+0xa6/0xb5
[  432.062197]  [<ffffffffa0196f1b>] :xfs:xfs_file_readdir+0xff/0x14c
[  432.066835]  [<ffffffff802a6bec>] filldir+0x0/0xb7
[  432.071987]  [<ffffffff802a6bec>] filldir+0x0/0xb7
[  432.076557]  [<ffffffff802a6d18>] vfs_readdir+0x75/0xa7
[  432.081137]  [<ffffffff802a6f68>] sys_getdents+0x75/0xbd
[  432.085716]  [<ffffffff802a65da>] sys_fcntl+0x2eb/0x2f7
[  432.090267]  [<ffffffff8020beca>] system_call_after_swapgs+0x8a/0x8f

The server has 2GB of RAM and 4GB of swap. I have tried several times to xfs_repair but it always fails because of memory problems, even if I use 'xfs_repair -m1500 /dev/sda1' to limit the memory usage. xfs_repair produces an awful lot of output. I could of course send it but it's 130k lines long. Here is what it is currently printing, I have started "xfs_repair -t15 -m500 /dev/sda1" again a few minutes ago. It is currently in Phase 3.

bp=(bno 1211109312, len 8192 bytes) key=(bno 1211109312, len 4096 bytes) 429f7950: Badness in key lookup (length)
bp=(bno 1211109328, len 8192 bytes) key=(bno 1211109328, len 4096 bytes) bad directory block magic # 0x2f49cdbc in block 2 for directory inode 2952790372
corrupt block 2 in directory inode 2952790372
will junk block
bad directory block magic # 0 in block 5 for directory inode 2952790372 corrupt block 5 in directory inode 2952790372
will junk block
no . entry for directory 2952790372
no .. entry for directory 2952790372
bad directory block magic # 0 in block 0 for directory inode 2952790373 corrupt block 0 in directory inode 2952790373
will junk block
bad directory block magic # 0 in block 1 for directory inode 2952790373 corrupt block 1 in directory inode 2952790373
will junk block
doubling cache size to 66768
doubling cache size to 133536
doubling cache size to 267072
doubling cache size to 534144
doubling cache size to 1068288

Any ideas on how I could repair this filesystem?

Thanks a lot!

Signature · 6 lines
Nico

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs