possible xfs corruption
Hi...
I'm stuck with a storage issue on reboot. Initially doubted the storage, but dmesg throws these errors. Now wondering whether this is a fs issue? Any thoughts as to whats going on here?
XFS: failed to locate log tail
XFS: log mount/recovery failed: error 117
XFS: log mount failed
XFS mounting filesystem cciss/c0d0
Filesystem "cciss/c0d0": XFS internal error xlog_clear_stale_blocks(2) at line 1237 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_log_recover.c. Caller 0xf8aa7892
[<f8aa774e>] xlog_find_tail+0x91b/0xa4a [xfs] [<f8aa7892>] xlog_recover+0x15/0x20b [xfs] [<f8aa7892>] xlog_recover+0x15/0x20b [xfs] [<f8ab9000>] xfs_buf_get_empty+0x2a/0x33 [xfs] [<f8aa36e5>] xfs_log_mount+0x467/0x4ab [xfs] [<f8ab702d>] kmem_zalloc+0xd/0x34 [xfs] [<f8aa9d8a>] xfs_mountfs+0x9f5/0xdfb [xfs] [<f8a92287>] xfs_fs_vcmn_err+0x5f/0x83 [xfs] [<f8abeea5>] xfs_mountfs_check_barriers+0xb8/0xd0 [xfs] [<f8ab02d2>] xfs_mount+0x75e/0x82a [xfs] [<f8aafb74>] xfs_mount+0x0/0x82a [xfs] [<f8abf57f>] vfs_mount+0x17/0x1a [xfs] [<f8abf42b>] xfs_fs_fill_super+0x68/0x1a5 [xfs] [<c04ef1a0>] snprintf+0x1c/0x1f [<c04a9b23>] disk_name+0x56/0x60 [<c047b131>] get_sb_bdev+0xc6/0x113 [<f8abe9d3>] xfs_fs_get_sb+0x12/0x16 [xfs] [<f8abf3c3>] xfs_fs_fill_super+0x0/0x1a5 [xfs] [<c047abf6>] vfs_kern_mount+0x7d/0xf2 [<c047ac9d>] do_kern_mount+0x25/0x36 [<c048e6aa>] do_mount+0x5fb/0x66b [<c04c63ab>] avc_has_perm+0x3c/0x46 [<c04c63ab>] avc_has_perm+0x3c/0x46 [<c04c7c1e>] selinux_inode_alloc_security+0x50/0x7b [<c04c7619>] inode_doinit_with_dentry+0x8a/0x495 [<c045ad0a>] get_page_from_freelist+0x96/0x370 [<c045b04d>] __alloc_pages+0x69/0x2cf [<c048d597>] copy_mount_options+0x26/0x109 [<c048e787>] sys_mount+0x6d/0xa5 [<c0404ead>] sysenter_past_esp+0x56/0x79
Thanks in advance
David
Signature · 7 lines
View this message in context: http://old.nabble.com/possible-xfs-corruption-tp30395558p30395558.html Sent from the Xfs - General mailing list archive at Nabble.com. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Le Tue, 7 Dec 2010 03:49:49 -0800 (PST)
blacknred <[email protected]> écrivait:
Quoted 4 lines
I'm stuck with a storage issue on reboot. Initially doubted the storage, but dmesg throws these errors. Now wondering whether this is a fs issue? Any thoughts as to whats going on here?
It looks like a part of the filesystem is physically missing. What is the underlying device? Apparently it's a SmartArray in some HP server, did you had a power failure?
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsYes, It's a Smart Array in HP Proliant Server.
No power failure, just upgraded the controller firmware and rebooted the server.....
Quoted 27 lines · Emmanuel Florac
Le Tue, 7 Dec 2010 03:49:49 -0800 (PST)
blacknred <[email protected]> écrivait:Quoted 4 lines
I'm stuck with a storage issue on reboot. Initially doubted the storage, but dmesg throws these errors. Now wondering whether this is a fs issue? Any thoughts as to whats going on here?
It looks like a part of the filesystem is physically missing. What is the underlying device? Apparently it's a SmartArray in some HP server, did you had a power failure?
Signature · 11 lines
------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <[email protected]> | +33 1 78 94 84 02 ------------------------------------------------------------------------ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Signature · 7 lines
View this message in context: http://old.nabble.com/possible-xfs-corruption-tp30395558p30397173.html Sent from the Xfs - General mailing list archive at Nabble.com. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Le Tue, 7 Dec 2010 07:11:04 -0800 (PST)
blacknred <[email protected]> écrivait:
Quoted 4 lines
Yes, It's a Smart Array in HP Proliant Server.
No power failure, just upgraded the controller firmware and rebooted the server.....
Hu oh, that stinks. Nothing in the firmware release notes? Do you have any other filesystem on this array?
First you could try to backup the filesystem metadata using
xfs_metadump -o -w /dev/DEVICE outfile.meta
It can be useful later in case things turn bad. Does it output any errors? IO errors particularly ?
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsIt's all xfs..... and didn't see any I/O errors as well...except that dmesg is flooded with similar traces to one i posted....
Quoted 33 lines · Emmanuel Florac
Le Tue, 7 Dec 2010 07:11:04 -0800 (PST)
blacknred <[email protected]> écrivait:Quoted 4 lines
Yes, It's a Smart Array in HP Proliant Server.
No power failure, just upgraded the controller firmware and rebooted the server.....Hu oh, that stinks. Nothing in the firmware release notes? Do you have any other filesystem on this array?
First you could try to backup the filesystem metadata using
xfs_metadump-o -w/dev/DEVICEoutfile.metaIt can be useful later in case things turn bad. Does it output any errors? IO errors particularly ?
Signature · 11 lines
------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <[email protected]> | +33 1 78 94 84 02 ------------------------------------------------------------------------ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Signature · 7 lines
View this message in context: http://old.nabble.com/possible-xfs-corruption-tp30395558p30398538.html Sent from the Xfs - General mailing list archive at Nabble.com. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
blacknred put forth on 12/7/2010 11:30 AM:
Quoted 3 lines
It's all xfs..... and didn't see any I/O errors as well...except that dmesg is flooded with similar traces to one i posted....
Out of the blue, one OP, with two Proliant servers having storage problems, within hours of one another, the same day?
Long odds, that.
Did you walk a mile, flat footed, on deep shag carpet, while wearing a wool sweater, in a room with zero humidity, then walk into the DC and touch each of these servers, or something? ;)
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines · On Dienstag, 7. Dezember 2010 blacknred wrote:
Yes, It's a Smart Array in HP Proliant Server.
No power failure, just upgraded the controller firmware and rebooted the server.....
What Server, controller, and from which firmware version did you go to which one?
Signature · 11 lines
mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services: Protéger http://proteger.at [gesprochen: Prot-e-schee] Tel: +43 660 / 415 6531 // ****** Radiointerview zum Thema Spam ****** // http://www.it-podcast.at/archiv.html#podcast-100716 // // Haus zu verkaufen: http://zmi.at/langegg/
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · blacknred
Hi...
I'm stuck with a storage issue on reboot. Initially doubted the storage, but dmesg throws these errors. Now wondering whether this is a fs issue? Any thoughts as to whats going on here?
Quoted 6 lines
XFS: failed to locate log tail
XFS: log mount/recovery failed: error 117
XFS: log mount failed
XFS mounting filesystem cciss/c0d0
Filesystem "cciss/c0d0": XFS internal errorxlog_clear_stale_blocks(2)at line 1237 of file
Which indicates that the head and/or the tail of the log are not valid. Can you provide the output of:
# xfs_logprint -d /dev/cciss/c0d0
So we can see what the head/tail values are in the log?
Quoted 1 line
/home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_log_recover.c.
CentOS kernel? How old?
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 19 lines · Dave Chinner
Quoted 8 lines · blacknred
Hi...
I'm stuck with a storage issue on reboot. Initially doubted the storage, but dmesg throws these errors. Now wondering whether this is a fs issue? Any thoughts as to whats going on here?
Quoted 6 lines
XFS: failed to locate log tail
XFS: log mount/recovery failed: error 117
XFS: log mount failed
XFS mounting filesystem cciss/c0d0
Filesystem "cciss/c0d0": XFS internal errorxlog_clear_stale_blocks(2)at line 1237 of fileWhich indicates that the head and/or the tail of the log are not valid. Can you provide the output of:
Quoted 7 lines
# xfs_logprint -d /dev/cciss/c0d0So we can see what the head/tail values are in the log?
Quoted 1 line
/home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_log_recover.c.CentOS kernel? How old?
Assuning it's centos5, there's really no need to be using an xfs kmod there anymore, the module shipped with the kernel in recent versions of the OS is really the one you want to use. That kmod is ancient and unmaintained.
Although I suspect the storage is more likely at fault here. :)
-Eric
Quoted 3 lines
Cheers,
Dave.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs