Skip to contentSkip to conversation

possible xfs corruption

DISCUSSION blacknred 6 participants 9 messages Dec 7, 2010 – Dec 9, 2010

lore
Share & tools
lore

blacknred

author
#0 lore raw

Hi...

I'm stuck with a storage issue on reboot. Initially doubted the storage, but dmesg throws these errors. Now wondering whether this is a fs issue? Any thoughts as to whats going on here?

XFS: failed to locate log tail
XFS: log mount/recovery failed: error 117
XFS: log mount failed
XFS mounting filesystem cciss/c0d0
Filesystem "cciss/c0d0": XFS internal error xlog_clear_stale_blocks(2) at line 1237 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_log_recover.c. Caller 0xf8aa7892

 [<f8aa774e>] xlog_find_tail+0x91b/0xa4a [xfs]
 [<f8aa7892>] xlog_recover+0x15/0x20b [xfs]
 [<f8aa7892>] xlog_recover+0x15/0x20b [xfs]
 [<f8ab9000>] xfs_buf_get_empty+0x2a/0x33 [xfs]
 [<f8aa36e5>] xfs_log_mount+0x467/0x4ab [xfs]
 [<f8ab702d>] kmem_zalloc+0xd/0x34 [xfs]
 [<f8aa9d8a>] xfs_mountfs+0x9f5/0xdfb [xfs]
 [<f8a92287>] xfs_fs_vcmn_err+0x5f/0x83 [xfs]
 [<f8abeea5>] xfs_mountfs_check_barriers+0xb8/0xd0 [xfs]
 [<f8ab02d2>] xfs_mount+0x75e/0x82a [xfs]
 [<f8aafb74>] xfs_mount+0x0/0x82a [xfs]
 [<f8abf57f>] vfs_mount+0x17/0x1a [xfs]
 [<f8abf42b>] xfs_fs_fill_super+0x68/0x1a5 [xfs]
 [<c04ef1a0>] snprintf+0x1c/0x1f
 [<c04a9b23>] disk_name+0x56/0x60
 [<c047b131>] get_sb_bdev+0xc6/0x113
 [<f8abe9d3>] xfs_fs_get_sb+0x12/0x16 [xfs]
 [<f8abf3c3>] xfs_fs_fill_super+0x0/0x1a5 [xfs]
 [<c047abf6>] vfs_kern_mount+0x7d/0xf2
 [<c047ac9d>] do_kern_mount+0x25/0x36
 [<c048e6aa>] do_mount+0x5fb/0x66b
 [<c04c63ab>] avc_has_perm+0x3c/0x46
 [<c04c63ab>] avc_has_perm+0x3c/0x46
 [<c04c7c1e>] selinux_inode_alloc_security+0x50/0x7b
 [<c04c7619>] inode_doinit_with_dentry+0x8a/0x495
 [<c045ad0a>] get_page_from_freelist+0x96/0x370
 [<c045b04d>] __alloc_pages+0x69/0x2cf
 [<c048d597>] copy_mount_options+0x26/0x109
 [<c048e787>] sys_mount+0x6d/0xa5
 [<c0404ead>] sysenter_past_esp+0x56/0x79

Thanks in advance
David

Signature · 7 lines
View this message in context: http://old.nabble.com/possible-xfs-corruption-tp30395558p30395558.html
Sent from the Xfs - General mailing list archive at Nabble.com.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#1 lore raw

Le Tue, 7 Dec 2010 03:49:49 -0800 (PST)
blacknred <[email protected]> écrivait:

Quoted 4 lines

I'm stuck with a storage issue on reboot. Initially doubted the storage, but dmesg throws these errors. Now wondering whether this is a fs issue? Any thoughts as to whats going on here?

It looks like a part of the filesystem is physically missing. What is the underlying device? Apparently it's a SmartArray in some HP server, did you had a power failure?

Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#2 lore raw

Yes, It's a Smart Array in HP Proliant Server.
No power failure, just upgraded the controller firmware and rebooted the server.....

Quoted 27 lines · Emmanuel Florac

Le Tue, 7 Dec 2010 03:49:49 -0800 (PST)
blacknred <[email protected]> écrivait:

Quoted 4 lines

I'm stuck with a storage issue on reboot. Initially doubted the storage, but dmesg throws these errors. Now wondering whether this is a fs issue? Any thoughts as to whats going on here?

It looks like a part of the filesystem is physically missing. What is the underlying device? Apparently it's a SmartArray in some HP server, did you had a power failure?

Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Signature · 7 lines
View this message in context: http://old.nabble.com/possible-xfs-corruption-tp30395558p30397173.html
Sent from the Xfs - General mailing list archive at Nabble.com.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw

Le Tue, 7 Dec 2010 07:11:04 -0800 (PST)
blacknred <[email protected]> écrivait:

Quoted 4 lines

Yes, It's a Smart Array in HP Proliant Server.
No power failure, just upgraded the controller firmware and rebooted the server.....

Hu oh, that stinks. Nothing in the firmware release notes? Do you have any other filesystem on this array?

First you could try to backup the filesystem metadata using

xfs_metadump -o -w /dev/DEVICE outfile.meta

It can be useful later in case things turn bad. Does it output any errors? IO errors particularly ?

Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#4 lore raw

It's all xfs..... and didn't see any I/O errors as well...except that dmesg is flooded with similar traces to one i posted....

Quoted 33 lines · Emmanuel Florac

Le Tue, 7 Dec 2010 07:11:04 -0800 (PST)
blacknred <[email protected]> écrivait:

Quoted 4 lines

Yes, It's a Smart Array in HP Proliant Server.
No power failure, just upgraded the controller firmware and rebooted the server.....

Hu oh, that stinks. Nothing in the firmware release notes? Do you have any other filesystem on this array?

First you could try to backup the filesystem metadata using

xfs_metadump -o -w /dev/DEVICE outfile.meta

It can be useful later in case things turn bad. Does it output any errors? IO errors particularly ?

Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Signature · 7 lines
View this message in context: http://old.nabble.com/possible-xfs-corruption-tp30395558p30398538.html
Sent from the Xfs - General mailing list archive at Nabble.com.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw

blacknred put forth on 12/7/2010 11:30 AM:

Quoted 3 lines

It's all xfs..... and didn't see any I/O errors as well...except that dmesg is flooded with similar traces to one i posted....

Out of the blue, one OP, with two Proliant servers having storage problems, within hours of one another, the same day?

Long odds, that.

Did you walk a mile, flat footed, on deep shag carpet, while wearing a wool sweater, in a room with zero humidity, then walk into the DC and touch each of these servers, or something? ;)

Signature · 6 lines
Stan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#6 lore raw
In reply to blacknred
Quoted 3 lines · On Dienstag, 7. Dezember 2010 blacknred wrote:

Yes, It's a Smart Array in HP Proliant Server.
No power failure, just upgraded the controller firmware and rebooted the server.....

What Server, controller, and from which firmware version did you go to which one?

Signature · 11 lines
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

// ****** Radiointerview zum Thema Spam ******
// http://www.it-podcast.at/archiv.html#podcast-100716
// 
// Haus zu verkaufen: http://zmi.at/langegg/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#7 lore raw
In reply to blacknred
Quoted 8 lines · blacknred

Hi...

I'm stuck with a storage issue on reboot. Initially doubted the storage, but dmesg throws these errors. Now wondering whether this is a fs issue? Any thoughts as to whats going on here?

Quoted 6 lines

XFS: failed to locate log tail
XFS: log mount/recovery failed: error 117
XFS: log mount failed
XFS mounting filesystem cciss/c0d0
Filesystem "cciss/c0d0": XFS internal error xlog_clear_stale_blocks(2) at line 1237 of file

Which indicates that the head and/or the tail of the log are not valid. Can you provide the output of:

# xfs_logprint -d /dev/cciss/c0d0

So we can see what the head/tail values are in the log?

Quoted 1 line

/home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_log_recover.c.

CentOS kernel? How old?

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#8 lore raw
Quoted 19 lines · Dave Chinner
Quoted 8 lines · blacknred

Hi...

I'm stuck with a storage issue on reboot. Initially doubted the storage, but dmesg throws these errors. Now wondering whether this is a fs issue? Any thoughts as to whats going on here?

Quoted 6 lines

XFS: failed to locate log tail
XFS: log mount/recovery failed: error 117
XFS: log mount failed
XFS mounting filesystem cciss/c0d0
Filesystem "cciss/c0d0": XFS internal error xlog_clear_stale_blocks(2) at line 1237 of file

Which indicates that the head and/or the tail of the log are not valid. Can you provide the output of:

Quoted 7 lines
# xfs_logprint -d /dev/cciss/c0d0

So we can see what the head/tail values are in the log?

Quoted 1 line

/home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_log_recover.c.

CentOS kernel? How old?

Assuning it's centos5, there's really no need to be using an xfs kmod there anymore, the module shipped with the kernel in recent versions of the OS is really the one you want to use. That kmod is ancient and unmaintained.

Although I suspect the storage is more likely at fault here. :)

-Eric

Quoted 3 lines

Cheers,

Dave.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs