xfs problems (possibly after upgrading from linux kernel 2.6.27.10 to .14)
Hi all,
within the past few days we hit many XFS internal errors like these. Are these errors known (and possibly already fixed)? I checked the commits till 2.6.27.17 and there does not seem anything related to this.
Do you need more information or can I send these nodes into a re-install?
Cheers
Carsten
PS: Please CC me, as I'm currently not on this list.
Feb 16 20:34:49 n0035 kernel: [275873.335916] Filesystem "sda6": XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs_ trans.c. Caller 0xffffffff8031b704 Feb 16 20:34:49 n0035 kernel: [275873.336262] Pid: 20060, comm: lalapps_Makefak Not tainted 2.6.27.14-nodes #1 Feb 16 20:34:49 n0035 kernel: [275873.336452] Feb 16 20:34:49 n0035 kernel: [275873.336452] Call Trace: Feb 16 20:34:49 n0035 kernel: [275873.336814] [<ffffffff8031b704>] xfs_create+0x3ee/0x47b Feb 16 20:34:49 n0035 kernel: [275873.337003] [<ffffffff803167c6>] xfs_trans_cancel+0x55/0xed Feb 16 20:34:49 n0035 kernel: [275873.337191] [<ffffffff8031b704>] xfs_create+0x3ee/0x47b Feb 16 20:34:49 n0035 kernel: [275873.341237] [<ffffffff802dfc91>] xfs_attr_get+0x90/0xa1 Feb 16 20:34:49 n0035 kernel: [275873.341424] [<ffffffff803249e4>] xfs_vn_mknod+0x14b/0x214 Feb 16 20:34:49 n0035 kernel: [275873.341612] [<ffffffff8027c6b1>] vfs_create+0x71/0xb9 Feb 16 20:34:49 n0035 kernel: [275873.341798] [<ffffffff8027ec64>] do_filp_open+0x1e6/0x733 Feb 16 20:34:49 n0035 kernel: [275873.341990] [<ffffffff802738c7>] do_sys_open+0x48/0xcc Feb 16 20:34:49 n0035 kernel: [275873.342177] [<ffffffff8020bddb>] system_call_fastpath+0x16/0x1b Feb 16 20:34:49 n0035 kernel: [275873.342366] Feb 16 20:34:49 n0035 kernel: [275873.342548] xfs_force_shutdown(sda6,0x8) called from line 1165 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff803167df Feb 16 20:34:49 n0035 kernel: [275873.343158] xfs_force_shutdown(sda6,0x2) called from line 818 of file fs/xfs/xfs_log.c. Return add ress = 0xffffffff8030d17b Feb 16 20:34:49 n0035 kernel: [275873.343508] Filesystem "sda6": Corruption of in-memory data detected. Shutting down filesystem: sd a6 Feb 16 20:34:49 n0035 kernel: [275873.343909] Please umount the filesystem, and rectify the problem(s) Feb 16 20:34:53 n0035 kernel: [275877.000013] Filesystem "sda6": xfs_log_force: error 5 returned.
***********************************8
Feb 16 22:01:28 n0260 kernel: [1129250.851451] Filesystem "sda6": xfs_iflush: Bad inode 1176564060 magic number 0x36b5, ptr 0xffff880 1a7c06c00 Feb 16 22:01:28 n0260 kernel: [1129250.852132] xfs_force_shutdown(sda6,0x8) called from line 3215 of file fs/xfs/xfs_inode.c. Return address = 0xffffffff803077b2 Feb 16 22:01:28 n0260 kernel: [1129250.871943] xfs_force_shutdown(sda6,0x2) called from line 818 of file fs/xfs/xfs_log.c. Return ad dress = 0xffffffff8030d17b Feb 16 22:01:28 n0260 kernel: [1129250.872270] xfs_force_shutdown(sda6,0x2) called from line 818 of file fs/xfs/xfs_log.c. Return ad dress = 0xffffffff8030d17b Feb 16 22:01:28 n0260 kernel: [1129250.872632] Filesystem "sda6": Corruption of in-memory data detected. Shutting down filesystem: s da6 Feb 16 22:01:28 n0260 kernel: [1129250.872955] Filesystem "sda6": xfs_inactive: xfs_trans_commit() returned error 5 Feb 16 22:01:28 n0260 kernel: [1129250.873317] Please umount the filesystem, and rectify the problem(s) Feb 16 22:01:28 n0260 kernel: [1129250.942070] Filesystem "sda6": xfs_log_force: error 5 returned.
*****************************************
Feb 17 07:48:08 n0393 kernel: [1163760.030411] Filesystem "sda6": xfs_iflush: Bad inode 2117711358 magic number 0x8334, ptr 0xffff880 06883de00 Feb 17 07:48:08 n0393 kernel: [1163760.030750] xfs_force_shutdown(sda6,0x8) called from line 3215 of file fs/xfs/xfs_inode.c. Return address = 0xffffffff803077b2 Feb 17 07:48:08 n0393 kernel: [1163760.092840] xfs_force_shutdown(sda6,0x2) called from line 818 of file fs/xfs/xfs_log.c. Return ad dress = 0xffffffff8030d17b Feb 17 07:48:08 n0393 kernel: [1163760.093211] Filesystem "sda6": Corruption of in-memory data detected. Shutting down filesystem: s da6 Feb 17 07:48:08 n0393 kernel: [1163760.093645] Please umount the filesystem, and rectify the problem(s) Feb 17 07:48:09 n0393 kernel: [1163761.100176] Filesystem "sda6": xfs_log_force: error 5 returned.
****************************************
Feb 17 05:57:44 n0463 kernel: [1156816.912129] Filesystem "sda6": XFS internal error xfs_btree_check_sblock at line 307 of file fs/xf s/xfs_btree.c. Caller 0xffffffff802dd15b Feb 17 05:57:44 n0463 kernel: [1156816.912566] Pid: 31011, comm: rm Not tainted 2.6.27.14-nodes #1 Feb 17 05:57:44 n0463 kernel: [1156816.912774] Feb 17 05:57:44 n0463 kernel: [1156816.912775] Call Trace: Feb 17 05:57:44 n0463 kernel: [1156816.913177] [<ffffffff802dd15b>] xfs_alloc_lookup+0x135/0x36a Feb 17 05:57:44 n0463 kernel: [1156816.913385] [<ffffffff802f15da>] xfs_btree_check_sblock+0xaf/0xbf Feb 17 05:57:44 n0463 kernel: [1156816.913593] [<ffffffff802dd15b>] xfs_alloc_lookup+0x135/0x36a Feb 17 05:57:44 n0463 kernel: [1156816.913802] [<ffffffff802f110a>] xfs_btree_init_cursor+0x31/0x1a5 Feb 17 05:57:44 n0463 kernel: [1156816.914011] [<ffffffff802dadeb>] xfs_free_ag_extent+0x351/0x6bb Feb 17 05:57:44 n0463 kernel: [1156816.914218] [<ffffffff802dc7df>] xfs_free_extent+0xa9/0xc9 Feb 17 05:57:44 n0463 kernel: [1156816.914426] [<ffffffff802e52de>] xfs_bmap_finish+0xee/0x15f Feb 17 05:57:44 n0463 kernel: [1156816.914633] [<ffffffff80305505>] xfs_itruncate_finish+0x190/0x2ba Feb 17 05:57:44 n0463 kernel: [1156816.914844] [<ffffffff8031d49b>] xfs_inactive+0x1de/0x40f Feb 17 05:57:44 n0463 kernel: [1156816.915050] [<ffffffff80327957>] xfs_fs_clear_inode+0xb0/0xf3 Feb 17 05:57:44 n0463 kernel: [1156816.915257] [<ffffffff8028654a>] clear_inode+0x6d/0xc4 Feb 17 05:57:44 n0463 kernel: [1156816.915461] [<ffffffff8028662a>] generic_delete_inode+0x89/0xe3 Feb 17 05:57:44 n0463 kernel: [1156816.915670] [<ffffffff8027e278>] do_unlinkat+0xda/0x157 Feb 17 05:57:44 n0463 kernel: [1156816.915878] [<ffffffff8034a017>] __up_read+0x13/0x8a Feb 17 05:57:44 n0463 kernel: [1156816.916086] [<ffffffff804a0c39>] error_exit+0x0/0x51 Feb 17 05:57:44 n0463 kernel: [1156816.916294] [<ffffffff8020bddb>] system_call_fastpath+0x16/0x1b Feb 17 05:57:44 n0463 kernel: [1156816.916500] Feb 17 05:57:44 n0463 kernel: [1156816.916700] xfs_force_shutdown(sda6,0x8) called from line 4269 of file fs/xfs/xfs_bmap.c. Return address = 0xffffffff802e5313 Feb 17 05:57:44 n0463 kernel: [1156816.920788] xfs_force_shutdown(sda6,0x2) called from line 818 of file fs/xfs/xfs_log.c. Return ad dress = 0xffffffff8030d17b Feb 17 05:57:44 n0463 kernel: [1156816.921144] xfs_force_shutdown(sda6,0x2) called from line 818 of file fs/xfs/xfs_log.c. Return ad dress = 0xffffffff8030d17b Feb 17 05:57:44 n0463 kernel: [1156816.921564] Filesystem "sda6": Corruption of in-memory data detected. Shutting down filesystem: s da6 Feb 17 05:57:44 n0463 kernel: [1156816.921937] Please umount the filesystem, and rectify the problem(s) Feb 17 05:57:48 n0463 kernel: [1156821.731882] Filesystem "sda6": xfs_log_force: error 5 returned.
*************************************************************************8
Feb 17 07:49:47 n0770 kernel: [1162181.596030] Filesystem "sda6": XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs _trans.c. Caller 0xffffffff8031b704 Feb 17 07:49:47 n0770 kernel: [1162181.596379] Pid: 15905, comm: lalapps_Makefak Not tainted 2.6.27.14-nodes #1 Feb 17 07:49:47 n0770 kernel: [1162181.596717] Feb 17 07:49:47 n0770 kernel: [1162181.596717] Call Trace: Feb 17 07:49:47 n0770 kernel: [1162181.597080] [<ffffffff8031b704>] xfs_create+0x3ee/0x47b Feb 17 07:49:47 n0770 kernel: [1162181.597275] [<ffffffff803167c6>] xfs_trans_cancel+0x55/0xed Feb 17 07:49:47 n0770 kernel: [1162181.597477] [<ffffffff8031b704>] xfs_create+0x3ee/0x47b Feb 17 07:49:47 n0770 kernel: [1162181.597675] [<ffffffff802dfc91>] xfs_attr_get+0x90/0xa1 Feb 17 07:49:47 n0770 kernel: [1162181.597876] [<ffffffff803249e4>] xfs_vn_mknod+0x14b/0x214 Feb 17 07:49:47 n0770 kernel: [1162181.598079] [<ffffffff8027c6b1>] vfs_create+0x71/0xb9 Feb 17 07:49:47 n0770 kernel: [1162181.598278] [<ffffffff8027ec64>] do_filp_open+0x1e6/0x733 Feb 17 07:49:47 n0770 kernel: [1162181.598477] [<ffffffff802738c7>] do_sys_open+0x48/0xcc Feb 17 07:49:47 n0770 kernel: [1162181.598672] [<ffffffff8020bddb>] system_call_fastpath+0x16/0x1b Feb 17 07:49:47 n0770 kernel: [1162181.598871] Feb 17 07:49:47 n0770 kernel: [1162181.599061] xfs_force_shutdown(sda6,0x8) called from line 1165 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff803167df Feb 17 07:49:47 n0770 kernel: [1162181.605374] xfs_force_shutdown(sda6,0x2) called from line 818 of file fs/xfs/xfs_log.c. Return ad dress = 0xffffffff8030d17b Feb 17 07:49:47 n0770 kernel: [1162181.605775] Filesystem "sda6": Corruption of in-memory data detected. Shutting down filesystem: s da6 Feb 17 07:49:47 n0770 kernel: [1162181.606227] Please umount the filesystem, and rectify the problem(s) Feb 17 07:49:49 n0770 kernel: [1162182.911372] Filesystem "sda6": xfs_log_force: error 5 returned.
***************************************************************************
Feb 17 00:47:33 n0986 kernel: [1136074.667418] Filesystem "sda6": xfs_iflush: Bad inode 1223835676 magic number 0xe5e5, ptr 0xffff880 1763f6c00 Feb 17 00:47:33 n0986 kernel: [1136074.667833] xfs_force_shutdown(sda6,0x8) called from line 3215 of file fs/xfs/xfs_inode.c. Return address = 0xffffffff803077b2 Feb 17 00:47:33 n0986 kernel: [1136074.678958] xfs_force_shutdown(sda6,0x2) called from line 818 of file fs/xfs/xfs_log.c. Return ad dress = 0xffffffff8030d17b Feb 17 00:47:33 n0986 kernel: [1136074.679430] Filesystem "sda6": Log I/O Error Detected. Shutting down filesystem: sda6 Feb 17 00:47:33 n0986 kernel: [1136074.679783] Please umount the filesystem, and rectify the problem(s) Feb 17 00:47:33 n0986 kernel: [1136075.686618] Filesystem "sda6": xfs_log_force: error 5 returned.
*******************************************************************************
Feb 17 01:18:08 n1003 kernel: [1137811.535542] Filesystem "sda6": XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs _trans.c. Caller 0xffffffff8031b704 Feb 17 01:18:08 n1003 kernel: [1137811.535925] Pid: 28435, comm: lalapps_Makefak Not tainted 2.6.27.14-nodes #1 Feb 17 01:18:08 n1003 kernel: [1137811.536287] Feb 17 01:18:08 n1003 kernel: [1137811.536288] Call Trace: Feb 17 01:18:08 n1003 kernel: [1137811.536684] [<ffffffff8031b704>] xfs_create+0x3ee/0x47b Feb 17 01:18:08 n1003 kernel: [1137811.536890] [<ffffffff803167c6>] xfs_trans_cancel+0x55/0xed Feb 17 01:18:08 n1003 kernel: [1137811.537096] [<ffffffff8031b704>] xfs_create+0x3ee/0x47b Feb 17 01:18:08 n1003 kernel: [1137811.537301] [<ffffffff802dfc91>] xfs_attr_get+0x90/0xa1 Feb 17 01:18:08 n1003 kernel: [1137811.537506] [<ffffffff803249e4>] xfs_vn_mknod+0x14b/0x214 Feb 17 01:18:08 n1003 kernel: [1137811.537711] [<ffffffff8027c6b1>] vfs_create+0x71/0xb9 Feb 17 01:18:08 n1003 kernel: [1137811.537917] [<ffffffff8027ec64>] do_filp_open+0x1e6/0x733 Feb 17 01:18:08 n1003 kernel: [1137811.538143] [<ffffffff802738c7>] do_sys_open+0x48/0xcc Feb 17 01:18:08 n1003 kernel: [1137811.538361] [<ffffffff8020bddb>] system_call_fastpath+0x16/0x1b Feb 17 01:18:08 n1003 kernel: [1137811.538566] Feb 17 01:18:08 n1003 kernel: [1137811.538764] xfs_force_shutdown(sda6,0x8) called from line 1165 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff803167df Feb 17 01:18:08 n1003 kernel: [1137811.622631] Filesystem "sda6": Corruption of in-memory data detected. Shutting down filesystem: s da6 Feb 17 01:18:08 n1003 kernel: [1137811.622996] Please umount the filesystem, and rectify the problem(s) Feb 17 01:18:12 n1003 kernel: [1137815.196767] Filesystem "sda6": xfs_log_force: error 5 returned.
******************************8
plus a few more nodes showing the same characteristics
Quoted 1 line · Carsten Aulbert
Feb 16 20:34:49 n0035 kernel: [275873.343508] Filesystem "sda6":
Corruption of
Quoted 1 line
in-memory data detected. Shutting down filesystem: sda6
Are you absolutely sure your RAM doesn't have problems? Run memtest86. At least I would from that error description.
mfg zmi
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Hi Michael,
Quoted 7 lines · Michael Monnerie
Quoted 1 line · Carsten Aulbert
Feb 16 20:34:49 n0035 kernel: [275873.343508] Filesystem "sda6":Corruption of
Quoted 1 line
in-memory data detected. Shutting down filesystem: sda6
Are you absolutely sure your RAM doesn't have problems? Run memtest86. At least I would from that error description.
Pretty sure, all boxes have ECC memory and mcelog has not reported anything on the nodes in question right now - we usually have only a few spurious MCE errors a week. We have encountered 17 of these incidences within the past week and basically nothing before - I don't think that memory in 17 nodes goes bad just by upgrading the kernel or out of coincidence.
I'll pick 4-5 nodes for memtests just to be sure.
Cheers
Carsten
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines · Carsten Aulbert
Hi all,
within the past few days we hit many XFS internal errors like these. Are these errors known (and possibly already fixed)? I checked the commits till 2.6.27.17 and there does not seem anything related to this.
Quoted 7 lines
Do you need more information or can I send these nodes into a re-install?
Cheers
Carsten
PS: Please CC me, as I'm currently not on this list.
It'd be worth running xfs_repair on one of these nodes, I think, to see if you're encountering on-disk corruption, which is what this looks like.
Anything funky about your storage? Any IO/storage issues before this? Does going back to .10 make it go away?
-Eric
Quoted 5 lines · Carsten Aulbert
Hi all,
within the past few days we hit many XFS internal errors like these. Are these errors known (and possibly already fixed)? I checked the commits till 2.6.27.17 and there does not seem anything related to this.
.....
Quoted 1 line
Feb 16 20:34:49 n0035 kernel: [275873.335916] Filesystem "sda6": XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs_
A transaction shutdown on create. That implies some kind of ENOSPC issue.
Quoted 1 line
Do you need more information or can I send these nodes into a re-install?
More information. Can you get a machine into a state where you can trigger this condition reproducably by doing:
mount filesystem
touch /mnt/filesystem/some_new_file
If you can get it to that state, and you can provide an xfs_metadump image of the filesystem when in that state, I can track down the problem and fix it.
Quoted 1 line
Feb 16 22:01:28 n0260 kernel: [1129250.851451] Filesystem "sda6": xfs_iflush: Bad inode 1176564060 magic number 0x36b5, ptr 0xffff8801a7c06c00
However, this implies some kind of memory corruption is occurring. That is reading the inode out of the buffer before flushing the in-memory state to disk. This implies someone has scribbled over page cache pages.
Quoted 1 line
Feb 17 05:57:44 n0463 kernel: [1156816.912129] Filesystem "sda6": XFS internal error xfs_btree_check_sblock at line 307 of file fs/xfs/xfs_btree.c. Caller 0xffffffff802dd15b
And that is another buffer that has been scribbled over. Something is corrupting the page cache, I think. Whether the original shutdown is caused by the some corruption, i don't know.
Quoted 1 line
plus a few more nodes showing the same characteristics
Hmmmm. Did this show up in 2.6.27.10? Or did it start occurring only after you upgraded from .10 to .14?
Cheers,
Dave.
Quoted 6 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 2 lines
Dave Chinner [email protected]
Hi Dave,
Quoted 19 lines · Dave Chinner
Quoted 5 lines · Carsten Aulbert
Hi all,
within the past few days we hit many XFS internal errors like these. Are these errors known (and possibly already fixed)? I checked the commits till 2.6.27.17 and there does not seem anything related to this.
.....
Quoted 1 line
Feb 16 20:34:49 n0035 kernel: [275873.335916] Filesystem "sda6": XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs_A transaction shutdown on create. That implies some kind of ENOSPC issue.
Quoted 1 line
Do you need more information or can I send these nodes into a re-install?
More information. Can you get a machine into a state where you can trigger this condition reproducably by doing:
Quoted 6 lines
mount filesystem
touch/mnt/filesystem/some_new_fileIf you can get it to that state, and you can provide an
xfs_metadumpimage of the filesystem when in that state, I can track down the problem and fix it.
I can try doing that on a few machines, would a metadump help on a machine where this corruption occurred some time ago and is still in this state?
Quoted 9 lines
Quoted 2 lines
Feb 16 22:01:28 n0260 kernel: [1129250.851451] Filesystem "sda6": xfs_iflush: Bad inode 1176564060 magic number 0x36b5, ptr 0xffff8801a7c06c00However, this implies some kind of memory corruption is occurring. That is reading the inode out of the buffer before flushing the in-memory state to disk. This implies someone has scribbled over page cache pages.
Quoted 7 lines
Quoted 1 line
Feb 17 05:57:44 n0463 kernel: [1156816.912129] Filesystem "sda6": XFS internal error xfs_btree_check_sblock at line 307 of file fs/xfs/xfs_btree.c. Caller 0xffffffff802dd15bAnd that is another buffer that has been scribbled over. Something is corrupting the page cache, I think. Whether the original shutdown is caused by the some corruption, i don't know.
At least on two nodes we ran memtest86+ overnight and so far no error.
Quoted 4 lines
Quoted 1 line
plus a few more nodes showing the same characteristics
Hmmmm. Did this show up in 2.6.27.10? Or did it start occurring only after you upgraded from .10 to .14?
As far as I can see this only happened after the upgrade about 14 days ago. What strikes me odd is that we only had this occurring massively on Monday and Tuesday this week.
I don't know if a certain access pattern could trigger this somehow.
Cheers
Carsten
Quoted 10 lines · Carsten Aulbert
Quoted 9 lines · Dave Chinner
Quoted 1 line · Carsten Aulbert
Do you need more information or can I send these nodes into a re-install?
More information. Can you get a machine into a state where you can trigger this condition reproducably by doing:
mount filesystem
touch/mnt/filesystem/some_new_file
Quoted 7 lines
Quoted 3 lines
If you can get it to that state, and you can provide an
xfs_metadumpimage of the filesystem when in that state, I can track down the problem and fix it.I can try doing that on a few machines, would a metadump help on a machine where this corruption occurred some time ago and is still in this state?
If you unmount the filesystem, mount it again and then touch a new file and it reports the error again, then yes, a metadump woul dbe great.
If the error doesn't show up after a unmount/mount, then I can't use a metadump image to reproduce the problem.
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
Feb 16 22:01:28 n0260 kernel: [1129250.851451] Filesystem "sda6": xfs_iflush: Bad inode 1176564060 magic number 0x36b5, ptr 0xffff8801a7c06c00However, this implies some kind of memory corruption is occurring. That is reading the inode out of the buffer before flushing the in-memory state to disk. This implies someone has scribbled over page cache pages.
Quoted 1 line
Feb 17 05:57:44 n0463 kernel: [1156816.912129] Filesystem "sda6": XFS internal error xfs_btree_check_sblock at line 307 of file fs/xfs/xfs_btree.c. Caller 0xffffffff802dd15b
Quoted 7 lines
Quoted 5 lines
And that is another buffer that has been scribbled over. Something is corrupting the page cache, I think. Whether the original shutdown is caused by the some corruption, i don't know.
At least on two nodes we ran memtest86+ overnight and so far no error.
I don't think it is hardware related.
Quoted 10 lines
Quoted 4 lines
Quoted 1 line
plus a few more nodes showing the same characteristics
Hmmmm. Did this show up in 2.6.27.10? Or did it start occurring only after you upgraded from .10 to .14?
As far as I can see this only happened after the upgrade about 14 days ago. What strikes me odd is that we only had this occurring massively on Monday and Tuesday this week.
I don't know if a certain access pattern could trigger this somehow.
I suspect so. We've already had XFS trigger one bug in the new lockless pagecache code, and the fix for that went in 2.6.27.11 - between the good version and the version that you've been seeing these memory corruptions on. I'm wondering if that fix exposed or introduced another bug that you've hit....
Nick?
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Hi again,
Quoted 11 lines · Dave Chinner
Quoted 3 lines
I can try doing that on a few machines, would a metadump help on a machine where this corruption occurred some time ago and is still in this state?
If you unmount the filesystem, mount it again and then touch a new file and it reports the error again, then yes, a metadump woul dbe great.
If the error doesn't show up after a unmount/mount, then I can't use a metadump image to reproduce the problem.
I've done it on two nodes so far and the result is not good (metadump wise):
[1344887.778232] Filesystem "sda6": xfs_log_force: error 5 returned. [1344887.778432] xfs_force_shutdown(sda6,0x1) called from line 420 of file fs/xfs/xfs_rw.c. Return address = 0xffffffff8031dd7e [1344889.579836] Filesystem "sda6": xfs_log_force: error 5 returned. [1344889.580044] Filesystem "sda6": xfs_log_force: error 5 returned. [1344889.580257] Filesystem "sda6": xfs_log_force: error 5 returned. [1344889.580450] Filesystem "sda6": xfs_log_force: error 5 returned. [1344889.624774] Filesystem "sda6": xfs_log_force: error 5 returned. [1344915.783844] XFS mounting filesystem sda6 [1344915.872333] Starting XFS recovery on filesystem: sda6 (logdev: internal) [1344917.399834] Ending XFS recovery on filesystem: sda6 (logdev: internal)
After that I can touch/create all files I want on the fs again.
Quoted 7 lines
I suspect so. We've already had XFS trigger one bug in the new lockless pagecache code, and the fix for that went in 2.6.27.11 - between the good version and the version that you've been seeing these memory corruptions on. I'm wondering if that fix exposed or introduced another bug that you've hit....
Nick?
If it was triggered by a user job, it might have been in the kernel for longer and the user just did not run it for a few weeks.
I'll try to gather more information.
Cheers
Carsten
Quoted 10 lines · Dave Chinner
Quoted 9 lines · Carsten Aulbert
Quoted 4 lines
Quoted 1 line
plus a few more nodes showing the same characteristics
Hmmmm. Did this show up in 2.6.27.10? Or did it start occurring only after you upgraded from .10 to .14?
As far as I can see this only happened after the upgrade about 14 days ago. What strikes me odd is that we only had this occurring massively on Monday and Tuesday this week.
Quoted 7 lines
Quoted 1 line
I don't know if a certain access pattern could trigger this somehow.
I suspect so. We've already had XFS trigger one bug in the new lockless pagecache code, and the fix for that went in 2.6.27.11 - between the good version and the version that you've been seeing these memory corruptions on. I'm wondering if that fix exposed or introduced another bug that you've hit....
Highly unlikely. It only introduces constraints on how the compiler may generate code, so it would have to be a compiler bug to cause a bug I think.
I wonder how long you've been running with 2.6.27 based kernels without corruption?
Hi Nick,
Quoted 2 lines · Nick Piggin
I wonder how long you've been running with 2.6.27 based kernels without corruption?
Let me see, we upgraded from 2.6.25.9 to 2.6.27.10 in mid January, i.e. our data base time line is about 6 weeks.
The upgrade to .14 happened about 16 days ago.
HTH
Carsten