XFS filesystem shutting down on linux 2.6.28.9 (xfs_rename)

DISCUSSION Gabriel Barazer 2 participants 5 messages Jul 22, 2009 – Jul 28, 2009

Share & tools

Gabriel Barazer

author

2009-07-22 15:26 UTC #0 lore raw

Hi,

I recently put a NFS file server into production, with mostly XFS volumes on LVM. The server was quite low on traffic until this morning and one of the filesystems crashed twice since this morning with the following backtrace:

Filesystem "dm-24": XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs_trans.c. Caller 0xffffffff811b09a7
Pid: 2053, comm: nfsd Not tainted 2.6.28.9-filer #1

Call Trace:
 [<ffffffff811b09a7>] xfs_rename+0x4a1/0x4f6
 [<ffffffff811b1806>] xfs_trans_cancel+0x56/0xed
 [<ffffffff811b09a7>] xfs_rename+0x4a1/0x4f6
 [<ffffffff811bfad1>] xfs_vn_rename+0x5e/0x65
 [<ffffffff8108a1de>] vfs_rename+0x1fb/0x2fb
 [<ffffffff8113acc2>] nfsd_rename+0x299/0x349
 [<ffffffff813e4eb1>] sunrpc_cache_lookup+0x4a/0x109
 [<ffffffff811416a9>] nfsd3_proc_rename+0xdb/0xea
 [<ffffffff811436ab>] decode_filename+0x16/0x45
 [<ffffffff81136eb9>] nfsd_dispatch+0xdf/0x1b5
 [<ffffffff813dd6f0>] svc_process+0x3f7/0x610
 [<ffffffff81137444>] nfsd+0x12e/0x185
 [<ffffffff81137316>] nfsd+0x0/0x185
 [<ffffffff810442e7>] kthread+0x47/0x71
 [<ffffffff8102e622>] schedule_tail+0x24/0x5c
 [<ffffffff8100cdb9>] child_rip+0xa/0x11
 [<ffffffff81011e0c>] read_tsc+0x0/0x19
 [<ffffffff810442a0>] kthread+0x0/0x71	
 [<ffffffff8100cdaf>] child_rip+0x0/0x11

xfs_force_shutdown(dm-24,0x8) called from line 1165 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff811b181f Filesystem "dm-24": Corruption of in-memory data detected. Shutting down filesystem: dm-24

The two crashed are related to the same function: xfs_rename.

I _really_ cannot upgrade to 2.6.29 or later because of the "reconnect_path: npd != pd" bug and the maybe related radix-tree bug ( http://bugzilla.kernel.org/show_bug.cgi?id=13375 ) affecting all kernel version afeter 2.6.28.

Unmounting then remounting the filesystem allow to access the mountpoint again without any error message or apparent file corruption. This filesystem is used by ~30 NFS clients and contains about 5M files (100GB).

Before using the volume over NFS, there was only local activity (rsync syncing) and we didn't get any error.

I expect to see this crash again in a few hours except if the volume is really corrupted. Does a full filesystem copy to a newly created volume would have a chance to solve the problem?

Thanks,

Gabriel

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Eric Sandeen

2009-07-23 04:10 UTC #1 lore raw

Quoted 10 lines · Gabriel Barazer

Hi,
I recently put a NFS file server into production, with mostly XFS volumes on LVM. The server was quite low on traffic until this morning and one of the filesystems crashed twice since this morning with the following backtrace:
Filesystem "dm-24": XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs_trans.c. Caller 0xffffffff811b09a7
Pid: 2053, comm: nfsd Not tainted 2.6.28.9-filer #1
Call Trace:
 [<ffffffff811b09a7>] xfs_rename+0x4a1/0x4f6
 [<ffffffff811b1806>] xfs_trans_cancel+0x56/0xed
 [<ffffffff811b09a7>] xfs_rename+0x4a1/0x4f6

...

Quoted 4 lines

xfs_force_shutdown(dm-24,0x8) called from line 1165 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff811b181f Filesystem "dm-24": Corruption of in-memory data detected. Shutting down filesystem: dm-24
The two crashed are related to the same function: xfs_rename.

Can you do objdump -d xfs.ko | grep "xfs_rename\|xfs_trans_cancel" and maybe we can see which call to xfs_trans_cancel in xfs_rename this was.

The problem relates to canceling a dirty transaction on an error path.

-Eric

Quoted 9 lines

I _really_ cannot upgrade to 2.6.29 or later because of the "reconnect_path: npd != pd" bug and the maybe related radix-tree bug ( http://bugzilla.kernel.org/show_bug.cgi?id=13375 ) affecting all kernel version afeter 2.6.28.
Unmounting then remounting the filesystem allow to access the mountpoint again without any error message or apparent file corruption. This filesystem is used by ~30 NFS clients and contains about 5M files (100GB).
Before using the volume over NFS, there was only local activity (rsync syncing) and we didn't get any error.
I expect to see this crash again in a few hours except if the volume is really corrupted. Does a full filesystem copy to a newly created volume would have a chance to solve the problem?

Quoted 9 lines

Thanks,
Gabriel
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Gabriel Barazer

author

2009-07-27 11:40 UTC #2 lore raw

Quoted 19 lines · Eric Sandeen

Quoted 11 lines · Gabriel Barazer
Hi,
I recently put a NFS file server into production, with mostly XFS volumes on LVM. The server was quite low on traffic until this morning and one of the filesystems crashed twice since this morning with the following backtrace:
Filesystem "dm-24": XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs_trans.c. Caller 0xffffffff811b09a7
Pid: 2053, comm: nfsd Not tainted 2.6.28.9-filer #1
Call Trace:
 [<ffffffff811b09a7>] xfs_rename+0x4a1/0x4f6
 [<ffffffff811b1806>] xfs_trans_cancel+0x56/0xed
 [<ffffffff811b09a7>] xfs_rename+0x4a1/0x4f6
...
Quoted 3 lines
xfs_force_shutdown(dm-24,0x8) called from line 1165 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff811b181f Filesystem "dm-24": Corruption of in-memory data detected. Shutting down filesystem: dm-24

Quoted 8 lines

Quoted 2 lines
The two crashed are related to the same function: xfs_rename.
Can you do objdump -d xfs.ko | grep "xfs_rename\|xfs_trans_cancel" and maybe we can see which call to xfs_trans_cancel in xfs_rename this was.
The problem relates to canceling a dirty transaction on an error path.

Hi,

sorry for the late reply

I don't have any xfs.ko as my kernel is compiled without CONFIG_MODULES. However I objdump'd the vmlinux uncompressed kernel, and here are the results:

ffffffff8116dcb8:       e8 f3 3a 04 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff8116f61b:       e8 90 21 04 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff8116f68f:       e8 1c 21 04 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff8116fbaa:       e8 01 1c 04 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff8116fbee:       e8 bd 1b 04 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff8117073c:       e8 6f 10 04 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff8117261b:       e8 90 f1 03 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff81174dde:       e8 cd c9 03 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff81175303:       e8 a8 c4 03 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff8117c08a:       e8 21 57 03 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff8117c146:       e8 65 56 03 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff8117cf06:       e8 a5 48 03 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff8117d000:       e8 ab 47 03 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff8117dd83:       e8 28 3a 03 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff8117dfa3:       e8 08 38 03 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811845fa:       e8 b1 d1 02 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff81184929:       e8 82 ce 02 00          callq  ffffffff811b17b0 <xfs_trans_cancel>

ffffffff81199b89:       e9 22 7c 01 00          jmpq   ffffffff811b17b0 <xfs_trans_cancel>
ffffffff8119aa30:       e8 7b 6d 01 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811a46d1:       e8 da d0 00 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811a4813:       e8 98 cf 00 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811a4929:       e8 82 ce 00 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811a4b8a:       e8 21 cc 00 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811a4e8b:       e8 20 c9 00 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811a509e:       e8 0d c7 00 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811a6bf7:       e8 b4 ab 00 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811a6c86:       e8 25 ab 00 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811aa18a:       e8 21 76 00 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811abe18:       e8 93 59 00 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811aeb5c:       e8 4f 2c 00 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811aecf9:       e8 b2 2a 00 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b04ca <xfs_rename_unlock4>:
ffffffff811b04e6:       74 19                   je     ffffffff811b0501 <xfs_rename_unlock4+0x37>
ffffffff811b04ed:       74 08                   je     ffffffff811b04f7 <xfs_rename_unlock4+0x2d>
ffffffff811b04ff:       75 dd                   jne    ffffffff811b04de <xfs_rename_unlock4+0x14>
ffffffff811b0506 <xfs_rename>:
ffffffff811b0563:       74 21                   je     ffffffff811b0586 <xfs_rename+0x80>
ffffffff811b0568:       75 1c                   jne    ffffffff811b0586 <xfs_rename+0x80>
ffffffff811b056f:       74 15                   je     ffffffff811b0586 <xfs_rename+0x80>
ffffffff811b0580:       0f 87 38 04 00 00       ja     ffffffff811b09be <xfs_rename+0x4b8>
ffffffff811b0628:       75 23                   jne    ffffffff811b064d <xfs_rename+0x147>
ffffffff811b064f:       74 04                   je     ffffffff811b0655 <xfs_rename+0x14f>
ffffffff811b0653:       eb 18                   jmp    ffffffff811b066d <xfs_rename+0x167>
ffffffff811b0666:       74 13                   je     ffffffff811b067b <xfs_rename+0x175>
ffffffff811b0676:       e9 27 03 00 00          jmpq   ffffffff811b09a2 <xfs_rename+0x49c>
ffffffff811b0695:       74 39                   je     ffffffff811b06d0 <xfs_rename+0x1ca>
ffffffff811b06a6:       74 28                   je     ffffffff811b06d0 <xfs_rename+0x1ca>
ffffffff811b06b2:       e8 13 fe ff ff          callq  ffffffff811b04ca <xfs_rename_unlock4>
ffffffff811b06c1:       e8 ea 10 00 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b06cb:       e9 ee 02 00 00          jmpq   ffffffff811b09be <xfs_rename+0x4b8>
ffffffff811b06ef:       74 1a                   je     ffffffff811b070b <xfs_rename+0x205>
ffffffff811b0729:       74 37                   je     ffffffff811b0762 <xfs_rename+0x25c>
ffffffff811b0757:       0f 85 ab 00 00 00       jne    ffffffff811b0808 <xfs_rename+0x302>
ffffffff811b075d:       e9 88 00 00 00          jmpq   ffffffff811b07ea <xfs_rename+0x2e4>
ffffffff811b0779:       0f 85 51 02 00 00       jne    ffffffff811b09d0 <xfs_rename+0x4ca>
ffffffff811b07a7:       0f 84 23 02 00 00       je     ffffffff811b09d0 <xfs_rename+0x4ca>
ffffffff811b07af:       0f 85 2e 02 00 00       jne    ffffffff811b09e3 <xfs_rename+0x4dd>
ffffffff811b07c7:       0f 84 a6 00 00 00       je     ffffffff811b0873 <xfs_rename+0x36d>
ffffffff811b07d2:       0f 84 9b 00 00 00       je     ffffffff811b0873 <xfs_rename+0x36d>
ffffffff811b07e5:       e9 81 00 00 00          jmpq   ffffffff811b086b <xfs_rename+0x365>
ffffffff811b07f4:       0f 84 dd 01 00 00       je     ffffffff811b09d7 <xfs_rename+0x4d1>
ffffffff811b0802:       0f 87 cf 01 00 00       ja     ffffffff811b09d7 <xfs_rename+0x4d1>
ffffffff811b082f:       0f 85 ae 01 00 00       jne    ffffffff811b09e3 <xfs_rename+0x4dd>
ffffffff811b0851:       0f 85 8c 01 00 00       jne    ffffffff811b09e3 <xfs_rename+0x4dd>
ffffffff811b085c:       74 15                   je     ffffffff811b0873 <xfs_rename+0x36d>
ffffffff811b086d:       0f 85 70 01 00 00       jne    ffffffff811b09e3 <xfs_rename+0x4dd>
ffffffff811b087d:       74 35                   je     ffffffff811b08b4 <xfs_rename+0x3ae>
ffffffff811b0884:       74 2e                   je     ffffffff811b08b4 <xfs_rename+0x3ae>
ffffffff811b08ae:       0f 85 2f 01 00 00       jne    ffffffff811b09e3 <xfs_rename+0x4dd>
ffffffff811b08c6:       74 21                   je     ffffffff811b08e9 <xfs_rename+0x3e3>
ffffffff811b08cb:       75 07                   jne    ffffffff811b08d4 <xfs_rename+0x3ce>
ffffffff811b08d2:       74 15                   je     ffffffff811b08e9 <xfs_rename+0x3e3>
ffffffff811b08e3:       0f 85 fa 00 00 00       jne    ffffffff811b09e3 <xfs_rename+0x4dd>
ffffffff811b0910:       0f 85 cd 00 00 00       jne    ffffffff811b09e3 <xfs_rename+0x4dd>
ffffffff811b0941:       74 18                   je     ffffffff811b095b <xfs_rename+0x455>
ffffffff811b0966:       74 09                   je     ffffffff811b0971 <xfs_rename+0x46b>
ffffffff811b098a:       74 21                   je     ffffffff811b09ad <xfs_rename+0x4a7>
ffffffff811b09a2:       e8 09 0e 00 00          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b09ab:       eb 11                   jmp    ffffffff811b09be <xfs_rename+0x4b8>
ffffffff811b09d5:       eb 11                   jmp    ffffffff811b09e8 <xfs_rename+0x4e2>
ffffffff811b09e1:       eb 05                   jmp    ffffffff811b09e8 <xfs_rename+0x4e2>
ffffffff811b09f8:       eb a3                   jmp    ffffffff811b099d <xfs_rename+0x497>
ffffffff811b17b0 <xfs_trans_cancel>:
ffffffff811b17c1:       74 0c                   je     ffffffff811b17cf <xfs_trans_cancel+0x1f>
ffffffff811b17d3:       74 4a                   je     ffffffff811b181f <xfs_trans_cancel+0x6f>
ffffffff811b17de:       75 3f                   jne    ffffffff811b181f <xfs_trans_cancel+0x6f>
ffffffff811b1839:       74 06                   je     ffffffff811b1841 <xfs_trans_cancel+0x91>
ffffffff811b1848:       74 12                   je     ffffffff811b185c <xfs_trans_cancel+0xac>
ffffffff811b3bb7:       e8 f4 db ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b3c32:       e8 79 db ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b4753:       e8 58 d0 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b53e9:       e8 c2 c3 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b5497:       e8 14 c3 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b5baa:       e8 01 bc ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b5f40:       e8 6b b8 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b6000:       e8 ab b7 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b6458:       e8 53 b3 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b6730:       e8 7b b0 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b6a58:       e8 53 ad ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b6c5c:       e8 4f ab ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b6c95:       e8 16 ab ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b6cf7:       e8 b4 aa ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b6d83:       e8 28 aa ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b706b:       e8 40 a7 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b715b:       e8 50 a6 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b7305:       e8 a6 a4 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b7372:       e8 39 a4 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b7407:       e8 a4 a3 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b74e5:       e8 c6 a2 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b77a9:       e8 02 a0 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b7f94:       e8 17 98 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b83e8:       e8 c3 93 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b866b:       e8 40 91 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b8838:       e8 73 8f ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b8bb0:       e8 fb 8b ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b8d2c:       e8 7f 8a ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b8f17:       e8 94 88 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b9463:       e8 48 83 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b950f:       e8 9c 82 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811b9677:       e8 34 81 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811be2af:       e8 fc 34 ff ff          callq  ffffffff811b17b0 <xfs_trans_cancel>
ffffffff811bfacc:       e8 35 0a ff ff          callq  ffffffff811b0506 <xfs_rename>

Gabriel

Quoted 17 lines

-Eric
Quoted 14 lines
I _really_ cannot upgrade to 2.6.29 or later because of the "reconnect_path: npd != pd" bug and the maybe related radix-tree bug ( http://bugzilla.kernel.org/show_bug.cgi?id=13375 ) affecting all kernel version afeter 2.6.28.
Unmounting then remounting the filesystem allow to access the mountpoint again without any error message or apparent file corruption. This filesystem is used by ~30 NFS clients and contains about 5M files (100GB).
Before using the volume over NFS, there was only local activity (rsync syncing) and we didn't get any error.
I expect to see this crash again in a few hours except if the volume is really corrupted. Does a full filesystem copy to a newly created volume would have a chance to solve the problem?
Thanks,
Gabriel

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Eric Sandeen

2009-07-27 17:39 UTC #3 lore raw

Quoted 27 lines · Gabriel Barazer

Quoted 19 lines · Eric Sandeen
Quoted 11 lines · Gabriel Barazer
Hi,
I recently put a NFS file server into production, with mostly XFS volumes on LVM. The server was quite low on traffic until this morning and one of the filesystems crashed twice since this morning with the following backtrace:
Filesystem "dm-24": XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs_trans.c. Caller 0xffffffff811b09a7
Pid: 2053, comm: nfsd Not tainted 2.6.28.9-filer #1
Call Trace:
 [<ffffffff811b09a7>] xfs_rename+0x4a1/0x4f6
 [<ffffffff811b1806>] xfs_trans_cancel+0x56/0xed
 [<ffffffff811b09a7>] xfs_rename+0x4a1/0x4f6
...
Quoted 3 lines
xfs_force_shutdown(dm-24,0x8) called from line 1165 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff811b181f Filesystem "dm-24": Corruption of in-memory data detected. Shutting down filesystem: dm-24
Quoted 7 lines
Quoted 2 lines
The two crashed are related to the same function: xfs_rename.
Can you do objdump -d xfs.ko | grep "xfs_rename\|xfs_trans_cancel" and maybe we can see which call to xfs_trans_cancel in xfs_rename this was.
The problem relates to canceling a dirty transaction on an error path.

Quoted 7 lines

Hi,
sorry for the late reply
I don't have any xfs.ko as my kernel is compiled without CONFIG_MODULES. However I objdump'd the vmlinux uncompressed kernel, and here are the results:

Ok, that was an over eager grep command, my apologies to the mail archives ;)

The relevant stuff:

ffffffff811b0506 <xfs_rename>:
ffffffff811b06c1: e8 ea 10 00 00 callq ffffffff811b17b0
<xfs_trans_cancel>
ffffffff811b09a2: e8 09 0e 00 00 callq ffffffff811b17b0
<xfs_trans_cancel>

hmm but there are only 2 obvious calls in the disassembly, and there are 4 calls in the function... and neither one seems to line up with your stated offset in the oops. :( I was hoping to sort out which xfs_trans_cancel call in xfs_rename it was.

Any chance you could add a couple printk's to xfs_rename in the cases where it calls trans_cancel so we can see which one it was?

Thanks,
-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Gabriel Barazer

author

2009-07-28 00:31 UTC #4 lore raw

Quoted 53 lines · Eric Sandeen

Quoted 30 lines · Gabriel Barazer
Quoted 28 lines · Eric Sandeen
Quoted 12 lines · Gabriel Barazer
Hi,
I recently put a NFS file server into production, with mostly XFS volumes on LVM. The server was quite low on traffic until this morning and one of the filesystems crashed twice since this morning with the following backtrace:
Filesystem "dm-24": XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs_trans.c. Caller 0xffffffff811b09a7
Pid: 2053, comm: nfsd Not tainted 2.6.28.9-filer #1
Call Trace:
 [<ffffffff811b09a7>] xfs_rename+0x4a1/0x4f6
 [<ffffffff811b1806>] xfs_trans_cancel+0x56/0xed
 [<ffffffff811b09a7>] xfs_rename+0x4a1/0x4f6
...
Quoted 6 lines
xfs_force_shutdown(dm-24,0x8) called from line 1165 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff811b181f Filesystem "dm-24": Corruption of in-memory data detected. Shutting down filesystem: dm-24
The two crashed are related to the same function: xfs_rename.
Can you do objdump -d xfs.ko | grep "xfs_rename\|xfs_trans_cancel" and maybe we can see which call to xfs_trans_cancel in xfs_rename this was.
Quoted 9 lines
Quoted 3 lines
The problem relates to canceling a dirty transaction on an error path.
Hi,
sorry for the late reply
I don't have any xfs.ko as my kernel is compiled without CONFIG_MODULES. However I objdump'd the vmlinux uncompressed kernel, and here are the results:
Ok, that was an over eager grep command, my apologies to the mail archives ;)
The relevant stuff:
ffffffff811b0506 <xfs_rename>:
ffffffff811b06c1: e8 ea 10 00 00 callq ffffffff811b17b0
<xfs_trans_cancel>
ffffffff811b09a2: e8 09 0e 00 00 callq ffffffff811b17b0
<xfs_trans_cancel>

Quoted 5 lines

hmm but there are only 2 obvious calls in the disassembly, and there are 4 calls in the function... and neither one seems to line up with your stated offset in the oops. :( I was hoping to sort out which xfs_trans_cancel call in xfs_rename it was.

I disassembled the uncompressed version of the linux kernel, generated at compile time in the build directory. I don't know if compressing the kernel to a bzImage file can change offsets compared to the uncompressed vmlinux kernel. I still have all the compiled sources for that kernel, including the .o files. Does any of these files could contain the offset you are looking for ?

Quoted 3 lines

Any chance you could add a couple printk's to xfs_rename in the cases where it calls trans_cancel so we can see which one it was?

This kernel and these bugs occured on a live production file server, and I really cannot mess with it. Good news though, I did not get any other shutdown since my last e-mail.

One detail that might be useful in case that bug is a race between 2 functions somewhere: The disks containing the filesystem is a SSD RAID plugged to a 3ware adapter with write cache enabled; that is as those SSD are very irregular in write speeds, the writes are occuring in short bursts, then any I/O to the disks are blocked for a few seconds until the next burst (see the purple line : http://pub.grosboulet.com/benchmark-seqwrite.jpg ). (BTW, I _really_ don't recommend using Intel X25-M SSDs in server systems, thoses are only good for desktop/laptop systems and are worse than SAS15K drives in multiuser writes). This very odd behaviour could lead the kernel to block/wait at unusual places in the code like where that bug is occuring.

Gabriel

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs