File system corruption

Hey Folks:

I'm periodically encountering an issue with XFS that you might perhaps be interested in. The environment in which this manifests itself is on a CentOS Linux machine (custom 2.6.28.7 kernel), which is serving the XFS mount point in question with the standard Linux nfsd. The XFS file system lives on an LVM device in a striping configuration (2 wide stripe), with two iSCSI volumes acting as the constituent physical volumes. This configuration is somewhat baroque, I know.

I'm experiencing periodic file system corruption, which manifests in the XFS file system going offline, and refusing subsequent mounts. The only way to recover from this has been to perform a xfs_repair -L, which has resulted in data loss on each occasion, as expected.

Now, here's what I witness in the system logs:

<snip>
kernel: XFS: bad magic number
kernel: XFS: SB validate failed

kernel: 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
kernel: Filesystem "dm-0": XFS internal error xfs_ialloc_read_agi at line 1408 of file fs/xfs/xfs_ialloc.c.  Caller 0xffffffff8118711a
kernel: Pid: 3842, comm: nfsd Not tainted 2.6.28.7.cs.8 #3 kernel: Call Trace:
kernel:  [<ffffffff8118711a>] xfs_ialloc_ag_select+0x22a/0x320
kernel:  [<ffffffff81186481>] xfs_ialloc_read_agi+0xe1/0x140
kernel:  [<ffffffff8118711a>] xfs_ialloc_ag_select+0x22a/0x320
kernel:  [<ffffffff811f5bfd>] swiotlb_map_single_attrs+0x1d/0xf0
kernel:  [<ffffffff8118711a>] xfs_ialloc_ag_select+0x22a/0x320
kernel:  [<ffffffff81187bfc>] xfs_dialloc+0x31c/0xa90
kernel:  [<ffffffff81076be5>] __alloc_pages_internal+0xf5/0x4f0
kernel:  [<ffffffff8109ac46>] cache_alloc_refill+0x96/0x5a0
kernel:  [<ffffffff8119012f>] xfs_ialloc+0x7f/0x6f0
kernel:  [<ffffffff811ad0c6>] kmem_zone_alloc+0x86/0xc0
kernel:  [<ffffffff811a66d8>] xfs_dir_ialloc+0xa8/0x360
kernel:  [<ffffffff811a4008>] xfs_trans_reserve+0xa8/0x220
kernel:  [<ffffffff813a29e7>] __down_write_nested+0x17/0xa0
kernel:  [<ffffffff811a952f>] xfs_create+0x2ef/0x4e0
kernel:  [<ffffffff811b523c>] xfs_vn_mknod+0x14c/0x1a0
kernel:  [<ffffffff810a864c>] vfs_create+0xec/0x160
kernel:  [<ffffffffa00c53c3>] nfsd_create_v3+0x3b3/0x500 [nfsd]
kernel:  [<ffffffffa00cc178>] nfsd3_proc_create+0x118/0x1b0 [nfsd]
kernel:  [<ffffffffa00be22a>] nfsd_dispatch+0xba/0x270 [nfsd]
kernel:  [<ffffffffa0061fde>] svc_process+0x49e/0x800 [sunrpc]
kernel:  [<ffffffff8102efc0>] default_wake_function+0x0/0x10
kernel:  [<ffffffff813a2a97>] __down_read+0x17/0xa6
kernel:  [<ffffffffa00be9a9>] nfsd+0x199/0x2c0 [nfsd]
kernel:  [<ffffffffa00be810>] nfsd+0x0/0x2c0 [nfsd]
kernel:  [<ffffffff8104a4b7>] kthread+0x47/0x90
kernel:  [<ffffffff810322a7>] schedule_tail+0x27/0x70
kernel:  [<ffffffff8100d0d9>] child_rip+0xa/0x11
kernel:  [<ffffffff8104a470>] kthread+0x0/0x90
kernel:  [<ffffffff8100d0cf>] child_rip+0x0/0x11

</snip>

The resultant stack trace coming from "XFS internal error xfs_ialloc_read_agi" repeats itself numerous times, at which point, the following is seen:

<snip>

kernel: 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
kernel: Filesystem "dm-0": XFS internal error xfs_alloc_read_agf at line 2194 of file fs/xfs/xfs_alloc.c.  Caller 0xffffffff8115cf09
kernel: Pid: 3756, comm: nfsd Not tainted 2.6.28.7.cs.8 #3
kernel: Call Trace:
kernel:  [<ffffffff8115cf09>] xfs_alloc_fix_freelist+0x3e9/0x480
kernel:  [<ffffffff8115abe3>] xfs_alloc_read_agf+0xd3/0x1e0
kernel:  [<ffffffff8115cf09>] xfs_alloc_fix_freelist+0x3e9/0x480
kernel:  [<ffffffff8100d0cf>] child_rip+0x0/0x11
kernel:  [<ffffffff8115cf09>] xfs_alloc_fix_freelist+0x3e9/0x480
kernel:  [<ffffffff811e8033>] vsnprintf+0x743/0x890
kernel:  [<ffffffff81268a8a>] wait_for_xmitr+0x5a/0xc0
kernel:  [<ffffffff8100d0cf>] child_rip+0x0/0x11
kernel:  [<ffffffff813a2a97>] __down_read+0x17/0xa6
kernel:  [<ffffffff8115d215>] xfs_alloc_vextent+0x1b5/0x4e0
kernel:  [<ffffffff8116c0e8>] xfs_bmap_btalloc+0x608/0xb00
kernel:  [<ffffffff8116f60a>] xfs_bmapi+0xa4a/0x12a0
kernel:  [<ffffffff8118e93c>] xfs_imap_to_bp+0xac/0x130
kernel:  [<ffffffff8117a37a>] xfs_dir2_grow_inode+0x15a/0x410
kernel:  [<ffffffff8117b26f>] xfs_dir2_sf_to_block+0x9f/0x5c0
kernel:  [<ffffffff811ad0c6>] kmem_zone_alloc+0x86/0xc0
kernel:  [<ffffffff811ad132>] kmem_zone_zalloc+0x32/0x50
kernel:  [<ffffffff811918ce>] xfs_inode_item_init+0x1e/0x80
kernel:  [<ffffffff81183880>] xfs_dir2_sf_addname+0x430/0x5d0
kernel:  [<ffffffff811903c8>] xfs_ialloc+0x318/0x6f0
kernel:  [<ffffffff8117b0a2>] xfs_dir_createname+0x182/0x1e0
kernel:  [<ffffffff811a95df>] xfs_create+0x39f/0x4e0
kernel:  [<ffffffff811b523c>] xfs_vn_mknod+0x14c/0x1a0
kernel:  [<ffffffff810a864c>] vfs_create+0xec/0x160
kernel:  [<ffffffffa00c53c3>] nfsd_create_v3+0x3b3/0x500 [nfsd]
kernel:  [<ffffffffa00cc178>] nfsd3_proc_create+0x118/0x1b0 [nfsd]
kernel:  [<ffffffffa00be22a>] nfsd_dispatch+0xba/0x270 [nfsd]
kernel:  [<ffffffffa0061fde>] svc_process+0x49e/0x800 [sunrpc]
kernel:  [<ffffffff813a2a97>] __down_read+0x17/0xa6
kernel:  [<ffffffffa00be9a9>] nfsd+0x199/0x2c0 [nfsd]
kernel:  [<ffffffffa00be810>] nfsd+0x0/0x2c0 [nfsd]
kernel:  [<ffffffff8104a4b7>] kthread+0x47/0x90
kernel:  [<ffffffff810322a7>] schedule_tail+0x27/0x70
kernel:  [<ffffffff8100d0d9>] child_rip+0xa/0x11
kernel:  [<ffffffff8104a470>] kthread+0x0/0x90
kernel:  [<ffffffff8100d0cf>] child_rip+0x0/0x11

kernel: Filesystem "dm-0": XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs_trans.c.  Caller 0xffffffff811a9411
kernel: Pid: 3756, comm: nfsd Not tainted 2.6.28.7.cs.8 #3
kernel: Call Trace:
kernel:  [<ffffffff811a9411>] xfs_create+0x1d1/0x4e0
kernel:  [<ffffffff811a3475>] xfs_trans_cancel+0xe5/0x110
kernel:  [<ffffffff811a9411>] xfs_create+0x1d1/0x4e0
kernel:  [<ffffffff811b523c>] xfs_vn_mknod+0x14c/0x1a0
kernel:  [<ffffffff810a864c>] vfs_create+0xec/0x160
kernel:  [<ffffffffa00c53c3>] nfsd_create_v3+0x3b3/0x500 [nfsd]
kernel:  [<ffffffffa00cc178>] nfsd3_proc_create+0x118/0x1b0 [nfsd]
kernel:  [<ffffffffa00be22a>] nfsd_dispatch+0xba/0x270 [nfsd]
kernel:  [<ffffffffa0061fde>] svc_process+0x49e/0x800 [sunrpc]
kernel:  [<ffffffff813a2a97>] __down_read+0x17/0xa6
kernel:  [<ffffffffa00be9a9>] nfsd+0x199/0x2c0 [nfsd]
kernel:  [<ffffffffa00be810>] nfsd+0x0/0x2c0 [nfsd]
kernel:  [<ffffffff8104a4b7>] kthread+0x47/0x90
kernel:  [<ffffffff810322a7>] schedule_tail+0x27/0x70
kernel:  [<ffffffff8100d0d9>] child_rip+0xa/0x11
kernel:  [<ffffffff8104a470>] kthread+0x0/0x90
kernel:  [<ffffffff8100d0cf>] child_rip+0x0/0x11
kernel: xfs_force_shutdown(dm-0,0x8) called from line 1165 of file fs/xfs/xfs_trans.c.  Return address = 0xffffffff811a348e
kernel: Filesystem "dm-0": Corruption of in-memory data detected.  Shutting down filesystem: dm-0
kernel: Please umount the filesystem, and rectify the problem(s)
kernel: nfsd: non-standard errno: -117

kernel: Filesystem "dm-0": xfs_log_force: error 5 returned.

</snip>

I'm somewhat at a loss with this one - it's been experienced on a customer's installation, so I don't have ready access to the machine. All internal tests to attempt reproduction with identical hardware/software configurations has been unfruitful. I'm concerned about the custom kernel, and may attempt to downgrade to the stock CentOS 5.3 kernel (2.6.18, if I remember correctly).

Any insight would be hugely appreciated, and of course tell me how I can help further. Thanks so much.

John Quigley
jquigley.com

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

The log corruption might be related to data reordering somewhere along your IO path, though I wouldn't swear to it. But often when write caches are on, barriers are off, and power is lost, this sort of thing shows up.

That's the first error?

This means that after you read an agi, it failed a sanity test:

1403 be32_to_cpu(agi->agi_magicnum) == XFS_AGI_MAGIC &&
1404 XFS_AGI_GOOD_VERSION(be32_to_cpu(agi->agi_versionnum));

bad magic number, etc. The "00 00 00 00 ..." is the contents of the buffer that it thought was the agi, containing all that wonderful magic
- but it's all 0s.

...

Similar, but bad info on the AGF:
2184 agf_ok =
2185 be32_to_cpu(agf->agf_magicnum) == XFS_AGF_MAGIC &&
2186 XFS_AGF_GOOD_VERSION(be32_to_cpu(agf->agf_versionnum)) &&
2187 be32_to_cpu(agf->agf_freeblks) <=
be32_to_cpu(agf->agf_length) &&

2188                 be32_to_cpu(agf->agf_flfirst) < XFS_AGFL_SIZE(mp) &&
2189                 be32_to_cpu(agf->agf_fllast) < XFS_AGFL_SIZE(mp) &&
2190                 be32_to_cpu(agf->agf_flcount) <= XFS_AGFL_SIZE(mp);

Again w/ the zeros ...

...

and then the fs tried to back out of a dirty transaction, which it can't do, but that's secondary.

117 EFSCORRUPTED IIRC?

EIO

I'm happy to blame the storage here, given the buffers full of 0s ... you could modify the messages to print the block nrs in question and go back directly to the storage, read it, and see what's there.

Were there no iscsi or other assorted messages before all this?

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs