A corruption that seems to span a few kernels
XFS file system, built using a 2.6.32.41 kernel. External journals.
repair the file system, mount, turn on gluster (3.2.5), do a find, and get a corruption. Actually it looks like xfs_repair doesn't fix the problem.
Rinse repeat, reproduce. Works (procedure to generate the corruption) in 2.6.32.41.scalable, 2.6.39.1, 3.2.2
[ 428.457044] XFS (sdc2): corrupt dinode 54105173, extent total = 1, nblocks = 0. [ 428.464596] ffff880bde999500: 49 4e 81 a4 01 02 00 01 00 00 04 65 00 00 07 df IN.........e.... [ 428.473587] XFS (sdc2): Internal error xfs_iformat(1) at line 319 of file fs/xfs/xfs_inode.c. Caller 0xffffffff8127b738 [ 428.473589] [ 428.486388] Pid: 7562, comm: glusterfsd Not tainted 3.2.2.scalable #1 [ 428.493014] Call Trace: [ 428.495544] [<ffffffff812361aa>] xfs_error_report+0x3a/0x40 [ 428.501391] [<ffffffff8127b738>] ? xfs_iread+0x118/0x1a0 [ 428.506894] [<ffffffff81236208>] xfs_corruption_error+0x58/0x80 [ 428.513027] [<ffffffff8127b4b1>] xfs_iformat+0x401/0x570 [ 428.518491] [<ffffffff8127b738>] ? xfs_iread+0x118/0x1a0 [ 428.524003] [<ffffffff8127b738>] xfs_iread+0x118/0x1a0 [ 428.529292] [<ffffffff8123aba3>] xfs_iget+0x2d3/0x8b0 [ 428.534537] [<ffffffff8123a252>] ? xfs_iunlock+0x92/0x130 [ 428.540122] [<ffffffff81246c68>] xfs_lookup+0xb8/0x140 [ 428.545413] [<ffffffff8123fc59>] xfs_vn_lookup+0x59/0xa0 [ 428.550922] [<ffffffff8113f545>] ? d_alloc+0x75/0x90 [ 428.556034] [<ffffffff81134d14>] d_alloc_and_lookup+0x44/0x80 [ 428.561976] [<ffffffff81135213>] do_lookup+0x243/0x350 [ 428.567326] [<ffffffff8113867a>] path_lookupat+0xea/0x740 [ 428.572942] [<ffffffff8124fea2>] ? xfs_attr_get+0xb2/0xc0 [ 428.578489] [<ffffffff81138d00>] do_path_lookup+0x30/0xc0 [ 428.584089] [<ffffffff81139974>] user_path_at_empty+0x54/0xa0 [ 428.589984] [<ffffffff8111cf42>] ? kmem_cache_free+0x22/0x120 [ 428.595932] [<ffffffff81137901>] ? putname+0x31/0x50 [ 428.601091] [<ffffffff8113997f>] ? user_path_at_empty+0x5f/0xa0 [ 428.607190] [<ffffffff811399cc>] user_path_at+0xc/0x10 [ 428.612586] [<ffffffff8112ef46>] vfs_fstatat+0x46/0x80 [ 428.617868] [<ffffffff8112efe9>] vfs_lstat+0x19/0x20 [ 428.623068] [<ffffffff8112f00f>] sys_newlstat+0x1f/0x50 [ 428.628626] [<ffffffff8109e2bd>] ? audit_syscall_entry+0x16d/0x1d0 [ 428.635009] [<ffffffff8153f752>] system_call_fastpath+0x16/0x1b [ 428.641352] XFS (sdc2): Corruption detected. Unmount and run xfs_repair
[root@jr4-3 ~]# /etc/init.d/glusterd stop
Stopping glusterd: [ OK ]
[root@jr4-3 ~]# umount /data/brick-sdc2
[root@jr4-3 ~]# nohup xfs_repair -vvv -l /dev/md1 /dev/sdc2 > /tmp/out2.sdc 2>&1 &
[1] 7689
xfs_repair runs. File system becomes normal. Can reboot or remain in the same system. Select any of the kernels. Mount the file system.
I haven't checked whether this only happens with Gluster yet or not, working on it next ... will do the find w/o gluster on the backing store. See if it croaks.
Currently booted into 3.2.2, can boot into something else.
Any hints? I can reproduce it quite well, and xfs_repair seems to work ... (repeated passes don't seem to help).
[root@jr4-3 ~]# grep "corrupt dinode 54105173" /var/log/messages
Feb 2 23:08:46 jr4-3 kernel: [ 1548.511502] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 2 23:13:32 jr4-3 kernel: [ 1834.627251] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 2 23:37:44 jr4-3 kernel: [ 3286.598877] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 00:07:43 jr4-3 kernel: [ 5085.429344] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 00:37:44 jr4-3 kernel: [ 6886.552328] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 01:07:43 jr4-3 kernel: [ 8686.384728] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 01:37:47 jr4-3 kernel: [10490.113301] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 02:07:46 jr4-3 kernel: [12289.248980] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 02:37:43 jr4-3 kernel: [14086.210521] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 03:07:45 jr4-3 kernel: [15888.798065] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 03:37:43 jr4-3 kernel: [17686.388882] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 04:04:06 jr4-3 kernel: [19269.902522] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 04:08:04 jr4-3 kernel: [19507.447343] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 04:38:11 jr4-3 kernel: [21314.248443] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 05:07:43 jr4-3 kernel: [23086.714157] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 05:37:43 jr4-3 kernel: [24886.444238] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 06:07:43 jr4-3 kernel: [26686.740589] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 06:37:45 jr4-3 kernel: [28488.946147] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 07:07:42 jr4-3 kernel: [30286.417803] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 07:37:42 jr4-3 kernel: [32085.847705] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 08:08:01 jr4-3 kernel: [33904.979795] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 08:37:46 jr4-3 kernel: [35690.043019] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 08:47:19 jr4-3 kernel: [36263.597170] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 09:07:42 jr4-3 kernel: [37486.293617] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 09:37:42 jr4-3 kernel: [39285.987621] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 10:07:42 jr4-3 kernel: [41086.440845] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 10:37:55 jr4-3 kernel: [42899.537075] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 11:08:24 jr4-3 kernel: [ 988.628970] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 13:08:20 jr4-3 kernel: [ 2531.085884] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 13:10:13 jr4-3 kernel: [ 2643.954262] Filesystem "sdc2": corrupt dinode 54105173, extent total = 1, nblocks = 0. Unmount and run xfs_repair. Feb 3 08:38:20 jr4-3 kernel: [ 428.457044] XFS (sdc2): corrupt dinode 54105173, extent total = 1, nblocks = 0.
Any clues/hints/suggestions? Zero out that dinode (and if so, how?) Let me know if you want any additional information from this system.
Its a large RAID6, with battery backed ECC cache. No disk events reported.
This appeared after finishing the xfs_repair
[root@jr4-3 ~]# [ 1102.898890] scsi_verify_blk_ioctl: 56 callbacks suppressed
[ 1102.904486] xfs_repair: sending ioctl 1261 to a partition! [ 1102.910286] xfs_repair: sending ioctl 1261 to a partition!
Output from xfs_repair below. Notes are in phase 3 and 4
phase 3:
data fork in regular inode 54105173 claims used block 6745835296
phase 4:
data fork in regular inode 54105173 claims used block 6745835296
xfs_repair doesn't seem to fix this, as a second pass through still indicates a problem.
First pass:
[root@jr4-3 ~]# cat !$
cat /tmp/out2.sdc
Phase 1 - find and verify superblock...
- max_mem = 37094007, icount = 5448960, imem = 21285, dblock =
4391112384, dmem = 2144097
- block cache size set to 4359824 entries
Phase 2 - using external log on /dev/md1
- zero log...zero_log: head block 48836 tail block 48836
- scan filesystem freespace and inode maps...
- found root inode chunklibxfs_bcache: 0x6814c0
Max supported entries = 4359824
Max utilized entries = 3278
Active entries = 3278
Hash table size = 544978
Hits = 0
Misses = 3278
Hit ratio = 0.00
MRU 0 entries = 3278 (100%)
MRU 1 entries = 0 ( 0%) MRU 2 entries = 0 ( 0%) MRU 3 entries = 0 ( 0%) MRU 4 entries = 0 ( 0%) MRU 5 entries = 0 ( 0%) MRU 6 entries = 0 ( 0%) MRU 7 entries = 0 ( 0%) MRU 8 entries = 0 ( 0%) MRU 9 entries = 0 ( 0%) MRU 10 entries = 0 ( 0%) MRU 11 entries = 0 ( 0%) MRU 12 entries = 0 ( 0%) MRU 13 entries = 0 ( 0%) MRU 14 entries = 0 ( 0%) MRU 15 entries = 0 ( 0%) Hash buckets with 0 entries 541912 ( 0%) Hash buckets with 1 entries 2903 ( 88%) Hash buckets with 2 entries 129 ( 7%) Hash buckets with 3 entries 21 ( 1%) Hash buckets with 4 entries 11 ( 1%) Hash buckets with 5 entries 2 ( 0%)
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
data fork in regular inode 54105173 claims used block 6745835296
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- process newly discovered inodes...libxfs_bcache: 0x6814c0
Max supported entries = 4359824
Max utilized entries = 1052814
Active entries = 1052814
Hash table size = 544978
Hits = 302945
Misses = 1052814
Hit ratio = 22.35
MRU 0 entries = 851716 ( 80%)
MRU 1 entries = 6724 ( 0%)
MRU 2 entries = 68352 ( 6%)
MRU 3 entries = 108276 ( 10%)
MRU 4 entries = 5278 ( 0%) MRU 5 entries = 0 ( 0%) MRU 6 entries = 12339 ( 1%) MRU 7 entries = 16 ( 0%) MRU 8 entries = 0 ( 0%) MRU 9 entries = 0 ( 0%) MRU 10 entries = 0 ( 0%) MRU 11 entries = 3 ( 0%) MRU 12 entries = 110 ( 0%) MRU 13 entries = 0 ( 0%) MRU 14 entries = 0 ( 0%) MRU 15 entries = 0 ( 0%) Hash buckets with 0 entries 206656 ( 0%) Hash buckets with 1 entries 86130 ( 8%) Hash buckets with 2 entries 71589 ( 13%) Hash buckets with 3 entries 20312 ( 5%) Hash buckets with 4 entries 99262 ( 37%) Hash buckets with 5 entries 25665 ( 12%) Hash buckets with 6 entries 21611 ( 12%) Hash buckets with 7 entries 3104 ( 2%) Hash buckets with 8 entries 10077 ( 7%) Hash buckets with 9 entries 535 ( 0%) Hash buckets with 10 entries 35 ( 0%) Hash buckets with 11 entries 2 ( 0%)
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 14
- agno = 15
- agno = 18
- agno = 16
- agno = 20
- agno = 22
- agno = 19
- agno = 25
- agno = 26
- agno = 21
- agno = 29
- agno = 24
- agno = 31
- agno = 17
- agno = 27
- agno = 28
- agno = 23
- agno = 30
- agno = 13
data fork in regular inode 54105173 claims used block 6745835296libxfs_bcache: 0x6814c0
Max supported entries = 4359824
Max utilized entries = 1052814
Active entries = 1052814
Hash table size = 544978
Hits = 646498
Misses = 1052814
Hit ratio = 38.04
MRU 0 entries = 1052701 ( 99%)
MRU 1 entries = 0 ( 0%) MRU 2 entries = 0 ( 0%) MRU 3 entries = 0 ( 0%) MRU 4 entries = 0 ( 0%) MRU 5 entries = 0 ( 0%) MRU 6 entries = 0 ( 0%) MRU 7 entries = 0 ( 0%) MRU 8 entries = 0 ( 0%) MRU 9 entries = 0 ( 0%) MRU 10 entries = 0 ( 0%) MRU 11 entries = 3 ( 0%) MRU 12 entries = 110 ( 0%) MRU 13 entries = 0 ( 0%) MRU 14 entries = 0 ( 0%) MRU 15 entries = 0 ( 0%) Hash buckets with 0 entries 206656 ( 0%) Hash buckets with 1 entries 86130 ( 8%) Hash buckets with 2 entries 71589 ( 13%) Hash buckets with 3 entries 20312 ( 5%) Hash buckets with 4 entries 99262 ( 37%) Hash buckets with 5 entries 25665 ( 12%) Hash buckets with 6 entries 21611 ( 12%) Hash buckets with 7 entries 3104 ( 2%) Hash buckets with 8 entries 10077 ( 7%) Hash buckets with 9 entries 535 ( 0%) Hash buckets with 10 entries 35 ( 0%) Hash buckets with 11 entries 2 ( 0%)
Phase 5 - rebuild AG headers and trees...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- reset superblock...libxfs_bcache: 0x6814c0
Max supported entries = 4359824
Max utilized entries = 1052814
Active entries = 1052814
Hash table size = 544978
Hits = 649707
Misses = 1052814
Hit ratio = 38.16
MRU 0 entries = 1052701 ( 99%)
MRU 1 entries = 0 ( 0%) MRU 2 entries = 0 ( 0%) MRU 3 entries = 0 ( 0%) MRU 4 entries = 0 ( 0%) MRU 5 entries = 0 ( 0%) MRU 6 entries = 0 ( 0%) MRU 7 entries = 0 ( 0%) MRU 8 entries = 0 ( 0%) MRU 9 entries = 0 ( 0%) MRU 10 entries = 0 ( 0%) MRU 11 entries = 3 ( 0%) MRU 12 entries = 110 ( 0%) MRU 13 entries = 0 ( 0%) MRU 14 entries = 0 ( 0%) MRU 15 entries = 0 ( 0%) Hash buckets with 0 entries 206656 ( 0%) Hash buckets with 1 entries 86130 ( 8%) Hash buckets with 2 entries 71589 ( 13%) Hash buckets with 3 entries 20312 ( 5%) Hash buckets with 4 entries 99262 ( 37%) Hash buckets with 5 entries 25665 ( 12%) Hash buckets with 6 entries 21611 ( 12%) Hash buckets with 7 entries 3104 ( 2%) Hash buckets with 8 entries 10077 ( 7%) Hash buckets with 9 entries 535 ( 0%) Hash buckets with 10 entries 35 ( 0%) Hash buckets with 11 entries 2 ( 0%)
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- traversal finished ...
- moving disconnected inodes to lost+found ...libxfs_bcache: 0x6814c0
Max supported entries = 4359824
Max utilized entries = 1052845
Active entries = 1052845
Hash table size = 544978
Hits = 1854913
Misses = 1052845
Hit ratio = 63.79
MRU 0 entries = 1016092 ( 96%)
MRU 1 entries = 0 ( 0%) MRU 2 entries = 0 ( 0%) MRU 3 entries = 3 ( 0%) MRU 4 entries = 141 ( 0%) MRU 5 entries = 0 ( 0%) MRU 6 entries = 0 ( 0%) MRU 7 entries = 11363 ( 1%) MRU 8 entries = 25246 ( 2%) MRU 9 entries = 0 ( 0%) MRU 10 entries = 0 ( 0%) MRU 11 entries = 0 ( 0%) MRU 12 entries = 0 ( 0%) MRU 13 entries = 0 ( 0%) MRU 14 entries = 0 ( 0%) MRU 15 entries = 0 ( 0%) Hash buckets with 0 entries 206646 ( 0%) Hash buckets with 1 entries 86134 ( 8%) Hash buckets with 2 entries 71593 ( 13%) Hash buckets with 3 entries 20312 ( 5%) Hash buckets with 4 entries 99256 ( 37%) Hash buckets with 5 entries 25670 ( 12%) Hash buckets with 6 entries 21614 ( 12%) Hash buckets with 7 entries 3104 ( 2%) Hash buckets with 8 entries 10077 ( 7%) Hash buckets with 9 entries 535 ( 0%) Hash buckets with 10 entries 35 ( 0%) Hash buckets with 11 entries 2 ( 0%)
Phase 7 - verify and correct link counts...libxfs_bcache: 0x6814c0
Max supported entries = 4359824
Max utilized entries = 1052845
Active entries = 1052845
Hash table size = 544978
Hits = 1854913
Misses = 1052845
Hit ratio = 63.79
MRU 0 entries = 1016092 ( 96%)
MRU 1 entries = 0 ( 0%) MRU 2 entries = 0 ( 0%) MRU 3 entries = 3 ( 0%) MRU 4 entries = 141 ( 0%) MRU 5 entries = 0 ( 0%) MRU 6 entries = 0 ( 0%) MRU 7 entries = 11363 ( 1%) MRU 8 entries = 25246 ( 2%) MRU 9 entries = 0 ( 0%) MRU 10 entries = 0 ( 0%) MRU 11 entries = 0 ( 0%) MRU 12 entries = 0 ( 0%) MRU 13 entries = 0 ( 0%) MRU 14 entries = 0 ( 0%) MRU 15 entries = 0 ( 0%) Hash buckets with 0 entries 206646 ( 0%) Hash buckets with 1 entries 86134 ( 8%) Hash buckets with 2 entries 71593 ( 13%) Hash buckets with 3 entries 20312 ( 5%) Hash buckets with 4 entries 99256 ( 37%) Hash buckets with 5 entries 25670 ( 12%) Hash buckets with 6 entries 21614 ( 12%) Hash buckets with 7 entries 3104 ( 2%) Hash buckets with 8 entries 10077 ( 7%) Hash buckets with 9 entries 535 ( 0%) Hash buckets with 10 entries 35 ( 0%) Hash buckets with 11 entries 2 ( 0%)
XFS_REPAIR Summary Fri Feb 3 08:49:34 2012
Phase Start End Duration
Phase 1: 02/03 08:43:06 02/03 08:43:06
Phase 2: 02/03 08:43:06 02/03 08:43:11 5 seconds
Phase 3: 02/03 08:43:11 02/03 08:48:48 5 minutes, 37 seconds
Phase 4: 02/03 08:48:48 02/03 08:49:26 38 seconds
Phase 5: 02/03 08:49:26 02/03 08:49:27 1 second
Phase 6: 02/03 08:49:27 02/03 08:49:33 6 seconds
Phase 7: 02/03 08:49:33 02/03 08:49:33
Total run time: 6 minutes, 27 seconds
done
Second pass:
[root@jr4-3 ~]# cat /tmp/out3.sdc
Phase 1 - find and verify superblock...
- max_mem = 37094007, icount = 5448960, imem = 21285, dblock =
4391112384, dmem = 2144097
- block cache size set to 4359824 entries
Phase 2 - using external log on /dev/md1
- zero log...zero_log: head block 2 tail block 2
- scan filesystem freespace and inode maps...
- found root inode chunklibxfs_bcache: 0x6814c0
Max supported entries = 4359824
Max utilized entries = 3240
Active entries = 3240
Hash table size = 544978
Hits = 0
Misses = 3240
Hit ratio = 0.00
MRU 0 entries = 3240 (100%)
MRU 1 entries = 0 ( 0%) MRU 2 entries = 0 ( 0%) MRU 3 entries = 0 ( 0%) MRU 4 entries = 0 ( 0%) MRU 5 entries = 0 ( 0%) MRU 6 entries = 0 ( 0%) MRU 7 entries = 0 ( 0%) MRU 8 entries = 0 ( 0%) MRU 9 entries = 0 ( 0%) MRU 10 entries = 0 ( 0%) MRU 11 entries = 0 ( 0%) MRU 12 entries = 0 ( 0%) MRU 13 entries = 0 ( 0%) MRU 14 entries = 0 ( 0%) MRU 15 entries = 0 ( 0%) Hash buckets with 0 entries 541938 ( 0%) Hash buckets with 1 entries 2881 ( 88%) Hash buckets with 2 entries 127 ( 7%) Hash buckets with 3 entries 23 ( 2%) Hash buckets with 4 entries 9 ( 1%)
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
data fork in regular inode 54105173 claims used block 6745835296
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- process newly discovered inodes...libxfs_bcache: 0x6814c0
Max supported entries = 4359824
Max utilized entries = 1052776
Active entries = 1052776
Hash table size = 544978
Hits = 302945
Misses = 1052776
Hit ratio = 22.35
MRU 0 entries = 851678 ( 80%)
MRU 1 entries = 6724 ( 0%)
MRU 2 entries = 68352 ( 6%)
MRU 3 entries = 107924 ( 10%)
MRU 4 entries = 5314 ( 0%) MRU 5 entries = 0 ( 0%) MRU 6 entries = 12655 ( 1%) MRU 7 entries = 16 ( 0%) MRU 8 entries = 0 ( 0%) MRU 9 entries = 0 ( 0%) MRU 10 entries = 0 ( 0%) MRU 11 entries = 3 ( 0%) MRU 12 entries = 110 ( 0%) MRU 13 entries = 0 ( 0%) MRU 14 entries = 0 ( 0%) MRU 15 entries = 0 ( 0%) Hash buckets with 0 entries 206666 ( 0%) Hash buckets with 1 entries 86129 ( 8%) Hash buckets with 2 entries 71585 ( 13%) Hash buckets with 3 entries 20311 ( 5%) Hash buckets with 4 entries 99263 ( 37%) Hash buckets with 5 entries 25662 ( 12%) Hash buckets with 6 entries 21611 ( 12%) Hash buckets with 7 entries 3103 ( 2%) Hash buckets with 8 entries 10076 ( 7%) Hash buckets with 9 entries 535 ( 0%) Hash buckets with 10 entries 35 ( 0%) Hash buckets with 11 entries 2 ( 0%)
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 2
- agno = 5
- agno = 7
- agno = 1
- agno = 3
- agno = 4
- agno = 11
- agno = 8
- agno = 14
- agno = 15
- agno = 9
- agno = 17
- agno = 18
- agno = 19
- agno = 10
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 6
- agno = 12
- agno = 25
- agno = 27
- agno = 26
- agno = 20
- agno = 28
- agno = 16
- agno = 31
- agno = 13
- agno = 29
- agno = 30
data fork in regular inode 54105173 claims used block 6745835296libxfs_bcache: 0x6814c0
Max supported entries = 4359824
Max utilized entries = 1052776
Active entries = 1052776
Hash table size = 544978
Hits = 646498
Misses = 1052776
Hit ratio = 38.05
MRU 0 entries = 1052663 ( 99%)
MRU 1 entries = 0 ( 0%) MRU 2 entries = 0 ( 0%) MRU 3 entries = 0 ( 0%) MRU 4 entries = 0 ( 0%) MRU 5 entries = 0 ( 0%) MRU 6 entries = 0 ( 0%) MRU 7 entries = 0 ( 0%) MRU 8 entries = 0 ( 0%) MRU 9 entries = 0 ( 0%) MRU 10 entries = 0 ( 0%) MRU 11 entries = 3 ( 0%) MRU 12 entries = 110 ( 0%) MRU 13 entries = 0 ( 0%) MRU 14 entries = 0 ( 0%) MRU 15 entries = 0 ( 0%) Hash buckets with 0 entries 206666 ( 0%) Hash buckets with 1 entries 86129 ( 8%) Hash buckets with 2 entries 71585 ( 13%) Hash buckets with 3 entries 20311 ( 5%) Hash buckets with 4 entries 99263 ( 37%) Hash buckets with 5 entries 25662 ( 12%) Hash buckets with 6 entries 21611 ( 12%) Hash buckets with 7 entries 3103 ( 2%) Hash buckets with 8 entries 10076 ( 7%) Hash buckets with 9 entries 535 ( 0%) Hash buckets with 10 entries 35 ( 0%) Hash buckets with 11 entries 2 ( 0%)
Phase 5 - rebuild AG headers and trees...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- reset superblock...libxfs_bcache: 0x6814c0
Max supported entries = 4359824
Max utilized entries = 1052776
Active entries = 1052776
Hash table size = 544978
Hits = 649707
Misses = 1052776
Hit ratio = 38.16
MRU 0 entries = 1052663 ( 99%)
MRU 1 entries = 0 ( 0%) MRU 2 entries = 0 ( 0%) MRU 3 entries = 0 ( 0%) MRU 4 entries = 0 ( 0%) MRU 5 entries = 0 ( 0%) MRU 6 entries = 0 ( 0%) MRU 7 entries = 0 ( 0%) MRU 8 entries = 0 ( 0%) MRU 9 entries = 0 ( 0%) MRU 10 entries = 0 ( 0%) MRU 11 entries = 3 ( 0%) MRU 12 entries = 110 ( 0%) MRU 13 entries = 0 ( 0%) MRU 14 entries = 0 ( 0%) MRU 15 entries = 0 ( 0%) Hash buckets with 0 entries 206666 ( 0%) Hash buckets with 1 entries 86129 ( 8%) Hash buckets with 2 entries 71585 ( 13%) Hash buckets with 3 entries 20311 ( 5%) Hash buckets with 4 entries 99263 ( 37%) Hash buckets with 5 entries 25662 ( 12%) Hash buckets with 6 entries 21611 ( 12%) Hash buckets with 7 entries 3103 ( 2%) Hash buckets with 8 entries 10076 ( 7%) Hash buckets with 9 entries 535 ( 0%) Hash buckets with 10 entries 35 ( 0%) Hash buckets with 11 entries 2 ( 0%)
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- agno = 21
- agno = 22
- agno = 23
- agno = 24
- agno = 25
- agno = 26
- agno = 27
- agno = 28
- agno = 29
- agno = 30
- agno = 31
- traversal finished ...
- moving disconnected inodes to lost+found ...libxfs_bcache: 0x6814c0
Max supported entries = 4359824
Max utilized entries = 1052807
Active entries = 1052807
Hash table size = 544978
Hits = 1854913
Misses = 1052807
Hit ratio = 63.79
MRU 0 entries = 1016054 ( 96%)
MRU 1 entries = 0 ( 0%) MRU 2 entries = 0 ( 0%) MRU 3 entries = 3 ( 0%) MRU 4 entries = 141 ( 0%) MRU 5 entries = 0 ( 0%) MRU 6 entries = 0 ( 0%) MRU 7 entries = 11363 ( 1%) MRU 8 entries = 25246 ( 2%) MRU 9 entries = 0 ( 0%) MRU 10 entries = 0 ( 0%) MRU 11 entries = 0 ( 0%) MRU 12 entries = 0 ( 0%) MRU 13 entries = 0 ( 0%) MRU 14 entries = 0 ( 0%) MRU 15 entries = 0 ( 0%) Hash buckets with 0 entries 206656 ( 0%) Hash buckets with 1 entries 86133 ( 8%) Hash buckets with 2 entries 71589 ( 13%) Hash buckets with 3 entries 20311 ( 5%) Hash buckets with 4 entries 99257 ( 37%) Hash buckets with 5 entries 25667 ( 12%) Hash buckets with 6 entries 21614 ( 12%) Hash buckets with 7 entries 3103 ( 2%) Hash buckets with 8 entries 10076 ( 7%) Hash buckets with 9 entries 535 ( 0%) Hash buckets with 10 entries 35 ( 0%) Hash buckets with 11 entries 2 ( 0%)
Phase 7 - verify and correct link counts...libxfs_bcache: 0x6814c0
Max supported entries = 4359824
Max utilized entries = 1052807
Active entries = 1052807
Hash table size = 544978
Hits = 1854913
Misses = 1052807
Hit ratio = 63.79
MRU 0 entries = 1016054 ( 96%)
MRU 1 entries = 0 ( 0%) MRU 2 entries = 0 ( 0%) MRU 3 entries = 3 ( 0%) MRU 4 entries = 141 ( 0%) MRU 5 entries = 0 ( 0%) MRU 6 entries = 0 ( 0%) MRU 7 entries = 11363 ( 1%) MRU 8 entries = 25246 ( 2%) MRU 9 entries = 0 ( 0%) MRU 10 entries = 0 ( 0%) MRU 11 entries = 0 ( 0%) MRU 12 entries = 0 ( 0%) MRU 13 entries = 0 ( 0%) MRU 14 entries = 0 ( 0%) MRU 15 entries = 0 ( 0%) Hash buckets with 0 entries 206656 ( 0%) Hash buckets with 1 entries 86133 ( 8%) Hash buckets with 2 entries 71589 ( 13%) Hash buckets with 3 entries 20311 ( 5%) Hash buckets with 4 entries 99257 ( 37%) Hash buckets with 5 entries 25667 ( 12%) Hash buckets with 6 entries 21614 ( 12%) Hash buckets with 7 entries 3103 ( 2%) Hash buckets with 8 entries 10076 ( 7%) Hash buckets with 9 entries 535 ( 0%) Hash buckets with 10 entries 35 ( 0%) Hash buckets with 11 entries 2 ( 0%)
XFS_REPAIR Summary Fri Feb 3 09:00:02 2012
Phase Start End Duration
Phase 1: 02/03 08:53:37 02/03 08:53:37
Phase 2: 02/03 08:53:37 02/03 08:53:38 1 second
Phase 3: 02/03 08:53:38 02/03 08:59:16 5 minutes, 38 seconds
Phase 4: 02/03 08:59:16 02/03 08:59:54 38 seconds
Phase 5: 02/03 08:59:54 02/03 08:59:55 1 second
Phase 6: 02/03 08:59:55 02/03 09:00:01 6 seconds
Phase 7: 02/03 09:00:01 02/03 09:00:01
Total run time: 6 minutes, 24 seconds
done
Signature · 14 lines
Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: [email protected] web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · Joe Landman
XFS file system, built using a 2.6.32.41 kernel. External journals.
repair the file system, mount, turn on gluster (3.2.5), do a find, and get a corruption. Actually it looks like
xfs_repairdoesn't fix the problem.Rinse repeat, reproduce. Works (procedure to generate the corruption) in 2.6.32.41.scalable, 2.6.39.1, 3.2.2
[PATCH v2] repair: update extent count after zapping duplicate blocks
which was sent to the list yesterday should make xfs_repair handle this case.
How do you reproduce the corruption? Just start gluster on a fresh fs an do a find? Any chance you could share command lines used?
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · Christoph Hellwig
Quoted 8 lines · Joe Landman
XFS file system, built using a 2.6.32.41 kernel. External journals.
repair the file system, mount, turn on gluster (3.2.5), do a find, and get a corruption. Actually it looks like
xfs_repairdoesn't fix the problem.Rinse repeat, reproduce. Works (procedure to generate the corruption) in 2.6.32.41.scalable, 2.6.39.1, 3.2.2
Quoted 7 lines
[PATCH v2] repair: update extent count after zapping duplicate blocks
which was sent to the list yesterday should make
xfs_repairhandle this case.How do you reproduce the corruption? Just start gluster on a fresh fs an do a find? Any chance you could share command lines used?
It looks like the corruption was just never repaired by xfs_repair. I just followed the notes here: http://xfs.org/index.php/XFS_FAQ#Q:_How_to_get_around_a_bad_inode_repair_is_unable_to_clean_up and got it operational again. I first blockget -n -i ### to figure out the mapping and let the user know.
I'll grab the patch and apply it. If this comes up again, I'll report.
To summarize, this was a corruption that xfs_repair didn't fix, so fixing it by hand (and rerunning xfs_repair) did.
Thanks!
Joe
Signature · 14 lines
Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: [email protected] web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs