Skip to contentSkip to conversation

xfs_efi_item slab corruption. (v3.9-10936-g51a26ae)

DISCUSSION Dave Jones 4 participants 18 messages May 7, 2013 – May 12, 2013 XFS filesystem

lore
Share & tools
lore

Dave Jones

author
#0 lore raw

started compiling a kernel, and then...

[  172.233200] =============================================================================
[  172.233205] BUG xfs_efi_item (Not tainted): Poison overwritten
[  172.233207] -----------------------------------------------------------------------------
[  172.233210] Disabling lock debugging due to kernel taint
[  172.233213] INFO: 0xffff8800aaac4ea8-0xffff8800aaac4ea8. First byte 0x6a instead of 0x6b
[  172.233235] INFO: Allocated in kmem_zone_alloc+0x67/0xf0 [xfs] age=29290 cpu=1 pid=1357
[  172.233239] 	__slab_alloc+0x468/0x52c
[  172.233243] 	kmem_cache_alloc+0x2b4/0x320
[  172.233256] 	kmem_zone_alloc+0x67/0xf0 [xfs]
[  172.233269] 	kmem_zone_zalloc+0x14/0x40 [xfs]
[  172.233287] 	xfs_efi_init+0x41/0xa0 [xfs]
[  172.233305] 	xfs_trans_get_efi+0x58/0x90 [xfs]
[  172.233320] 	xfs_bmap_finish+0x76/0x1b0 [xfs]
[  172.233338] 	xfs_itruncate_extents+0x2cd/0x610 [xfs]
[  172.233353] 	xfs_inactive+0x401/0x530 [xfs]
[  172.233367] 	xfs_fs_evict_inode+0x8c/0x1b0 [xfs]
[  172.233370] 	evict+0xa3/0x1a0
[  172.233372] 	iput+0xf5/0x180
[  172.233374] 	dput+0x208/0x2f0
[  172.233378] 	SYSC_renameat+0x3be/0x430
[  172.233380] 	SyS_renameat+0xe/0x10
[  172.233383] 	SyS_rename+0x1b/0x20
[  172.233399] INFO: Freed in xfs_efi_item_free+0x21/0x40 [xfs] age=27303 cpu=1 pid=177
[  172.233402] 	__slab_free+0x41/0x39f
[  172.233405] 	kmem_cache_free+0x326/0x370
[  172.233420] 	xfs_efi_item_free+0x21/0x40 [xfs]
[  172.233435] 	__xfs_efi_release+0x4e/0x60 [xfs]
[  172.233449] 	xfs_efi_release+0x50/0x70 [xfs]
[  172.233463] 	xfs_efd_item_committed+0x22/0x40 [xfs]
[  172.233479] 	xfs_trans_committed_bulk+0xcf/0x290 [xfs]
[  172.233494] 	xlog_cil_committed+0x37/0x110 [xfs]
[  172.233510] 	xlog_state_do_callback+0x1b8/0x3f0 [xfs]
[  172.233525] 	xlog_state_done_syncing+0xf2/0x110 [xfs]
[  172.233539] 	xlog_iodone+0x87/0x110 [xfs]
[  172.233550] 	xfs_buf_iodone_work+0x5e/0xd0 [xfs]
[  172.233554] 	process_one_work+0x211/0x700
[  172.233556] 	worker_thread+0x11d/0x3a0
[  172.233559] 	kthread+0xed/0x100
[  172.233562] 	ret_from_fork+0x7c/0xb0
[  172.233565] INFO: Slab 0xffffea0002aab100 objects=22 used=22 fp=0x          (null) flags=0x20000000004080
[  172.233567] INFO: Object 0xffff8800aaac4e38 @offset=3640 fp=0xffff8800aaac7058
[  172.233570] Bytes b4 ffff8800aaac4e28: 07 a2 fd ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
[  172.233573] Object ffff8800aaac4e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233575] Object ffff8800aaac4e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233577] Object ffff8800aaac4e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233579] Object ffff8800aaac4e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233581] Object ffff8800aaac4e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233583] Object ffff8800aaac4e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233586] Object ffff8800aaac4e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233588] Object ffff8800aaac4ea8: 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  jkkkkkkkkkkkkkkk
[  172.233590] Object ffff8800aaac4eb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233592] Object ffff8800aaac4ec8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233594] Object ffff8800aaac4ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233597] Object ffff8800aaac4ee8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233599] Object ffff8800aaac4ef8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233601] Object ffff8800aaac4f08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233603] Object ffff8800aaac4f18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233605] Object ffff8800aaac4f28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233608] Object ffff8800aaac4f38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233610] Object ffff8800aaac4f48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233612] Object ffff8800aaac4f58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233614] Object ffff8800aaac4f68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233616] Object ffff8800aaac4f78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233618] Object ffff8800aaac4f88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233620] Object ffff8800aaac4f98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233622] Object ffff8800aaac4fa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233625] Object ffff8800aaac4fb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
[  172.233627] Redzone ffff8800aaac4fc8: bb bb bb bb bb bb bb bb                          ........
[  172.233629] Padding ffff8800aaac5108: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
[  172.233632] CPU: 1 PID: 1658 Comm: gcc Tainted: G    B        3.9.0+ #1 [loadavg: 0.68 0.43 0.17 4/333 1660]
[  172.233636] Hardware name: LENOVO 2356JK8/2356JK8, BIOS G7ET92WW (2.52 ) 02/18/2013
[  172.233638]  ffffea0002aab100 ffff8800a4b03918 ffffffff815f5588 ffff8800a4b03968
[  172.233644]  ffffffff81185944 0000000000000008 ffff880000000001 ffffffff81078016
[  172.233648]  ffff8800aaac4ea8 ffff8800aaac4ea9 ffff880115937cc0 000000000000006b
[  172.233653] Call Trace:
[  172.233658]  [<ffffffff815f5588>] dump_stack+0x19/0x1b
[  172.233662]  [<ffffffff81185944>] print_trailer+0x154/0x210
[  172.233666]  [<ffffffff81078016>] ? perf_trace_sched_process_template+0xe6/0x100
[  172.233669]  [<ffffffff81185b3f>] check_bytes_and_report+0xcf/0x110
[  172.233673]  [<ffffffff81187037>] check_object+0x1e7/0x260
[  172.233676]  [<ffffffff811864f7>] ? check_slab+0x87/0x130
[  172.233691]  [<ffffffffa0543fa7>] ? kmem_zone_alloc+0x67/0xf0 [xfs]
[  172.233694]  [<ffffffff815f2ea3>] alloc_debug_processing+0x76/0x118
[  172.233697]  [<ffffffff815f3b8d>] __slab_alloc+0x468/0x52c
[  172.233716]  [<ffffffffa0592787>] ? xfs_iext_bno_to_ext+0x97/0x180 [xfs]
[  172.233731]  [<ffffffffa0543fa7>] ? kmem_zone_alloc+0x67/0xf0 [xfs]
[  172.233735]  [<ffffffff8118a754>] kmem_cache_alloc+0x2b4/0x320
[  172.233748]  [<ffffffffa0543fa7>] ? kmem_zone_alloc+0x67/0xf0 [xfs]
[  172.233762]  [<ffffffffa0543fa7>] kmem_zone_alloc+0x67/0xf0 [xfs]
[  172.233776]  [<ffffffffa0544044>] kmem_zone_zalloc+0x14/0x40 [xfs]
[  172.233792]  [<ffffffffa05ae031>] xfs_efi_init+0x41/0xa0 [xfs]
[  172.233809]  [<ffffffffa05b32b8>] xfs_trans_get_efi+0x58/0x90 [xfs]
[  172.233825]  [<ffffffffa055a0a6>] xfs_bmap_finish+0x76/0x1b0 [xfs]
[  172.233844]  [<ffffffffa058ef0d>] xfs_itruncate_extents+0x2cd/0x610 [xfs]
[  172.233859]  [<ffffffffa0540e41>] xfs_inactive+0x401/0x530 [xfs]
[  172.233874]  [<ffffffffa053dbec>] xfs_fs_evict_inode+0x8c/0x1b0 [xfs]
[  172.233877]  [<ffffffff811c1033>] evict+0xa3/0x1a0
[  172.233880]  [<ffffffff811c19c5>] iput+0xf5/0x180
[  172.233883]  [<ffffffff811bcf18>] dput+0x208/0x2f0
[  172.233887]  [<ffffffff811b43ae>] SYSC_renameat+0x3be/0x430
[  172.233892]  [<ffffffff810ad32e>] ? lock_release_holdtime.part.30+0xee/0x170
[  172.233895]  [<ffffffff8107c18d>] ? get_parent_ip+0xd/0x50
[  172.233899]  [<ffffffff810afe7d>] ? trace_hardirqs_on+0xd/0x10
[  172.233904]  [<ffffffff8100e358>] ? syscall_trace_enter+0x18/0x230
[  172.233907]  [<ffffffff811b67fe>] SyS_renameat+0xe/0x10
[  172.233911]  [<ffffffff811b681b>] SyS_rename+0x1b/0x20
[  172.233914]  [<ffffffff816051d4>] tracesys+0xdd/0xe2
[  172.233917] FIX xfs_efi_item: Restoring 0xffff8800aaac4ea8-0xffff8800aaac4ea8=0x6b
[  172.233919] FIX xfs_efi_item: Marking all objects used
#1 lore raw
Quoted 9 lines · Dave Jones
   172.233570] Bytes b4 ffff8800aaac4e28: 07 a2 fd ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
[  172.233573] Object ffff8800aaac4e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233575] Object ffff8800aaac4e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233577] Object ffff8800aaac4e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233579] Object ffff8800aaac4e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233581] Object ffff8800aaac4e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233583] Object ffff8800aaac4e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233586] Object ffff8800aaac4e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233588] Object ffff8800aaac4ea8: 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  jkkkkkkkkkkkkkkk
                                          ^^

xfs_efi_log_item.efi_refcount being decremented on the xfs_efi_release() CAI Qian had the same thing in his May 6 "3.9.0: XFS rootfs corruption" email.

I have not reproduced it yet.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#2 lore raw
Quoted 10 lines · Mark Tinguely
Quoted 9 lines · Dave Jones
   172.233570] Bytes b4 ffff8800aaac4e28: 07 a2 fd ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
[  172.233573] Object ffff8800aaac4e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233575] Object ffff8800aaac4e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233577] Object ffff8800aaac4e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233579] Object ffff8800aaac4e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233581] Object ffff8800aaac4e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233583] Object ffff8800aaac4e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233586] Object ffff8800aaac4e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233588] Object ffff8800aaac4ea8: 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  jkkkkkkkkkkkkkkk
Quoted 7 lines
                                           ^^

xfs_efi_log_item.efi_refcount being decremented on the xfs_efi_release() CAI Qian had the same thing in his May 6 "3.9.0: XFS rootfs corruption" email.

I have not reproduced it yet.

I've hit it on two different machines today. The good news is that the corruption never makes it onto disk. xfs_repair doesn't pick up anything.

Dave

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
Quoted 17 lines · Dave Jones
Quoted 16 lines · Mark Tinguely
Quoted 9 lines · Dave Jones
    172.233570] Bytes b4 ffff8800aaac4e28: 07 a2 fd ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
 [  172.233573] Object ffff8800aaac4e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 [  172.233575] Object ffff8800aaac4e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 [  172.233577] Object ffff8800aaac4e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 [  172.233579] Object ffff8800aaac4e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 [  172.233581] Object ffff8800aaac4e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 [  172.233583] Object ffff8800aaac4e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 [  172.233586] Object ffff8800aaac4e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 [  172.233588] Object ffff8800aaac4ea8: 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  jkkkkkkkkkkkkkkk
                                            ^^

xfs_efi_log_item.efi_refcount being decremented on the xfs_efi_release() CAI Qian had the same thing in his May 6 "3.9.0: XFS rootfs corruption" email.

Quoted 7 lines
Quoted 1 line

I have not reproduced it yet.

I've hit it on two different machines today. The good news is that the corruption never makes it onto disk. xfs_repair doesn't pick up anything.

Dave

There was a new patch in the efi/efd code that must be misbehaving. You are correct, this is not an on-disk value.

I now have poisoning on and I can see this doing a compile like you suggested. I will ASSERT to see who is doing the decrement after free.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#4 lore raw
Quoted 23 lines · Mark Tinguely
Quoted 22 lines · Dave Jones
Quoted 10 lines · Mark Tinguely
Quoted 9 lines · Dave Jones
    172.233570] Bytes b4 ffff8800aaac4e28: 07 a2 fd ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
 [  172.233573] Object ffff8800aaac4e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 [  172.233575] Object ffff8800aaac4e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 [  172.233577] Object ffff8800aaac4e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 [  172.233579] Object ffff8800aaac4e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 [  172.233581] Object ffff8800aaac4e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 [  172.233583] Object ffff8800aaac4e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 [  172.233586] Object ffff8800aaac4e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 [  172.233588] Object ffff8800aaac4ea8: 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  jkkkkkkkkkkkkkkk
Quoted 7 lines
                                            ^^

xfs_efi_log_item.efi_refcount being decremented on the xfs_efi_release() CAI Qian had the same thing in his May 6 "3.9.0: XFS rootfs corruption" email.

I have not reproduced it yet.

I've hit it on two different machines today. The good news is that the corruption never makes it onto disk. xfs_repair doesn't pick up anything.

Quoted 7 lines
Quoted 2 lines

Dave

There was a new patch in the efi/efd code that must be misbehaving. You are correct, this is not an on-disk value.

I now have poisoning on and I can see this doing a compile like you suggested. I will ASSERT to see who is doing the decrement after free.

I can hit this almost instantly with fsx. I'll do a bisect, though it sounds like you already have a suspect.

Dave

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw
Quoted 29 lines · Dave Jones
Quoted 28 lines · Mark Tinguely
Quoted 17 lines · Dave Jones
Quoted 16 lines · Mark Tinguely
Quoted 9 lines · Dave Jones
     172.233570] Bytes b4 ffff8800aaac4e28: 07 a2 fd ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
  [  172.233573] Object ffff8800aaac4e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
  [  172.233575] Object ffff8800aaac4e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
  [  172.233577] Object ffff8800aaac4e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
  [  172.233579] Object ffff8800aaac4e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
  [  172.233581] Object ffff8800aaac4e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
  [  172.233583] Object ffff8800aaac4e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
  [  172.233586] Object ffff8800aaac4e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
  [  172.233588] Object ffff8800aaac4ea8: 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  jkkkkkkkkkkkkkkk
                                             ^^

xfs_efi_log_item.efi_refcount being decremented on the xfs_efi_release() CAI Qian had the same thing in his May 6 "3.9.0: XFS rootfs corruption" email.

Quoted 7 lines
Quoted 1 line

I have not reproduced it yet.

I've hit it on two different machines today. The good news is that the corruption never makes it onto disk. xfs_repair doesn't pick up anything.

Dave

There was a new patch in the efi/efd code that must be misbehaving. You are correct, this is not an on-disk value.

Quoted 8 lines
Quoted 2 lines

I now have poisoning on and I can see this doing a compile like you suggested. I will ASSERT to see who is doing the decrement after free.

I can hit this almost instantly with fsx. I'll do a bisect, though it sounds like you already have a suspect.

Dave

If you want to try kmem debug of Linux 3.8 that would help.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#6 lore raw
Quoted 5 lines · Mark Tinguely
Quoted 3 lines

I can hit this almost instantly with fsx. I'll do a bisect, though it sounds like you already have a suspect.

If you want to try kmem debug of Linux 3.8 that would help.

I'm not sure what that is.

Dave

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#7 lore raw
Quoted 6 lines · Dave Jones
Quoted 4 lines · Mark Tinguely
Quoted 3 lines

I can hit this almost instantly with fsx. I'll do a bisect, though it sounds like you already have a suspect.

Quoted 6 lines
Quoted 1 line

If you want to try kmem debug of Linux 3.8 that would help.

I'm not sure what that is.

Dave

Sorry, if you would test Linux 3.8 with "CONFIG_DEBUG_SLAB=y".

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#8 lore raw
Quoted 7 lines · Mark Tinguely
Quoted 6 lines · Dave Jones
Quoted 4 lines · Mark Tinguely
Quoted 3 lines

I can hit this almost instantly with fsx. I'll do a bisect, though it sounds like you already have a suspect.

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

If you want to try kmem debug of Linux 3.8 that would help.

I'm not sure what that is.

Sorry, if you would test Linux 3.8 with "CONFIG_DEBUG_SLAB=y".

Ah, done that. (I pretty much always run with it).

This is something new. Even 3.9 was fine. It's only since the recent xfs merge.

Dave

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#9 lore raw
Quoted 12 lines · Dave Jones
Quoted 11 lines · Mark Tinguely
Quoted 9 lines · Dave Jones
Quoted 5 lines · Mark Tinguely
Quoted 3 lines

I can hit this almost instantly with fsx. I'll do a bisect, though it sounds like you already have a suspect.

If you want to try kmem debug of Linux 3.8 that would help.

I'm not sure what that is.

Quoted 9 lines
Quoted 1 line

Sorry, if you would test Linux 3.8 with "CONFIG_DEBUG_SLAB=y".

Ah, done that. (I pretty much always run with it).

This is something new. Even 3.9 was fine. It's only since the recent xfs merge.

Dave

git revert 666d644cd72a9ec58b353209ff191d7430f3b357

until it gets fixed.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#10 lore raw
Quoted 17 lines · Mark Tinguely
Quoted 16 lines · Dave Jones
Quoted 7 lines · Mark Tinguely
Quoted 6 lines · Dave Jones
Quoted 4 lines · Mark Tinguely
Quoted 3 lines

I can hit this almost instantly with fsx. I'll do a bisect, though it sounds like you already have a suspect.

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

If you want to try kmem debug of Linux 3.8 that would help.

I'm not sure what that is.

Sorry, if you would test Linux 3.8 with "CONFIG_DEBUG_SLAB=y".

Ah, done that. (I pretty much always run with it).

Quoted 7 lines
Quoted 5 lines

This is something new. Even 3.9 was fine. It's only since the recent xfs merge.

Dave

git revert 666d644cd72a9ec58b353209ff191d7430f3b357

That won't prevent the use after free. That commit fixed a problem that could lead to a use after free, but what we are seeing here is that it has ultimately exposed a previously unknown issue that causes the use after free.

Basically what is happening is that there are two commits for the EFD being processed, when there should only be one. I'm not sure how this is happening yet, but these three traces came out from my debug sequentially when running generic/006:

__xfs_efi_release 0xffff88006c4d7058 2
Pid: 4756, comm: kworker/1:1H Tainted: G B 3.9.0-rc8-dgc+ #592

Call Trace:
 [<ffffffff814e43a5>] __xfs_efi_release+0x35/0x80
 [<ffffffff814e444d>] xfs_efi_item_unpin+0x5d/0x70
 [<ffffffff814dbb88>] xfs_trans_committed_bulk+0x248/0x2e0
 [<ffffffff814e18db>] xlog_cil_committed+0x3b/0x140
 [<ffffffff814ddfae>] xlog_state_do_callback+0x19e/0x3a0

.....

That's the EFI being unpinned and it's reference going away during transaction commit completion procesing.

__xfs_efi_release 0xffff88006c4d7058 1
Pid: 4756, comm: kworker/1:1H Tainted: G B 3.9.0-rc8-dgc+ #592

Call Trace:
 [<ffffffff814e43a5>] __xfs_efi_release+0x35/0x80
 [<ffffffff814e4620>] xfs_efi_release+0x60/0x80
 [<ffffffff814e46a6>] xfs_efd_item_committed+0x26/0x40
 [<ffffffff814dba21>] xfs_trans_committed_bulk+0xe1/0x2e0
 [<ffffffff814e18db>] xlog_cil_committed+0x3b/0x140
 [<ffffffff814ddfae>] xlog_state_do_callback+0x19e/0x3a0

.....

That's the EFD releasing the EFI, which drops it's reference count and frees the both the EFI and the EFD.

__xfs_efi_release 0xffff88006c4d7058 1802201963
Pid: 4756, comm: kworker/1:1H Tainted: G B 3.9.0-rc8-dgc+ #592

Call Trace:
 [<ffffffff814e43a5>] __xfs_efi_release+0x35/0x80
 [<ffffffff814e4630>] xfs_efi_release+0x70/0x80
 [<ffffffff814e46a6>] xfs_efd_item_committed+0x26/0x40
 [<ffffffff814dba21>] xfs_trans_committed_bulk+0xe1/0x2e0
 [<ffffffff814e18db>] xlog_cil_committed+0x3b/0x140

.....

And there's the EFI being released by an EFD after it has been freed. It's definitely been freed because 1802201963 = 0x6b6b6b6b....

Like I said, I don't understand how this is happening yet, but just seeing this behaviour explains an awful lot about why we've had long term issues with this code....

FWIW, I used to run with slab debugging turned on for all my debug builds. I switched my .config from SLAB to SLUB a few months ago, and didn't realise this silently turned of memory poisoning. That's how it slipped through my testing....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#11 lore raw
Quoted 31 lines · Dave Chinner
Quoted 17 lines · Mark Tinguely
Quoted 16 lines · Dave Jones
Quoted 7 lines · Mark Tinguely
Quoted 6 lines · Dave Jones
Quoted 4 lines · Mark Tinguely
Quoted 3 lines

I can hit this almost instantly with fsx. I'll do a bisect, though it sounds like you already have a suspect.

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

If you want to try kmem debug of Linux 3.8 that would help.

I'm not sure what that is.

Sorry, if you would test Linux 3.8 with "CONFIG_DEBUG_SLAB=y".

Ah, done that. (I pretty much always run with it).

Quoted 7 lines
Quoted 5 lines

This is something new. Even 3.9 was fine. It's only since the recent xfs merge.

Dave

git revert 666d644cd72a9ec58b353209ff191d7430f3b357

That won't prevent the use after free. That commit fixed a problem that could lead to a use after free, but what we are seeing here is that it has ultimately exposed a previously unknown issue that causes the use after free.

Quoted 5 lines

Basically what is happening is that there are two commits for the EFD being processed, when there should only be one. I'm not sure how this is happening yet, but these three traces came out from my debug sequentially when running generic/006:

Sorry for the misleading statement. Yes, I agree that patch is a good thing. I meant that Dave and only Dave revert it and only to test if that patch was the change that caused the new symptom - which we know now that it is.

I added some asserts and did not learn anything new except where the efi item was already freed.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#12 lore raw
Quoted 37 lines · Mark Tinguely
Quoted 31 lines · Dave Chinner
Quoted 17 lines · Mark Tinguely
Quoted 16 lines · Dave Jones
Quoted 7 lines · Mark Tinguely
Quoted 6 lines · Dave Jones
Quoted 4 lines · Mark Tinguely
Quoted 3 lines

I can hit this almost instantly with fsx. I'll do a bisect, though it sounds like you already have a suspect.

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

If you want to try kmem debug of Linux 3.8 that would help.

I'm not sure what that is.

Sorry, if you would test Linux 3.8 with "CONFIG_DEBUG_SLAB=y".

Ah, done that. (I pretty much always run with it).

Quoted 7 lines
Quoted 5 lines

This is something new. Even 3.9 was fine. It's only since the recent xfs merge.

Dave

git revert 666d644cd72a9ec58b353209ff191d7430f3b357

That won't prevent the use after free. That commit fixed a problem that could lead to a use after free, but what we are seeing here is that it has ultimately exposed a previously unknown issue that causes the use after free.

Quoted 4 lines

Basically what is happening is that there are two commits for the EFD being processed, when there should only be one. I'm not sure how this is happening yet, but these three traces came out from my debug sequentially when running generic/006:

Quoted 4 lines

Sorry for the misleading statement. Yes, I agree that patch is a good thing. I meant that Dave and only Dave revert it and only to test if that patch was the change that caused the new symptom - which we know now that it is.

Sure, I realise that, and it turns out I'm wrong - it is a bug in commit 666d644cd. Poisoning turns a "will probably never occur" problem into an instant reproducer, because it sets a bit in the efi structure that is normally zero when the EFI is freed and hence triggers a second free of the EFI when reading it after the first free....

Dave, the patch below should fix the problem.

Cheers,

Dave.

Signature · 17 lines
Dave Chinner
[email protected]

xfs: Don't reference the EFI after it is freed

From: Dave Chinner <[email protected]>

Checking the EFI for whether it is being released from recovery
after we've already released the known active reference is a mistake
worthy of a brown paper bag. Fix the (now) obvious use after free
that it can cause.

Reported-by: Dave Jones <[email protected]>
Signed-off-by: Dave Chinner <[email protected]>
---
 fs/xfs/xfs_extfree_item.c |   14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
fs/xfs/xfs_extfree_item.c +13 -1
index c0f3750..98c437d 100644
@@ -305,10 +305,22 @@ xfs_efi_release(xfs_efi_log_item_t	*efip,
 {
 	ASSERT(atomic_read(&efip->efi_next_extent) >= nextents);
 	if (atomic_sub_and_test(nextents, &efip->efi_next_extent)) {
+		int recovered;
+
+		/*
+		 * __xfs_efi_release() can release the last reference to the EFI
+		 * and free it, so it is unsafe to reference it after we've
+		 * released the reference. The only case this is safe to do is
+		 * if we are in recovery and the XFS_EFI_RECOVERED bit is set,
+		 * meaning that we have two references to release. Check the
+		 * recovered bit before the initial release, as we cannot
+		 * reliably check it afterwards.
+		 */
+		recovered = test_bit(XFS_EFI_RECOVERED, &efip->efi_flags);
 		__xfs_efi_release(efip);
 
 		/* recovery needs us to drop the EFI reference, too */
-		if (test_bit(XFS_EFI_RECOVERED, &efip->efi_flags))
+		if (recovered)
 			__xfs_efi_release(efip);
 	}
 }

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

author
#13 lore raw
Quoted 10 lines · Dave Chinner

Dave, the patch below should fix the problem.

xfs: Don't reference the EFI after it is freed

From: Dave Chinner <[email protected]>

Checking the EFI for whether it is being released from recovery after we've already released the known active reference is a mistake worthy of a brown paper bag. Fix the (now) obvious use after free that it can cause.

Yup, that works for me.

thanks,

Dave

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by
#14 lore raw
In reply to Dave Chinner

On 05/07/13 18:54, Dave Chinner wrote:

Checking the EFI for whether it is being released from recovery after we've already released the known active reference is a mistake worthy of a brown paper bag. Fix the (now) obvious use after free that it can cause.

---
 fs/xfs/xfs_extfree_item.c |   14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

Looks good.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#15 lore raw
Quoted 10 lines · Mark Tinguely

On 05/07/13 18:54, Dave Chinner wrote:

Checking the EFI for whether it is being released from recovery after we've already released the known active reference is a mistake worthy of a brown paper bag. Fix the (now) obvious use after free that it can cause.

Quoted 7 lines
---
 fs/xfs/xfs_extfree_item.c |   14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

Looks good.

Zach pointed out that the fix is much more complex than it needs to be. I'll respin the patch and resend it later today.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#16 lore raw
In reply to Dave Jones
Quoted 2 lines · Dave Jones

started compiling a kernel, and then...

I've bisected this one.
commit 666d644cd72a9ec58b353209ff191d7430f3b357

Author: Dave Chinner <[email protected]>
Date:   Wed Apr 3 14:09:21 2013 +1100

xfs: don't free EFIs before the EFDs are committed

#xfstests ./check generic/007 generic/007 generic/007

Quoted 117 lines
[  172.233200] =============================================================================
[  172.233205] BUG xfs_efi_item (Not tainted): Poison overwritten
[  172.233207] -----------------------------------------------------------------------------
[  172.233210] Disabling lock debugging due to kernel taint
[  172.233213] INFO: 0xffff8800aaac4ea8-0xffff8800aaac4ea8. First byte 0x6a instead of 0x6b
[  172.233235] INFO: Allocated in kmem_zone_alloc+0x67/0xf0 [xfs] age=29290 cpu=1 pid=1357
[  172.233239] 	__slab_alloc+0x468/0x52c
[  172.233243] 	kmem_cache_alloc+0x2b4/0x320
[  172.233256] 	kmem_zone_alloc+0x67/0xf0 [xfs]
[  172.233269] 	kmem_zone_zalloc+0x14/0x40 [xfs]
[  172.233287] 	xfs_efi_init+0x41/0xa0 [xfs]
[  172.233305] 	xfs_trans_get_efi+0x58/0x90 [xfs]
[  172.233320] 	xfs_bmap_finish+0x76/0x1b0 [xfs]
[  172.233338] 	xfs_itruncate_extents+0x2cd/0x610 [xfs]
[  172.233353] 	xfs_inactive+0x401/0x530 [xfs]
[  172.233367] 	xfs_fs_evict_inode+0x8c/0x1b0 [xfs]
[  172.233370] 	evict+0xa3/0x1a0
[  172.233372] 	iput+0xf5/0x180
[  172.233374] 	dput+0x208/0x2f0
[  172.233378] 	SYSC_renameat+0x3be/0x430
[  172.233380] 	SyS_renameat+0xe/0x10
[  172.233383] 	SyS_rename+0x1b/0x20
[  172.233399] INFO: Freed in xfs_efi_item_free+0x21/0x40 [xfs] age=27303 cpu=1 pid=177
[  172.233402] 	__slab_free+0x41/0x39f
[  172.233405] 	kmem_cache_free+0x326/0x370
[  172.233420] 	xfs_efi_item_free+0x21/0x40 [xfs]
[  172.233435] 	__xfs_efi_release+0x4e/0x60 [xfs]
[  172.233449] 	xfs_efi_release+0x50/0x70 [xfs]
[  172.233463] 	xfs_efd_item_committed+0x22/0x40 [xfs]
[  172.233479] 	xfs_trans_committed_bulk+0xcf/0x290 [xfs]
[  172.233494] 	xlog_cil_committed+0x37/0x110 [xfs]
[  172.233510] 	xlog_state_do_callback+0x1b8/0x3f0 [xfs]
[  172.233525] 	xlog_state_done_syncing+0xf2/0x110 [xfs]
[  172.233539] 	xlog_iodone+0x87/0x110 [xfs]
[  172.233550] 	xfs_buf_iodone_work+0x5e/0xd0 [xfs]
[  172.233554] 	process_one_work+0x211/0x700
[  172.233556] 	worker_thread+0x11d/0x3a0
[  172.233559] 	kthread+0xed/0x100
[  172.233562] 	ret_from_fork+0x7c/0xb0
[  172.233565] INFO: Slab 0xffffea0002aab100 objects=22 used=22 fp=0x          (null) flags=0x20000000004080
[  172.233567] INFO: Object 0xffff8800aaac4e38 @offset=3640 fp=0xffff8800aaac7058
[  172.233570] Bytes b4 ffff8800aaac4e28: 07 a2 fd ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
[  172.233573] Object ffff8800aaac4e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233575] Object ffff8800aaac4e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233577] Object ffff8800aaac4e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233579] Object ffff8800aaac4e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233581] Object ffff8800aaac4e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233583] Object ffff8800aaac4e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233586] Object ffff8800aaac4e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233588] Object ffff8800aaac4ea8: 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  jkkkkkkkkkkkkkkk
[  172.233590] Object ffff8800aaac4eb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233592] Object ffff8800aaac4ec8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233594] Object ffff8800aaac4ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233597] Object ffff8800aaac4ee8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233599] Object ffff8800aaac4ef8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233601] Object ffff8800aaac4f08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233603] Object ffff8800aaac4f18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233605] Object ffff8800aaac4f28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233608] Object ffff8800aaac4f38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233610] Object ffff8800aaac4f48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233612] Object ffff8800aaac4f58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233614] Object ffff8800aaac4f68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233616] Object ffff8800aaac4f78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233618] Object ffff8800aaac4f88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233620] Object ffff8800aaac4f98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233622] Object ffff8800aaac4fa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[  172.233625] Object ffff8800aaac4fb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
[  172.233627] Redzone ffff8800aaac4fc8: bb bb bb bb bb bb bb bb                          ........
[  172.233629] Padding ffff8800aaac5108: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
[  172.233632] CPU: 1 PID: 1658 Comm: gcc Tainted: G    B        3.9.0+ #1 [loadavg: 0.68 0.43 0.17 4/333 1660]
[  172.233636] Hardware name: LENOVO 2356JK8/2356JK8, BIOS G7ET92WW (2.52 ) 02/18/2013
[  172.233638]  ffffea0002aab100 ffff8800a4b03918 ffffffff815f5588 ffff8800a4b03968
[  172.233644]  ffffffff81185944 0000000000000008 ffff880000000001 ffffffff81078016
[  172.233648]  ffff8800aaac4ea8 ffff8800aaac4ea9 ffff880115937cc0 000000000000006b
[  172.233653] Call Trace:
[  172.233658]  [<ffffffff815f5588>] dump_stack+0x19/0x1b
[  172.233662]  [<ffffffff81185944>] print_trailer+0x154/0x210
[  172.233666]  [<ffffffff81078016>] ? perf_trace_sched_process_template+0xe6/0x100
[  172.233669]  [<ffffffff81185b3f>] check_bytes_and_report+0xcf/0x110
[  172.233673]  [<ffffffff81187037>] check_object+0x1e7/0x260
[  172.233676]  [<ffffffff811864f7>] ? check_slab+0x87/0x130
[  172.233691]  [<ffffffffa0543fa7>] ? kmem_zone_alloc+0x67/0xf0 [xfs]
[  172.233694]  [<ffffffff815f2ea3>] alloc_debug_processing+0x76/0x118
[  172.233697]  [<ffffffff815f3b8d>] __slab_alloc+0x468/0x52c
[  172.233716]  [<ffffffffa0592787>] ? xfs_iext_bno_to_ext+0x97/0x180 [xfs]
[  172.233731]  [<ffffffffa0543fa7>] ? kmem_zone_alloc+0x67/0xf0 [xfs]
[  172.233735]  [<ffffffff8118a754>] kmem_cache_alloc+0x2b4/0x320
[  172.233748]  [<ffffffffa0543fa7>] ? kmem_zone_alloc+0x67/0xf0 [xfs]
[  172.233762]  [<ffffffffa0543fa7>] kmem_zone_alloc+0x67/0xf0 [xfs]
[  172.233776]  [<ffffffffa0544044>] kmem_zone_zalloc+0x14/0x40 [xfs]
[  172.233792]  [<ffffffffa05ae031>] xfs_efi_init+0x41/0xa0 [xfs]
[  172.233809]  [<ffffffffa05b32b8>] xfs_trans_get_efi+0x58/0x90 [xfs]
[  172.233825]  [<ffffffffa055a0a6>] xfs_bmap_finish+0x76/0x1b0 [xfs]
[  172.233844]  [<ffffffffa058ef0d>] xfs_itruncate_extents+0x2cd/0x610 [xfs]
[  172.233859]  [<ffffffffa0540e41>] xfs_inactive+0x401/0x530 [xfs]
[  172.233874]  [<ffffffffa053dbec>] xfs_fs_evict_inode+0x8c/0x1b0 [xfs]
[  172.233877]  [<ffffffff811c1033>] evict+0xa3/0x1a0
[  172.233880]  [<ffffffff811c19c5>] iput+0xf5/0x180
[  172.233883]  [<ffffffff811bcf18>] dput+0x208/0x2f0
[  172.233887]  [<ffffffff811b43ae>] SYSC_renameat+0x3be/0x430
[  172.233892]  [<ffffffff810ad32e>] ? lock_release_holdtime.part.30+0xee/0x170
[  172.233895]  [<ffffffff8107c18d>] ? get_parent_ip+0xd/0x50
[  172.233899]  [<ffffffff810afe7d>] ? trace_hardirqs_on+0xd/0x10
[  172.233904]  [<ffffffff8100e358>] ? syscall_trace_enter+0x18/0x230
[  172.233907]  [<ffffffff811b67fe>] SyS_renameat+0xe/0x10
[  172.233911]  [<ffffffff811b681b>] SyS_rename+0x1b/0x20
[  172.233914]  [<ffffffff816051d4>] tracesys+0xdd/0xe2
[  172.233917] FIX xfs_efi_item: Restoring 0xffff8800aaac4ea8-0xffff8800aaac4ea8=0x6b
[  172.233919] FIX xfs_efi_item: Marking all objects used

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#17 lore raw
Quoted 11 lines · Dmitry Monakhov
Quoted 2 lines · Dave Jones

started compiling a kernel, and then...

I've bisected this one.
commit 666d644cd72a9ec58b353209ff191d7430f3b357

Author: Dave Chinner <[email protected]>
Date:   Wed Apr 3 14:09:21 2013 +1100

xfs: don't free EFIs before the EFDs are committed

#xfstests ./check generic/007 generic/007 generic/007

Yuo, we already have a candidate fix for it. Thanks for another independent confirmation of the cause, though.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]