kernel BUG at fs/buffer.c:2886! Linux 3.5.0
Hello
Get this on first write made ( by deliver sending mail to inform of the restart of services )
Home partition (the one receiving the mail) is based on ocfs2 created from drbd block device in primary/primary mode
These drbd devices are based on lvm.
system is running linux-3.5.0, identical symptom with linux 3.3 and 3.2 but working with linux 3.0 kernel
reproduced on two machines ( so different hardware involved on this one software md raid on SATA, on second one areca hardware raid card ) but the 2 machines are the one sharing this partition ( so share the same data )
Jul 27 23:41:41 jupiter2 postfix/qmgr[4400]: D12661C487: from=<[email protected]>, size=388, nrcpt=1 (queue active) Jul 27 23:41:41 jupiter2 postfix/qmgr[4400]: D0AEE1C01F: from=<[email protected]>, size=401, nrcpt=1 (queue active) Jul 27 23:41:41 jupiter2 postfix/qmgr[4400]: DD5D91BFE9: from=<[email protected]>, size=393, nrcpt=1 (queue active) Jul 27 23:41:41 jupiter2 postfix/qmgr[4400]: A705B1BFDC: from=<[email protected]>, size=395, nrcpt=1 (queue active) Jul 27 23:41:41 jupiter2 postfix/qmgr[4400]: B6AB41C48C: from=<[email protected]>, size=388, nrcpt=1 (queue active) Jul 27 23:41:41 jupiter2 postfix/qmgr[4400]: 8DAE11C48F: from=<[email protected]>, size=382, nrcpt=1 (queue active) Jul 27 23:41:41 jupiter2 postfix/qmgr[4400]: 715241C489: from=<[email protected]>, size=380, nrcpt=1 (queue active) Jul 27 23:41:41 jupiter2 postfix/qmgr[4400]: F18601C024: from=<[email protected]>, size=401, nrcpt=1 (queue active) Jul 27 23:41:41 jupiter2 kernel: [ 351.169213] ------------[ cut here ]------------ Jul 27 23:41:41 jupiter2 kernel: [ 351.169261] kernel BUG at fs/buffer.c:2886! Jul 27 23:41:41 jupiter2 kernel: [ 351.169303] invalid opcode: 0000 [#1] SMP Jul 27 23:41:41 jupiter2 kernel: [ 351.169409] CPU 1 Jul 27 23:41:41 jupiter2 kernel: [ 351.169446] Modules linked in: drbd lru_cache Jul 27 23:41:41 jupiter2 kernel: [ 351.169620] Jul 27 23:41:41 jupiter2 kernel: [ 351.169655] Pid: 5783, comm: deliver Not tainted 3.5.0-gentoo #2 HP ProLiant ML150 G3/ML150 G3 Jul 27 23:41:41 jupiter2 kernel: [ 351.169803] RIP: 0010:[<ffffffff81180862>] [<ffffffff81180862>] submit_bh+0x112/0x120 Jul 27 23:41:41 jupiter2 kernel: [ 351.169889] RSP: 0018:ffff8800574c7b38 EFLAGS: 00010246 Jul 27 23:41:41 jupiter2 kernel: [ 351.169932] RAX: 4000000001000004 RBX: ffffea0001791ac0 RCX: 00000003ffffffff Jul 27 23:41:41 jupiter2 kernel: [ 351.169978] RDX: 0000000000000001 RSI: ffffea0001791ac0 RDI: 0000000000000000 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff81346ad0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] R10: dead000000200200 R11: 0000000000000000 R12: 0000000004cc4789 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] R13: 00000003ffffffff R14: 0000000000000000 R15: 0000000000000000 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] FS: 00007ff70e943700(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] CR2: 00007fb4968f6b6c CR3: 000000005fe8a000 CR4: 00000000000007e0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] Process deliver (pid: 5783, threadinfo ffff8800574c6000, task ffff88007b8fad00) Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] Stack: Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] ffffea0001791ac0 0000000000000001 0000000004cc4789 ffffffff81327546 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] 000000005893c5b8 ffffffff8114e541 ffff88007b8fb368 ffff8800574c7c10 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] 0000000000000000 0000000100000000 ffff8800589236f0 ffff88005bac4000 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] Call Trace: Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81327546>] ? ocfs2_read_blocks+0x176/0x6c0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8114e541>] ? T.1552+0x91/0x2b0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81346ad0>] ? ocfs2_find_actor+0x120/0x120 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813464f7>] ? ocfs2_read_inode_block_full+0x37/0x60 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813964ff>] ? ocfs2_fast_symlink_readpage+0x2f/0x160 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81111585>] ? do_read_cache_page+0x85/0x180 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813964d0>] ? ocfs2_fill_super+0x2500/0x2500 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff811116d9>] ? read_cache_page+0x9/0x20 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115c705>] ? page_getlink+0x25/0x80 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115c77b>] ? page_follow_link_light+0x1b/0x30 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8116099b>] ? path_lookupat+0x38b/0x720 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81160d5c>] ? do_path_lookup+0x2c/0xd0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81346f31>] ? ocfs2_inode_revalidate+0x71/0x160 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81161c0c>] ? user_path_at_empty+0x5c/0xb0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8106714a>] ? do_page_fault+0x1aa/0x3c0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81156f2d>] ? cp_new_stat+0x10d/0x120 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81157021>] ? vfs_fstatat+0x41/0x80 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115715f>] ? sys_newstat+0x1f/0x50 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff817ecee2>] ? system_call_fastpath+0x16/0x1b Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] Code: b6 44 24 18 4c 89 e7 83 e0 80 3c 01 19 db e8 76 3f 00 00 f7 d3 83 e3 a1 89 d8 5b 5d 41 5c c3 0f 0b eb fe 0f 0b eb fe 0f 0$ Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] RIP [<ffffffff81180862>] submit_bh+0x112/0x120 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] RSP <ffff8800574c7b38> Jul 27 23:41:41 jupiter2 kernel: [ 351.177405] ---[ end trace e1e88bdf12146104 ]--- Jul 27 23:41:41 jupiter2 kernel: [ 351.177868] deliver (5783) used
greatest stack depth: 3032 bytes left
Regards,
Vincent ETIENNE
Joel Becker
maintainerQuoted 8 lines · Vincent ETIENNE
Hello
Get this on first write made ( by deliver sending mail to inform of the restart of services )
Home partition (the one receiving the mail) is based on ocfs2 created from drbd block device in primary/primary mode
These drbd devices are based on lvm.
Quoted 7 lines
system is running linux-3.5.0, identical symptom with linux 3.3 and 3.2 but working with linux 3.0 kernel
reproduced on two machines ( so different hardware involved on this one software md raid on SATA, on second one areca hardware raid card ) but the 2 machines are the one sharing this partition ( so share the same data )
Hmm. Any chance you can bisect this further?
Quoted 4 lines
Jul 27 23:41:41 jupiter2 kernel: [ 351.169213] ------------[ cut here ]------------ Jul 27 23:41:41 jupiter2 kernel: [ 351.169261] kernel BUG at
fs/buffer.c:2886!
This is: BUG_ON(!buffer_mapped(bh));
in submit_bh().
Quoted 38 lines
Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] Call Trace: Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81327546>] ? ocfs2_read_blocks+0x176/0x6c0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8114e541>] ? T.1552+0x91/0x2b0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81346ad0>] ? ocfs2_find_actor+0x120/0x120 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813464f7>] ? ocfs2_read_inode_block_full+0x37/0x60 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813964ff>] ? ocfs2_fast_symlink_readpage+0x2f/0x160 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81111585>] ? do_read_cache_page+0x85/0x180 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813964d0>] ? ocfs2_fill_super+0x2500/0x2500 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff811116d9>] ? read_cache_page+0x9/0x20 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115c705>] ? page_getlink+0x25/0x80 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115c77b>] ? page_follow_link_light+0x1b/0x30 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8116099b>] ? path_lookupat+0x38b/0x720 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81160d5c>] ? do_path_lookup+0x2c/0xd0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81346f31>] ? ocfs2_inode_revalidate+0x71/0x160 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81161c0c>] ? user_path_at_empty+0x5c/0xb0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8106714a>] ? do_page_fault+0x1aa/0x3c0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81156f2d>] ? cp_new_stat+0x10d/0x120 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81157021>] ? vfs_fstatat+0x41/0x80 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115715f>] ? sys_newstat+0x1f/0x50 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff817ecee2>] ?
Quoted 1 line
system_call_fastpath+0x16/0x1b
This stack trace is from 3.5, because of the location of the
BUG. The call path in the trace suggests the code added by Al's ea022d, but you say it breaks in 3.2 and 3.3 as well. Can you give me a trace from 3.2?
Joel
Signature · 6 lines
Life's Little Instruction Book #139 "Never deprive someone of hope; it might be all they have." http://www.jlbec.org/ [email protected]
HI,
Quoted 9 lines · Joel Becker
Quoted 8 lines · Vincent ETIENNE
Hello
Get this on first write made ( by deliver sending mail to inform of the restart of services )
Home partition (the one receiving the mail) is based on ocfs2 created from drbd block device in primary/primary mode
These drbd devices are based on lvm.
Quoted 8 lines
Quoted 7 lines
system is running linux-3.5.0, identical symptom with linux 3.3 and 3.2 but working with linux 3.0 kernel
reproduced on two machines ( so different hardware involved on this one software md raid on SATA, on second one areca hardware raid card ) but the 2 machines are the one sharing this partition ( so share the same data )
Hmm. Any chance you can bisect this further?
Will try to. Will take a few days as the server is in production ( but used as backup so...)
Quoted 50 lines
Quoted 4 lines
Jul 27 23:41:41 jupiter2 kernel: [ 351.169213] ------------[ cut here ]------------ Jul 27 23:41:41 jupiter2 kernel: [ 351.169261] kernel BUG at
fs/buffer.c:2886!This is: BUG_ON(!buffer_mapped(bh));in
submit_bh().Quoted 38 lines
Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] Call Trace: Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81327546>] ? ocfs2_read_blocks+0x176/0x6c0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8114e541>] ? T.1552+0x91/0x2b0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81346ad0>] ? ocfs2_find_actor+0x120/0x120 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813464f7>] ? ocfs2_read_inode_block_full+0x37/0x60 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813964ff>] ? ocfs2_fast_symlink_readpage+0x2f/0x160 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81111585>] ? do_read_cache_page+0x85/0x180 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813964d0>] ? ocfs2_fill_super+0x2500/0x2500 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff811116d9>] ? read_cache_page+0x9/0x20 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115c705>] ? page_getlink+0x25/0x80 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115c77b>] ? page_follow_link_light+0x1b/0x30 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8116099b>] ? path_lookupat+0x38b/0x720 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81160d5c>] ? do_path_lookup+0x2c/0xd0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81346f31>] ? ocfs2_inode_revalidate+0x71/0x160 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81161c0c>] ? user_path_at_empty+0x5c/0xb0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8106714a>] ? do_page_fault+0x1aa/0x3c0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81156f2d>] ? cp_new_stat+0x10d/0x120 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81157021>] ? vfs_fstatat+0x41/0x80 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115715f>] ? sys_newstat+0x1f/0x50 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff817ecee2>] ?Quoted 1 line
system_call_fastpath+0x16/0x1b
Quoted 4 lines
This stack trace is from 3.5, because of the location of the
BUG. The call path in the trace suggests the code added by Al's ea022d, but you say it breaks in 3.2 and 3.3 as well. Can you give me a trace from 3.2?
For a 3.2 kernel i get this stack trace. Different trace form 3.5 but exactly at the same moment. and for the same reasons.
Seems to be less immmediate than with 3.5 but more a subjective imrpession than something based on fact. ( it takes a few seconds after deliver is started to have the bug )
[ 716.402833] o2dlm: Joining domain B43153ED20B942E291251F2C138ADA9E ( 0 1 ) 2 nodes [ 716.501511] ocfs2: Mounting device (147,2) on (node 1, slot 0) with ordered data mode. [ 716.505744] mount.ocfs2 used greatest stack depth: 2936 bytes left [ 727.133743] deliver used greatest stack depth: 2632 bytes left [ 764.167029] deliver used greatest stack depth: 1896 bytes left [ 764.778872] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 [ 764.778897] IP: [<ffffffff8133c51a>] __ocfs2_change_file_space+0x75a/0x1690 [ 764.778922] PGD 62697067 PUD 67a81067 PMD 0 [ 764.778939] Oops: 0000 [#1] SMP [ 764.778953] CPU 0 [ 764.778959] Modules linked in: drbd lru_cache ipv6 [last unloaded: drbd] [ 764.778986] [ 764.778993] Pid: 5909, comm: deliver Not tainted 3.2.12-gentoo #2 HP ProLiant ML150 G3/ML150 G3 [ 764.779017] RIP: 0010:[<ffffffff8133c51a>] [<ffffffff8133c51a>] __ocfs2_change_file_space+0x75a/0x1690 [ 764.779041] RSP: 0018:ffff880067b2dd98 EFLAGS: 00010246 [ 764.779053] RAX: 0000000000000000 RBX: ffff880067f82000 RCX: ffff880063d11000 [ 764.779069] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88007ae83288 [ 764.779085] RBP: ffff880055d1f138 R08: 0010000000000000 R09: ffff880063d11000 [ 764.779100] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88007ae83288 [ 764.779115] R13: 0000000000000000 R14: 0000000000000000 R15: 00000000000000df [ 764.779132] FS: 00007f1e40eb5700(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000 [ 764.779149] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 764.779219] CR2: 0000000000000038 CR3: 0000000067ab5000 CR4: 00000000000006f0 [ 764.779291] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 764.779364] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 764.779436] Process deliver (pid: 5909, threadinfo ffff880067b2c000, task ffff88007bedbc00) [ 764.779569] Stack: [ 764.779634] ffffea0001647840 ffffffff8112983f 0000000000000000 ffff880000000000 [ 764.779768] 00000000000de000 ffffffff81333f35 ffffffff8133f880 0000000000000000 [ 764.779903] 000000017d002240 ffff880055d1f1d8 ffff880000000001 ffff880067976708 [ 764.780009] Call Trace: [ 764.780009] [<ffffffff8112983f>] ? handle_pte_fault+0x7cf/0x9e0 [ 764.780009] [<ffffffff81333f35>] ? ocfs2_inode_lock_full_nested+0x355/0xb40 [ 764.780009] [<ffffffff8133f880>] ? ocfs2_inode_revalidate+0x70/0x160 [ 764.780009] [<ffffffff8106337a>] ? do_page_fault+0x1aa/0x3c0 [ 764.780009] [<ffffffff8114e780>] ? cp_new_stat+0xe0/0x100 [ 764.780009] [<ffffffff8133d4cd>] ? ocfs2_fallocate+0x7d/0x90 [ 764.780009] [<ffffffff811489e7>] ? do_fallocate+0x117/0x120 [ 764.780009] [<ffffffff81148a34>] ? sys_fallocate+0x44/0x70 [ 764.780009] [<ffffffff81771bbb>] ? system_call_fastpath+0x16/0x1b [ 764.780009] Code: 89 45 60 48 89 55 68 48 89 45 70 48 89 55 78 4c 89 e7 48 8b 94 24 00 01 00 00 e8 12 31 00 00 41 89 c2 85 c0 78 2e 48 8b 54 24 38 <f7> 42 38 00 10 10 00 74 06 41 80 4c 24 14 01 44 89 54 24 18 4c [ 764.780785] RIP [<ffffffff8133c51a>] __ocfs2_change_file_space+0x75a/0x1690 [ 764.780785] RSP <ffff880067b2dd98> [ 764.780785] CR2: 0000000000000038 [ 764.781561] ---[ end trace 654757aba94c3768 ]---
Vincent
Quoted 2 lines
Joel
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Joel Becker
maintainerQuoted 80 lines · Vincent ETIENNE
HI,
Quoted 9 lines · Joel Becker
Quoted 8 lines · Vincent ETIENNE
Hello
Get this on first write made ( by deliver sending mail to inform of the restart of services )
Home partition (the one receiving the mail) is based on ocfs2 created from drbd block device in primary/primary mode
These drbd devices are based on lvm.Quoted 8 lines
Quoted 7 lines
system is running linux-3.5.0, identical symptom with linux 3.3 and 3.2 but working with linux 3.0 kernel
reproduced on two machines ( so different hardware involved on this one software md raid on SATA, on second one areca hardware raid card ) but the 2 machines are the one sharing this partition ( so share the same data )
Hmm. Any chance you can bisect this further?
Will try to. Will take a few days as the server is in production ( but used as backup so...)
Quoted 50 lines
Quoted 4 lines
Jul 27 23:41:41 jupiter2 kernel: [ 351.169213] ------------[ cut here ]------------ Jul 27 23:41:41 jupiter2 kernel: [ 351.169261] kernel BUG at
fs/buffer.c:2886!This is: BUG_ON(!buffer_mapped(bh));in
submit_bh().Quoted 38 lines
Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] Call Trace: Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81327546>] ? ocfs2_read_blocks+0x176/0x6c0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8114e541>] ? T.1552+0x91/0x2b0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81346ad0>] ? ocfs2_find_actor+0x120/0x120 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813464f7>] ? ocfs2_read_inode_block_full+0x37/0x60 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813964ff>] ? ocfs2_fast_symlink_readpage+0x2f/0x160 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81111585>] ? do_read_cache_page+0x85/0x180 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813964d0>] ? ocfs2_fill_super+0x2500/0x2500 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff811116d9>] ? read_cache_page+0x9/0x20 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115c705>] ? page_getlink+0x25/0x80 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115c77b>] ? page_follow_link_light+0x1b/0x30 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8116099b>] ? path_lookupat+0x38b/0x720 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81160d5c>] ? do_path_lookup+0x2c/0xd0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81346f31>] ? ocfs2_inode_revalidate+0x71/0x160 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81161c0c>] ? user_path_at_empty+0x5c/0xb0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8106714a>] ? do_page_fault+0x1aa/0x3c0 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81156f2d>] ? cp_new_stat+0x10d/0x120 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81157021>] ? vfs_fstatat+0x41/0x80 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115715f>] ? sys_newstat+0x1f/0x50 Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff817ecee2>] ?Quoted 1 line
system_call_fastpath+0x16/0x1bQuoted 4 lines
This stack trace is from 3.5, because of the location of the
BUG. The call path in the trace suggests the code added by Al's ea022d, but you say it breaks in 3.2 and 3.3 as well. Can you give me a trace from 3.2?
Quoted 5 lines
For a 3.2 kernel i get this stack trace. Different trace form 3.5 but exactly at the same moment. and for the same reasons.
Seems to be less immmediate than with 3.5 but more a subjective imrpession than something based on fact. ( it takes a few seconds after deliver is started to have the bug )
Totally different stack trace. Not in symlink code, but instead in fallocate. Weird. I wonder if you are hitting two things. Bisection will definitely help.
Joel
Quoted 79 lines
[ 716.402833] o2dlm: Joining domain B43153ED20B942E291251F2C138ADA9E ( 0 1 ) 2 nodes [ 716.501511] ocfs2: Mounting device (147,2) on (node 1, slot 0) with ordered data mode. [ 716.505744] mount.ocfs2 used greatest stack depth: 2936 bytes left [ 727.133743] deliver used greatest stack depth: 2632 bytes left [ 764.167029] deliver used greatest stack depth: 1896 bytes left [ 764.778872] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 [ 764.778897] IP: [<ffffffff8133c51a>] __ocfs2_change_file_space+0x75a/0x1690 [ 764.778922] PGD 62697067 PUD 67a81067 PMD 0 [ 764.778939] Oops: 0000 [#1] SMP [ 764.778953] CPU 0 [ 764.778959] Modules linked in: drbd lru_cache ipv6 [last unloaded: drbd] [ 764.778986] [ 764.778993] Pid: 5909, comm: deliver Not tainted 3.2.12-gentoo #2 HP ProLiant ML150 G3/ML150 G3 [ 764.779017] RIP: 0010:[<ffffffff8133c51a>] [<ffffffff8133c51a>] __ocfs2_change_file_space+0x75a/0x1690 [ 764.779041] RSP: 0018:ffff880067b2dd98 EFLAGS: 00010246 [ 764.779053] RAX: 0000000000000000 RBX: ffff880067f82000 RCX: ffff880063d11000 [ 764.779069] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88007ae83288 [ 764.779085] RBP: ffff880055d1f138 R08: 0010000000000000 R09: ffff880063d11000 [ 764.779100] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88007ae83288 [ 764.779115] R13: 0000000000000000 R14: 0000000000000000 R15: 00000000000000df [ 764.779132] FS: 00007f1e40eb5700(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000 [ 764.779149] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 764.779219] CR2: 0000000000000038 CR3: 0000000067ab5000 CR4: 00000000000006f0 [ 764.779291] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 764.779364] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 764.779436] Process deliver (pid: 5909, threadinfo ffff880067b2c000, task ffff88007bedbc00) [ 764.779569] Stack: [ 764.779634] ffffea0001647840 ffffffff8112983f 0000000000000000 ffff880000000000 [ 764.779768] 00000000000de000 ffffffff81333f35 ffffffff8133f880 0000000000000000 [ 764.779903] 000000017d002240 ffff880055d1f1d8 ffff880000000001 ffff880067976708 [ 764.780009] Call Trace: [ 764.780009] [<ffffffff8112983f>] ? handle_pte_fault+0x7cf/0x9e0 [ 764.780009] [<ffffffff81333f35>] ? ocfs2_inode_lock_full_nested+0x355/0xb40 [ 764.780009] [<ffffffff8133f880>] ? ocfs2_inode_revalidate+0x70/0x160 [ 764.780009] [<ffffffff8106337a>] ? do_page_fault+0x1aa/0x3c0 [ 764.780009] [<ffffffff8114e780>] ? cp_new_stat+0xe0/0x100 [ 764.780009] [<ffffffff8133d4cd>] ? ocfs2_fallocate+0x7d/0x90 [ 764.780009] [<ffffffff811489e7>] ? do_fallocate+0x117/0x120 [ 764.780009] [<ffffffff81148a34>] ? sys_fallocate+0x44/0x70 [ 764.780009] [<ffffffff81771bbb>] ? system_call_fastpath+0x16/0x1b [ 764.780009] Code: 89 45 60 48 89 55 68 48 89 45 70 48 89 55 78 4c 89 e7 48 8b 94 24 00 01 00 00 e8 12 31 00 00 41 89 c2 85 c0 78 2e 48 8b 54 24 38 <f7> 42 38 00 10 10 00 74 06 41 80 4c 24 14 01 44 89 54 24 18 4c [ 764.780785] RIP [<ffffffff8133c51a>] __ocfs2_change_file_space+0x75a/0x1690 [ 764.780785] RSP <ffff880067b2dd98> [ 764.780785] CR2: 0000000000000038 [ 764.781561] ---[ end trace 654757aba94c3768 ]---Vincent
Quoted 2 lines
Joel
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 6 lines
Life's Little Instruction Book #456 "Send your loved one flowers. Think of a reason later." http://www.jlbec.org/ [email protected]
The fallocate() oops is probably the same that is fixed by this patch.
https://oss.oracle.com/git/?p=smushran/linux-2.6.git;a=commit;h=a2118b301104a24381b414bc93371d666fe8d43a
Is in the list of patches that are ready to be pushed.
https://oss.oracle.com/git/?p=smushran/linux-2.6.git;a=shortlog;h=mw-3.4-mar15
Quoted 202 lines · Joel Becker
Quoted 8 lines · Vincent ETIENNE
HI,
Quoted 4 lines · Joel Becker
Quoted 3 lines · Vincent ETIENNE
Hello
Get this on first write made ( by deliver sending mail to inform of
the
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
restart of services )
Home partition (the one receiving the mail) is based on ocfs2 created from drbd block device in primary/primary mode
These drbd devices are based on lvm.system is running linux-3.5.0, identical symptom with linux 3.3 and
3.2
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
but working with linux 3.0 kernel
reproduced on two machines ( so different hardware involved on this
one
Quoted 21 lines
Quoted 4 lines
Quoted 3 lines
software md raid on SATA, on second one areca hardware raid card ) but the 2 machines are the one sharing this partition ( so share the same data )
Hmm. Any chance you can bisect this further?
Will try to. Will take a few days as the server is in production ( but used as backup so...)
Quoted 13 lines
Quoted 4 lines
Jul 27 23:41:41 jupiter2 kernel: [ 351.169213] ------------[ cut here ]------------ Jul 27 23:41:41 jupiter2 kernel: [ 351.169261] kernel BUG at
fs/buffer.c:2886!This is: BUG_ON(!buffer_mapped(bh));in
submit_bh().Quoted 2 lines
Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] Call Trace: Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81327546>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
ocfs2_read_blocks+0x176/0x6c0Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8114e541>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
T.1552+0x91/0x2b0
Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81346ad0>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
ocfs2_find_actor+0x120/0x120Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813464f7>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
ocfs2_read_inode_block_full+0x37/0x60Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813964ff>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
ocfs2_fast_symlink_readpage+0x2f/0x160Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81111585>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
do_read_cache_page+0x85/0x180Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff813964d0>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
ocfs2_fill_super+0x2500/0x2500Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff811116d9>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
read_cache_page+0x9/0x20Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115c705>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
page_getlink+0x25/0x80Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115c77b>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
page_follow_link_light+0x1b/0x30Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8116099b>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
path_lookupat+0x38b/0x720Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81160d5c>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
do_path_lookup+0x2c/0xd0Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81346f31>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
ocfs2_inode_revalidate+0x71/0x160Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81161c0c>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
user_path_at_empty+0x5c/0xb0Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8106714a>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
do_page_fault+0x1aa/0x3c0Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81156f2d>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
cp_new_stat+0x10d/0x120Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff81157021>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
vfs_fstatat+0x41/0x80Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff8115715f>]?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
sys_newstat+0x1f/0x50Jul 27 23:41:41 jupiter2 kernel: [ 351.170003] [<ffffffff817ecee2>]?
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
system_call_fastpath+0x16/0x1bThis stack trace is from 3.5, because of the location of the
BUG. The call path in the trace suggests the code added by Al'sea022d,
Quoted 8 lines
Quoted 2 lines
but you say it breaks in 3.2 and 3.3 as well. Can you give me a trace from 3.2?
For a 3.2 kernel i get this stack trace. Different trace form 3.5 but exactly at the same moment. and for the same reasons.
Seems to be less immmediate than with 3.5 but more a subjective imrpession than something based on fact. ( it takes a few seconds after deliver is started to have the bug )Totally different stack trace. Not in symlink code, but instead in fallocate. Weird. I wonder if you are hitting two things. Bisection will definitely help.
Joel
Quoted 7 lines
[ 716.402833] o2dlm: Joining domain B43153ED20B942E291251F2C138ADA9E ( 0 1 ) 2 nodes [ 716.501511] ocfs2: Mounting device (147,2) on (node 1, slot 0) with ordered data mode. [ 716.505744] mount.ocfs2 used greatest stack depth: 2936 bytes left [ 727.133743] deliver used greatest stack depth: 2632 bytes left [ 764.167029] deliver used greatest stack depth: 1896 bytes leftQuoted 8 lines
[ 764.778872] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 [ 764.778897] IP: [<ffffffff8133c51a>] __ocfs2_change_file_space+0x75a/0x1690 [ 764.778922] PGD 62697067 PUD 67a81067 PMD 0 [ 764.778939] Oops: 0000 [#1] SMP [ 764.778953] CPU 0 [ 764.778959] Modules linked in: drbd lru_cache ipv6 [last unloaded:drbd]
Quoted 54 lines
[ 764.778986] [ 764.778993] Pid: 5909, comm: deliver Not tainted 3.2.12-gentoo #2 HP ProLiant ML150 G3/ML150 G3 [ 764.779017] RIP: 0010:[<ffffffff8133c51a>] [<ffffffff8133c51a>] __ocfs2_change_file_space+0x75a/0x1690 [ 764.779041] RSP: 0018:ffff880067b2dd98 EFLAGS: 00010246 [ 764.779053] RAX: 0000000000000000 RBX: ffff880067f82000 RCX: ffff880063d11000 [ 764.779069] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88007ae83288 [ 764.779085] RBP: ffff880055d1f138 R08: 0010000000000000 R09: ffff880063d11000 [ 764.779100] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88007ae83288 [ 764.779115] R13: 0000000000000000 R14: 0000000000000000 R15: 00000000000000df [ 764.779132] FS: 00007f1e40eb5700(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000 [ 764.779149] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 764.779219] CR2: 0000000000000038 CR3: 0000000067ab5000 CR4: 00000000000006f0 [ 764.779291] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 764.779364] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 764.779436] Process deliver (pid: 5909, threadinfo ffff880067b2c000, task ffff88007bedbc00) [ 764.779569] Stack: [ 764.779634] ffffea0001647840 ffffffff8112983f 0000000000000000 ffff880000000000 [ 764.779768] 00000000000de000 ffffffff81333f35 ffffffff8133f880 0000000000000000 [ 764.779903] 000000017d002240 ffff880055d1f1d8 ffff880000000001 ffff880067976708 [ 764.780009] Call Trace: [ 764.780009] [<ffffffff8112983f>] ? handle_pte_fault+0x7cf/0x9e0 [ 764.780009] [<ffffffff81333f35>] ? ocfs2_inode_lock_full_nested+0x355/0xb40 [ 764.780009] [<ffffffff8133f880>] ? ocfs2_inode_revalidate+0x70/0x160 [ 764.780009] [<ffffffff8106337a>] ? do_page_fault+0x1aa/0x3c0 [ 764.780009] [<ffffffff8114e780>] ? cp_new_stat+0xe0/0x100 [ 764.780009] [<ffffffff8133d4cd>] ? ocfs2_fallocate+0x7d/0x90 [ 764.780009] [<ffffffff811489e7>] ? do_fallocate+0x117/0x120 [ 764.780009] [<ffffffff81148a34>] ? sys_fallocate+0x44/0x70 [ 764.780009] [<ffffffff81771bbb>] ? system_call_fastpath+0x16/0x1b [ 764.780009] Code: 89 45 60 48 89 55 68 48 89 45 70 48 89 55 78 4c 89 e7 48 8b 94 24 00 01 00 00 e8 12 31 00 00 41 89 c2 85 c0 78 2e 48 8b 54 24 38 <f7> 42 38 00 10 10 00 74 06 41 80 4c 24 14 01 44 89 54 24 18 4c [ 764.780785] RIP [<ffffffff8133c51a>] __ocfs2_change_file_space+0x75a/0x1690 [ 764.780785] RSP <ffff880067b2dd98> [ 764.780785] CR2: 0000000000000038 [ 764.781561] ---[ end trace 654757aba94c3768 ]---Quoted 7 lines
Vincent
Quoted 2 lines
Joel
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel"in
Quoted 3 lines
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/--
Life's Little Instruction Book #456
Quoted 9 lines
"Send your loved one flowers. Think of a reason later."
http://www.jlbec.org/
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
Quoted 37 lines · Joel Becker
Quoted 36 lines · Vincent ETIENNE
Quoted 9 lines · Joel Becker
Quoted 8 lines · Vincent ETIENNE
Hello
Get this on first write made ( by deliver sending mail to inform of the restart of services )
Home partition (the one receiving the mail) is based on ocfs2 created from drbd block device in primary/primary mode
These drbd devices are based on lvm.Quoted 8 lines
Quoted 7 lines
system is running linux-3.5.0, identical symptom with linux 3.3 and 3.2 but working with linux 3.0 kernel
reproduced on two machines ( so different hardware involved on this one software md raid on SATA, on second one areca hardware raid card ) but the 2 machines are the one sharing this partition ( so share the same data )
Hmm. Any chance you can bisect this further?
Will try to. Will take a few days as the server is in production ( but used as backup so...)
Quoted 15 lines
Quoted 4 lines
Jul 27 23:41:41 jupiter2 kernel: [ 351.169213] ------------[ cut here ]------------ Jul 27 23:41:41 jupiter2 kernel: [ 351.169261] kernel BUG at
fs/buffer.c:2886!This is: BUG_ON(!buffer_mapped(bh));in
submit_bh().system_call_fastpath+0x16/0x1b This stack trace is from 3.5, because of the location of theBUG. The call path in the trace suggests the code added by Al's ea022d, but you say it breaks in 3.2 and 3.3 as well. Can you give me a trace from 3.2?
Quoted 8 lines
Quoted 5 lines
For a 3.2 kernel i get this stack trace. Different trace form 3.5 but exactly at the same moment. and for the same reasons.
Seems to be less immmediate than with 3.5 but more a subjective imrpession than something based on fact. ( it takes a few seconds after deliver is started to have the bug )Totally different stack trace. Not in symlink code, but instead in fallocate. Weird. I wonder if you are hitting two things. Bisection will definitely help.
Yes could be, that would explain the 2 stack trace ( and the different timing observed )
Bisection is in progress. The fallocate bug is certainly already corrected ( info sent by
[email protected] but unavailable on the list for the moment ?)
------
The fallocate() oops is probably the same that is fixed by this patch.
https://oss.oracle.com/git/?p=smushran/linux-2.6.git;a=commit;h=a2118b301104a24381b414bc93371d666fe8d43a
Is in the list of patches that are ready to be pushed.
https://oss.oracle.com/git/?p=smushran/linux-2.6.git;a=shortlog;h=mw-3.4-mar15
----
But not sure it will correct all i observed. So i will continue to bisect to confirm/infirm.
( But i seems to have lost network on my server after a reboot and so no more access before tomorrow , I have certainly forget to do make modules_install before installing new kernel ... Being stupid is not very helpful... ) . I hope to finish the bisection tomorrow or wednesday.
Thanks a lot for the support.
Quoted 3 lines
Joel
Some progress
the fallocate bug is not the only bug
latest head with the fallocate correction still crash ( in read_blocks )
So i have restart bisection but at each stage i reinject the fallocate patch ( is it a corerct way to do this ?)
Bisection is not very fast but for the moment (sometimes i need to rebot harsly and it kicks a rebuild of the raid array ) :
git bisect start
# bad: [2d534926205db9ffce4bbbde67cb9b2cee4b835c] Merge tag 'irqdomain-for-linus' of git://git.secretlab.ca/git/linux-2.6
git bisect bad 2d534926205db9ffce4bbbde67cb9b2cee4b835c
# good: [c3b92c8787367a8bb53d57d9789b558f1295cc96] Linux 3.1 git bisect good c3b92c8787367a8bb53d57d9789b558f1295cc96
# good: [95211279c5ad00a317c98221d7e4365e02f20836] Merge branch 'akpm' (Andrew's patch-bomb)
git bisect good 95211279c5ad00a317c98221d7e4365e02f20836
# good: [654443e20dfc0617231f28a07c96a979ee1a0239] Merge branch 'perf-uprobes-for-linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 654443e20dfc0617231f28a07c96a979ee1a0239
# bad: [f0a08fcb5972167e55faa330c4a24fbaa3328b1f] Mergegit://git.kernel.org/pub/scm/linux/kernel/git/cmetcalf/linux-tile
git bisect bad f0a08fcb5972167e55faa330c4a24fbaa3328b1f
# bad: [f5e7e844a571124ffc117d4696787d6afc4fc5ae] Merge tag 'for-linus-3.5-20120601' of git://git.infradead.org/linux-mtd
git bisect bad f5e7e844a571124ffc117d4696787d6afc4fc5ae
Each bad has failed with the read_block OOPS ( so somewhat consistent for now )
Quoted 76 lines · Vincent ETIENNE
Quoted 37 lines · Joel Becker
Quoted 36 lines · Vincent ETIENNE
Quoted 9 lines · Joel Becker
Quoted 8 lines · Vincent ETIENNE
Hello
Get this on first write made ( by deliver sending mail to inform of the restart of services )
Home partition (the one receiving the mail) is based on ocfs2 created from drbd block device in primary/primary mode
These drbd devices are based on lvm.Quoted 8 lines
Quoted 7 lines
system is running linux-3.5.0, identical symptom with linux 3.3 and 3.2 but working with linux 3.0 kernel
reproduced on two machines ( so different hardware involved on this one software md raid on SATA, on second one areca hardware raid card ) but the 2 machines are the one sharing this partition ( so share the same data )
Hmm. Any chance you can bisect this further?
Will try to. Will take a few days as the server is in production ( but used as backup so...)
Quoted 15 lines
Quoted 4 lines
Jul 27 23:41:41 jupiter2 kernel: [ 351.169213] ------------[ cut here ]------------ Jul 27 23:41:41 jupiter2 kernel: [ 351.169261] kernel BUG at
fs/buffer.c:2886!This is: BUG_ON(!buffer_mapped(bh));in
submit_bh().system_call_fastpath+0x16/0x1b This stack trace is from 3.5, because of the location of theBUG. The call path in the trace suggests the code added by Al's ea022d, but you say it breaks in 3.2 and 3.3 as well. Can you give me a trace from 3.2?
Quoted 8 lines
Quoted 5 lines
For a 3.2 kernel i get this stack trace. Different trace form 3.5 but exactly at the same moment. and for the same reasons.
Seems to be less immmediate than with 3.5 but more a subjective imrpession than something based on fact. ( it takes a few seconds after deliver is started to have the bug )Totally different stack trace. Not in symlink code, but instead in fallocate. Weird. I wonder if you are hitting two things. Bisection will definitely help.
Yes could be, that would explain the 2 stack trace ( and the different timing observed )
Bisection is in progress. The fallocate bug is certainly already corrected ( info sent by
[email protected] but unavailable on the list for the moment ?)------
The
fallocate()oops is probably the same that is fixed by this patch.
https://oss.oracle.com/git/?p=smushran/linux-2.6.git;a=commit;h=a2118b301104a24381b414bc93371d666fe8d43aIs in the list of patches that are ready to be pushed.
https://oss.oracle.com/git/?p=smushran/linux-2.6.git;a=shortlog;h=mw-3.4-mar15----
But not sure it will correct all i observed. So i will continue to bisect to confirm/infirm.
( But i seems to have lost network on my server after a reboot and so no more access before tomorrow , I have certainly forget to do makemodules_installbefore installing new kernel ... Being stupid is not very helpful... ) . I hope to finish the bisection tomorrow or wednesday.Thanks a lot for the support.
Quoted 3 lines
Joel
Hi
Some further progress on bisection
I'm now here
git bisect start
# bad: [2d534926205db9ffce4bbbde67cb9b2cee4b835c] Merge tag 'irqdomain-for-linus' of git://git.secretlab.ca/git/linux-2.6
git bisect bad 2d534926205db9ffce4bbbde67cb9b2cee4b835c
# good: [c3b92c8787367a8bb53d57d9789b558f1295cc96] Linux 3.1 git bisect good c3b92c8787367a8bb53d57d9789b558f1295cc96
# good: [95211279c5ad00a317c98221d7e4365e02f20836] Merge branch 'akpm' (Andrew's patch-bomb)
git bisect good 95211279c5ad00a317c98221d7e4365e02f20836
# good: [654443e20dfc0617231f28a07c96a979ee1a0239] Merge branch 'perf-uprobes-for-linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 654443e20dfc0617231f28a07c96a979ee1a0239
# bad: [f0a08fcb5972167e55faa330c4a24fbaa3328b1f] Mergegit://git.kernel.org/pub/scm/linux/kernel/git/cmetcalf/linux-tile
git bisect bad f0a08fcb5972167e55faa330c4a24fbaa3328b1f
# bad: [f5e7e844a571124ffc117d4696787d6afc4fc5ae] Merge tag 'for-linus-3.5-20120601' of git://git.infradead.org/linux-mtd
git bisect bad f5e7e844a571124ffc117d4696787d6afc4fc5ae
# good: [f465d145d76803fe6332092775d891c8c509aa44] Merge tag 'cleanup-initcall' ofgit://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect good f465d145d76803fe6332092775d891c8c509aa44
# good: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch 'for-3.5/drivers' of git://git.kernel.dk/linux-block
git bisect good a70f35af4e49f87ba4b6c4b30220fbb66cd74af6
# good: [a00b6151a2ae4c52576c35d3998e144a993d50b8] Merge branch 'for-3.5-take-2' of git://linux-nfs.org/~bfields/linux
git bisect good a00b6151a2ae4c52576c35d3998e144a993d50b8
# bad: [1193755ac6328ad240ba987e6ec41d5e8baf0680] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
git bisect bad 1193755ac6328ad240ba987e6ec41d5e8baf0680
# good: [51eab603f5c86dd1eae4c525df3e7f7eeab401d6] Merge branch 'for-linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs
git bisect good 51eab603f5c86dd1eae4c525df3e7f7eeab401d6
# bad: [eb36c5873b96e8c7376768d3906da74aae6e3839] new helper:vm_mmap_pgoff()
git bisect bad eb36c5873b96e8c7376768d3906da74aae6e3839
but got a problem ( kernel does not compile ) at next iteration need to dig into git bisect for how to select another entry
Vincent
Quoted 45 lines · Vincent ETIENNE
Hi
Some further progress on bisectionI'm now here
git bisect start
# bad: [2d534926205db9ffce4bbbde67cb9b2cee4b835c] Merge tag 'irqdomain-for-linus' ofgit://git.secretlab.ca/git/linux-2.6
git bisect bad2d534926205db9ffce4bbbde67cb9b2cee4b835c
# good: [c3b92c8787367a8bb53d57d9789b558f1295cc96] Linux 3.1 git bisect goodc3b92c8787367a8bb53d57d9789b558f1295cc96
# good: [95211279c5ad00a317c98221d7e4365e02f20836] Merge branch 'akpm' (Andrew's patch-bomb)
git bisect good95211279c5ad00a317c98221d7e4365e02f20836
# good: [654443e20dfc0617231f28a07c96a979ee1a0239] Merge branch 'perf-uprobes-for-linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good654443e20dfc0617231f28a07c96a979ee1a0239
# bad: [f0a08fcb5972167e55faa330c4a24fbaa3328b1f] Mergegit://git.kernel.org/pub/scm/linux/kernel/git/cmetcalf/linux-tile
git bisect badf0a08fcb5972167e55faa330c4a24fbaa3328b1f
# bad: [f5e7e844a571124ffc117d4696787d6afc4fc5ae] Merge tag 'for-linus-3.5-20120601' ofgit://git.infradead.org/linux-mtd
git bisect badf5e7e844a571124ffc117d4696787d6afc4fc5ae
# good: [f465d145d76803fe6332092775d891c8c509aa44] Merge tag 'cleanup-initcall' ofgit://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect goodf465d145d76803fe6332092775d891c8c509aa44
# good: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch 'for-3.5/drivers' ofgit://git.kernel.dk/linux-block
git bisect gooda70f35af4e49f87ba4b6c4b30220fbb66cd74af6
# good: [a00b6151a2ae4c52576c35d3998e144a993d50b8] Merge branch 'for-3.5-take-2' ofgit://linux-nfs.org/~bfields/linux
git bisect gooda00b6151a2ae4c52576c35d3998e144a993d50b8
# bad: [1193755ac6328ad240ba987e6ec41d5e8baf0680] Merge branch 'for-linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
git bisect bad1193755ac6328ad240ba987e6ec41d5e8baf0680
# good: [51eab603f5c86dd1eae4c525df3e7f7eeab401d6] Merge branch 'for-linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs
git bisect good51eab603f5c86dd1eae4c525df3e7f7eeab401d6
# bad: [eb36c5873b96e8c7376768d3906da74aae6e3839] new helper:vm_mmap_pgoff()
git bisect badeb36c5873b96e8c7376768d3906da74aae6e3839
Quoted 6 lines
but got a problem ( kernel does not compile ) at next iteration need to dig into git bisect for how to select another entry
Vincent
git bisect skip looks like the way to do it ?
Hi,
So 12 commits left, corresponding to this bisection log
git bisect start
# bad: [2d534926205db9ffce4bbbde67cb9b2cee4b835c] Merge tag 'irqdomain-for-linus' of git://git.secretlab.ca/git/linux-2.6
git bisect bad 2d534926205db9ffce4bbbde67cb9b2cee4b835c
# good: [c3b92c8787367a8bb53d57d9789b558f1295cc96] Linux 3.1 git bisect good c3b92c8787367a8bb53d57d9789b558f1295cc96
# good: [95211279c5ad00a317c98221d7e4365e02f20836] Merge branch 'akpm' (Andrew's patch-bomb)
git bisect good 95211279c5ad00a317c98221d7e4365e02f20836
# good: [654443e20dfc0617231f28a07c96a979ee1a0239] Merge branch 'perf-uprobes-for-linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 654443e20dfc0617231f28a07c96a979ee1a0239
# bad: [f0a08fcb5972167e55faa330c4a24fbaa3328b1f] Mergegit://git.kernel.org/pub/scm/linux/kernel/git/cmetcalf/linux-tile
git bisect bad f0a08fcb5972167e55faa330c4a24fbaa3328b1f
# bad: [f5e7e844a571124ffc117d4696787d6afc4fc5ae] Merge tag 'for-linus-3.5-20120601' of git://git.infradead.org/linux-mtd
git bisect bad f5e7e844a571124ffc117d4696787d6afc4fc5ae
# good: [f465d145d76803fe6332092775d891c8c509aa44] Merge tag 'cleanup-initcall' ofgit://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect good f465d145d76803fe6332092775d891c8c509aa44
# good: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch 'for-3.5/drivers' of git://git.kernel.dk/linux-block
git bisect good a70f35af4e49f87ba4b6c4b30220fbb66cd74af6
# good: [a00b6151a2ae4c52576c35d3998e144a993d50b8] Merge branch 'for-3.5-take-2' of git://linux-nfs.org/~bfields/linux
git bisect good a00b6151a2ae4c52576c35d3998e144a993d50b8
# bad: [1193755ac6328ad240ba987e6ec41d5e8baf0680] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
git bisect bad 1193755ac6328ad240ba987e6ec41d5e8baf0680
# good: [51eab603f5c86dd1eae4c525df3e7f7eeab401d6] Merge branch 'for-linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs
git bisect good 51eab603f5c86dd1eae4c525df3e7f7eeab401d6
# bad: [eb36c5873b96e8c7376768d3906da74aae6e3839] new helper:vm_mmap_pgoff()
git bisect bad eb36c5873b96e8c7376768d3906da74aae6e3839
# skip: [eea62f831b8030b0eeea8314eed73b6132d1de26] brlocks/lglocks: turn into functions
git bisect skip eea62f831b8030b0eeea8314eed73b6132d1de26
# good: [52576da3545e78c534d901a39f6f2391665c641b] hpfs: bitmaps are little-endian
git bisect good 52576da3545e78c534d901a39f6f2391665c641b
# bad: [3ed37648e1cbf1bbebc200c6ea8fd8daf8325843] fs: move file_remove_suid() to fs/inode.c
git bisect bad 3ed37648e1cbf1bbebc200c6ea8fd8daf8325843
# bad: [962830df366b66e71849040770ae6ba55a8b4aec] brlocks/lglocks: API cleanups
git bisect bad 962830df366b66e71849040770ae6ba55a8b4aec
the commit left are
commit 962830df366b66e71849040770ae6ba55a8b4aec
Author: Andi Kleen <[email protected]> Date: Tue May 8 13:32:02 2012 +0930
brlocks/lglocks: API cleanups
commit eea62f831b8030b0eeea8314eed73b6132d1de26
Author: Andi Kleen <[email protected]> Date: Tue May 8 13:32:24 2012 +0930
brlocks/lglocks: turn into functions
commit 9dd6fa03ab31bb57cee4623a689d058d222fbe68
Author: Rusty Russell <[email protected]> Date: Tue May 8 13:29:45 2012 +0930
lglock: remove online variants of lock
commit ea022dfb3c2a4680483b00eb2fecc9fc4f6091d1
Author: Al Viro <[email protected]> Date: Thu May 3 10:14:29 2012 -0400
ocfs: simplify symlink handling
commit 408bd629badbd4353b238ab6f58001529b274d73
Author: Al Viro <[email protected]> Date: Thu May 3 09:34:20 2012 -0400
get rid of pointless allocations and copying in ecryptfs_follow_link()
switch to generic_readlink(), while we are at it
commit 28fe3c1963b0bafa56ec92df1987828090151d87
Author: Al Viro <[email protected]> Date: Tue Apr 17 16:41:13 2012 -0400
hpfs: assorted endianness annotations
commit 77ee26e44c28823a29bc09091950544566ae7cea
Author: Al Viro <[email protected]> Date: Tue Apr 17 16:26:46 2012 -0400
hpfs: annotate ea
commit 46287aa652fa8ea1edac41817ddc63332495ffc3
Author: Al Viro <[email protected]> Date: Tue Apr 17 16:20:49 2012 -0400
hpfs: annotate struct hpfs_dirent
commit 6ce2bbba5266c1dd5c27dd8af1887ed8ca564919
Author: Al Viro <[email protected]> Date: Tue Apr 17 16:11:25 2012 -0400
hpfs: annotate struct anode
commit 2b9f1cc29ba0e56089fe04501ec6d3b49eee3c3e
Author: Al Viro <[email protected]> Date: Tue Apr 17 16:09:25 2012 -0400
hpfs: annotate struct fnode
commit ddc19e6e04c1131a48f5b9a25aa433bbd8430cdd
Author: Al Viro <[email protected]> Date: Tue Apr 17 15:59:35 2012 -0400
hpfs: annotate btree nodes, get rid of bitfields mess
commit 39413c6046de282a92739110cfafb8f1e862680d
Author: Al Viro <[email protected]> Date: Tue Apr 17 15:32:22 2012 -0400
hpfs: annotate struct dnode
After that bisection start to be quite hard : i have compile error or unbootable kernel or unrelated OOPS
For the record the BUG that i'm chasing is this one :
[ 934.933390] kernel BUG at fs/buffer.c:2882! [ 934.933401] invalid opcode: 0000 [#1] SMP [ 934.933412] CPU 0 [ 934.933419] Modules linked in: drbd lru_cache [ 934.933429] [ 934.933437] Pid: 4301, comm: deliver Not tainted 3.4.0+ #16 HP ProLiant ML150 G3/ML150 G3 [ 934.933461] RIP: 0010:[<ffffffff8117a252>] [<ffffffff8117a252>] submit_bh+0x112/0x120 [ 934.933477] RSP: 0018:ffff880076c07b38 EFLAGS: 00010246 [ 934.933489] RAX: 4000000000800004 RBX: ffffea0001d82dc0 RCX: 00000003ffffffff [ 934.933504] RDX: 0000000000000001 RSI: ffffea0001d82dc0 RDI: 0000000000000000 [ 934.933519] RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff8133b0a0 [ 934.933534] R10: dead000000200200 R11: dead000000100100 R12: 0000000004cc4789 [ 934.933549] R13: 00000003ffffffff R14: 0000000000000000 R15: 0000000000000000 [ 934.933564] FS: 00007fdb397ff700(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000 [ 934.933581] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 934.933593] CR2: 00007fe29e575024 CR3: 00000000766b3000 CR4: 00000000000007f0 [ 934.933608] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 934.933677] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 934.933750] Process deliver (pid: 4301, threadinfo ffff880076c06000, task ffff88007d10a4e0) [ 934.935047] Stack: [ 934.935113] ffffea0001d82dc0 0000000000000001 0000000004cc4789 ffffffff8131bad6 [ 934.935236] 000000001725db78 ffffffff81ca03c8 ffff88007d10ab48 ffff880076c07c10 [ 934.935355] 0000000076c06010 0000000100000000 ffff8800172269b0 ffff88000d2a4000 [ 934.935471] Call Trace: [ 934.935534] [<ffffffff8131bad6>] ? ocfs2_read_blocks+0x176/0x6c0 [ 934.935600] [<ffffffff8133b0a0>] ? ocfs2_find_actor+0x120/0x120 [ 934.935665] [<ffffffff8133aac7>] ? ocfs2_read_inode_block_full+0x37/0x60 [ 934.935734] [<ffffffff8138ab1f>] ? ocfs2_fast_symlink_readpage+0x2f/0x160 [ 934.935803] [<ffffffff8110fff5>] ? do_read_cache_page+0x85/0x180 [ 934.935868] [<ffffffff8138aaf0>] ? ocfs2_fill_super+0x2500/0x2500 [ 934.935934] [<ffffffff81110149>] ? read_cache_page+0x9/0x20 [ 934.935999] [<ffffffff811565e5>] ? page_getlink+0x25/0x80 [ 934.936062] [<ffffffff8115665b>] ? page_follow_link_light+0x1b/0x30 [ 934.936128] [<ffffffff8115ab27>] ? path_lookupat+0x327/0x6c0 [ 934.936193] [<ffffffff8115aeec>] ? do_path_lookup+0x2c/0xc0 [ 934.936258] [<ffffffff8133b501>] ? ocfs2_inode_revalidate+0x71/0x160 [ 934.936325] [<ffffffff8113210a>] ? handle_pte_fault+0x8a/0x920 [ 934.936390] [<ffffffff8115bd8c>] ? user_path_at_empty+0x5c/0xb0 [ 934.936457] [<ffffffff81066fca>] ? do_page_fault+0x1aa/0x3c0 [ 934.936523] [<ffffffff81150ded>] ? cp_new_stat+0x10d/0x120 [ 934.936588] [<ffffffff81150ee1>] ? vfs_fstatat+0x41/0x80 [ 934.936653] [<ffffffff8115101f>] ? sys_newstat+0x1f/0x50 [ 934.936720] [<ffffffff817cb8a2>] ? system_call_fastpath+0x16/0x1b
Does it gives some better information ? Do you haev some ideas of thinks that could be tested ?
Vincent
Hi
based on current git ( commit 1a9b4993b70fb1884716902774dc9025b457760d ) and reverting commit ea022dfb3c2a4680483b00eb2fecc9fc4f6091d1
commit ea022dfb3c2a4680483b00eb2fecc9fc4f6091d1
Author: Al Viro <[email protected]> Date: Thu May 3 10:14:29 2012 -0400
ocfs: simplify symlink handling
Suppress
Jul 31 09:42:12 jupiter2 kernel: [ 594.244726] kernel BUG at fs/buffer.c:2886! Jul 31 09:42:12 jupiter2 kernel: [ 594.244768] invalid opcode: 0000 [#1] SMP Jul 31 09:42:12 jupiter2 kernel: [ 594.244874] CPU 0 Jul 31 09:42:12 jupiter2 kernel: [ 594.244911] Modules linked in: drbd lru_cache [last unloaded: drbd] Jul 31 09:42:12 jupiter2 kernel: [ 594.245121] Jul 31 09:42:12 jupiter2 kernel: [ 594.245156] Pid: 5725, comm: deliver Not tainted 3.5.0-gentoo #3 HP ProLiant ML150 G3/ML150 G3 Jul 31 09:42:12 jupiter2 kernel: [ 594.245302] RIP: 0010:[<ffffffff81180862>] [<ffffffff81180862>] submit_bh+0x112/0x120 Jul 31 09:42:12 jupiter2 kernel: [ 594.245389] RSP: 0018:ffff88006032fb38 EFLAGS: 00010246 Jul 31 09:42:12 jupiter2 kernel: [ 594.245432] RAX: 4000000001000004 RBX: ffffea00014a1a80 RCX: 00000003ffffffff Jul 31 09:42:12 jupiter2 kernel: [ 594.245478] RDX: 0000000000000001 RSI: ffffea00014a1a80 RDI: 0000000000000000 Jul 31 09:42:12 jupiter2 kernel: [ 594.245523] RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff81346ad0 Jul 31 09:42:12 jupiter2 kernel: [ 594.245569] R10: dead000000200200 R11: 0000000000000000 R12: 0000000004cc4789 Jul 31 09:42:12 jupiter2 kernel: [ 594.245614] R13: 00000003ffffffff R14: 0000000000000000 R15: 0000000000000000 Jul 31 09:42:12 jupiter2 kernel: [ 594.245661] FS: 00007f23be7e6700(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000 Jul 31 09:42:12 jupiter2 kernel: [ 594.245708] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Jul 31 09:42:12 jupiter2 kernel: [ 594.245752] CR2: 00007f23bd098b6c CR3: 0000000061cfd000 CR4: 00000000000007f0 Jul 31 09:42:12 jupiter2 kernel: [ 594.245853] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jul 31 09:42:12 jupiter2 kernel: [ 594.245954] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jul 31 09:42:12 jupiter2 kernel: [ 594.246058] Process deliver (pid: 5725, threadinfo ffff88006032e000, task ffff88007c7f1e00) Jul 31 09:42:12 jupiter2 kernel: [ 594.246218] Stack: Jul 31 09:42:12 jupiter2 kernel: [ 594.246311] ffffea00014a1a80 0000000000000001 0000000004cc4789 ffffffff81327546 Jul 31 09:42:12 jupiter2 kernel: [ 594.246598] 0000000053a6db78 00000001800e000e ffff88007c7f2468 ffff88006032fc10 Jul 31 09:42:12 jupiter2 kernel: [ 594.246885] 0000000000000000 0000000100000000 ffff880053a7e9b0 ffff880056f32000 Jul 31 09:42:12 jupiter2 kernel: [ 594.247173] Call Trace: Jul 31 09:42:12 jupiter2 kernel: [ 594.247271] [<ffffffff81327546>] ? ocfs2_read_blocks+0x176/0x6c0 Jul 31 09:42:12 jupiter2 kernel: [ 594.247373] [<ffffffff81346ad0>] ? ocfs2_find_actor+0x120/0x120 Jul 31 09:42:12 jupiter2 kernel: [ 594.247474] [<ffffffff813464f7>] ? ocfs2_read_inode_block_full+0x37/0x60 Jul 31 09:42:12 jupiter2 kernel: [ 594.247578] [<ffffffff813964ff>] ? ocfs2_fast_symlink_readpage+0x2f/0x160 Jul 31 09:42:12 jupiter2 kernel: [ 594.247683] [<ffffffff81111585>] ? do_read_cache_page+0x85/0x180 Jul 31 09:42:12 jupiter2 kernel: [ 594.247784] [<ffffffff813964d0>] ? ocfs2_fill_super+0x2500/0x2500 Jul 31 09:42:12 jupiter2 kernel: [ 594.247883] [<ffffffff811116d9>] ? read_cache_page+0x9/0x20 Jul 31 09:42:12 jupiter2 kernel: [ 594.247984] [<ffffffff8115c705>] ? page_getlink+0x25/0x80 Jul 31 09:42:12 jupiter2 kernel: [ 594.248083] [<ffffffff8115c77b>] ? page_follow_link_light+0x1b/0x30 Jul 31 09:42:12 jupiter2 kernel: [ 594.248186] [<ffffffff8116099b>] ? path_lookupat+0x38b/0x720 Jul 31 09:42:12 jupiter2 kernel: [ 594.248286] [<ffffffff81160d5c>] ? do_path_lookup+0x2c/0xd0 Jul 31 09:42:12 jupiter2 kernel: [ 594.248385] [<ffffffff81346f31>] ? ocfs2_inode_revalidate+0x71/0x160 Jul 31 09:42:12 jupiter2 kernel: [ 594.248492] [<ffffffff8106b9d7>] ? flush_tlb_others_ipi+0x107/0x130 Jul 31 09:42:12 jupiter2 kernel: [ 594.248594] [<ffffffff81161c0c>] ? user_path_at_empty+0x5c/0xb0 Jul 31 09:42:12 jupiter2 kernel: [ 594.248694] [<ffffffff8106714a>] ? do_page_fault+0x1aa/0x3c0 Jul 31 09:42:12 jupiter2 kernel: [ 594.248789] [<ffffffff81156f2d>] ? cp_new_stat+0x10d/0x120 Jul 31 09:42:12 jupiter2 kernel: [ 594.248884] [<ffffffff81157021>] ? vfs_fstatat+0x41/0x80 Jul 31 09:42:12 jupiter2 kernel: [ 594.248978] [<ffffffff8115715f>] ? sys_newstat+0x1f/0x50 Jul 31 09:42:12 jupiter2 kernel: [ 594.249075] [<ffffffff817ecee2>] ? system_call_fastpath+0x16/0x1b Jul 31 09:42:12 jupiter2 kernel: [ 594.249169] Code: b6 44 24 18 4c 89 e7 83 e0 80 3c 01 19 db e8 76 3f 00 00 f7 d3 83 e3 a1 89 d8 5b 5d 41 5c c3 0f 0b eb fe 0f 0b eb fe 0f 0$ Jul 31 09:42:12 jupiter2 kernel: [ 594.250003] RIP [<ffffffff81180862>] submit_bh+0x112/0x120
and adding the correction from
suppres the fallocate bug
and lead to no oops. At least immediatly. Will let it run some times
Quoted 214 lines · Vincent ETIENNE
Hi,
So 12 commits left, corresponding to this bisection log
git bisect start
# bad: [2d534926205db9ffce4bbbde67cb9b2cee4b835c] Merge tag 'irqdomain-for-linus' ofgit://git.secretlab.ca/git/linux-2.6
git bisect bad2d534926205db9ffce4bbbde67cb9b2cee4b835c
# good: [c3b92c8787367a8bb53d57d9789b558f1295cc96] Linux 3.1 git bisect goodc3b92c8787367a8bb53d57d9789b558f1295cc96
# good: [95211279c5ad00a317c98221d7e4365e02f20836] Merge branch 'akpm' (Andrew's patch-bomb)
git bisect good95211279c5ad00a317c98221d7e4365e02f20836
# good: [654443e20dfc0617231f28a07c96a979ee1a0239] Merge branch 'perf-uprobes-for-linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good654443e20dfc0617231f28a07c96a979ee1a0239
# bad: [f0a08fcb5972167e55faa330c4a24fbaa3328b1f] Mergegit://git.kernel.org/pub/scm/linux/kernel/git/cmetcalf/linux-tile
git bisect badf0a08fcb5972167e55faa330c4a24fbaa3328b1f
# bad: [f5e7e844a571124ffc117d4696787d6afc4fc5ae] Merge tag 'for-linus-3.5-20120601' ofgit://git.infradead.org/linux-mtd
git bisect badf5e7e844a571124ffc117d4696787d6afc4fc5ae
# good: [f465d145d76803fe6332092775d891c8c509aa44] Merge tag 'cleanup-initcall' ofgit://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect goodf465d145d76803fe6332092775d891c8c509aa44
# good: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch 'for-3.5/drivers' ofgit://git.kernel.dk/linux-block
git bisect gooda70f35af4e49f87ba4b6c4b30220fbb66cd74af6
# good: [a00b6151a2ae4c52576c35d3998e144a993d50b8] Merge branch 'for-3.5-take-2' ofgit://linux-nfs.org/~bfields/linux
git bisect gooda00b6151a2ae4c52576c35d3998e144a993d50b8
# bad: [1193755ac6328ad240ba987e6ec41d5e8baf0680] Merge branch 'for-linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
git bisect bad1193755ac6328ad240ba987e6ec41d5e8baf0680
# good: [51eab603f5c86dd1eae4c525df3e7f7eeab401d6] Merge branch 'for-linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs
git bisect good51eab603f5c86dd1eae4c525df3e7f7eeab401d6
# bad: [eb36c5873b96e8c7376768d3906da74aae6e3839] new helper:vm_mmap_pgoff()
git bisect badeb36c5873b96e8c7376768d3906da74aae6e3839
# skip: [eea62f831b8030b0eeea8314eed73b6132d1de26] brlocks/lglocks: turn into functions
git bisect skipeea62f831b8030b0eeea8314eed73b6132d1de26
# good: [52576da3545e78c534d901a39f6f2391665c641b] hpfs: bitmaps are little-endian
git bisect good52576da3545e78c534d901a39f6f2391665c641b
# bad: [3ed37648e1cbf1bbebc200c6ea8fd8daf8325843] fs: movefile_remove_suid()tofs/inode.c
git bisect bad3ed37648e1cbf1bbebc200c6ea8fd8daf8325843
# bad: [962830df366b66e71849040770ae6ba55a8b4aec] brlocks/lglocks: API cleanups
git bisect bad962830df366b66e71849040770ae6ba55a8b4aecthe commit left are
commit
962830df366b66e71849040770ae6ba55a8b4aecAuthor: Andi Kleen <[email protected]> Date: Tue May 8 13:32:02 2012 +0930brlocks/lglocks: API cleanups
commit
eea62f831b8030b0eeea8314eed73b6132d1de26Author: Andi Kleen <[email protected]> Date: Tue May 8 13:32:24 2012 +0930brlocks/lglocks: turn into functions
commit
9dd6fa03ab31bb57cee4623a689d058d222fbe68Author: Rusty Russell <[email protected]> Date: Tue May 8 13:29:45 2012 +0930lglock: remove online variants of lock
commit
ea022dfb3c2a4680483b00eb2fecc9fc4f6091d1Author: Al Viro <[email protected]> Date: Thu May 3 10:14:29 2012 -0400ocfs: simplify symlink handling
commit
408bd629badbd4353b238ab6f58001529b274d73Author: Al Viro <[email protected]> Date: Thu May 3 09:34:20 2012 -0400get rid of pointless allocations and copying in
ecryptfs_follow_link()switch to
generic_readlink(), while we are at itcommit
28fe3c1963b0bafa56ec92df1987828090151d87Author: Al Viro <[email protected]> Date: Tue Apr 17 16:41:13 2012 -0400hpfs: assorted endianness annotations
commit
77ee26e44c28823a29bc09091950544566ae7ceaAuthor: Al Viro <[email protected]> Date: Tue Apr 17 16:26:46 2012 -0400hpfs: annotate ea
commit
46287aa652fa8ea1edac41817ddc63332495ffc3Author: Al Viro <[email protected]> Date: Tue Apr 17 16:20:49 2012 -0400hpfs: annotate struct
hpfs_direntcommit
6ce2bbba5266c1dd5c27dd8af1887ed8ca564919Author: Al Viro <[email protected]> Date: Tue Apr 17 16:11:25 2012 -0400hpfs: annotate struct anode
commit
2b9f1cc29ba0e56089fe04501ec6d3b49eee3c3eAuthor: Al Viro <[email protected]> Date: Tue Apr 17 16:09:25 2012 -0400hpfs: annotate struct fnode
commit
ddc19e6e04c1131a48f5b9a25aa433bbd8430cddAuthor: Al Viro <[email protected]> Date: Tue Apr 17 15:59:35 2012 -0400hpfs: annotate btree nodes, get rid of bitfields mess
commit
39413c6046de282a92739110cfafb8f1e862680dAuthor: Al Viro <[email protected]> Date: Tue Apr 17 15:32:22 2012 -0400hpfs: annotate struct dnode
After that bisection start to be quite hard : i have compile error or unbootable kernel or unrelated OOPS
For the record the BUG that i'm chasing is this one :
[ 934.933390] kernel BUG at fs/buffer.c:2882! [ 934.933401] invalid opcode: 0000 [#1] SMP [ 934.933412] CPU 0 [ 934.933419] Modules linked in: drbd lru_cache [ 934.933429] [ 934.933437] Pid: 4301, comm: deliver Not tainted 3.4.0+ #16 HP ProLiant ML150 G3/ML150 G3 [ 934.933461] RIP: 0010:[<ffffffff8117a252>] [<ffffffff8117a252>] submit_bh+0x112/0x120 [ 934.933477] RSP: 0018:ffff880076c07b38 EFLAGS: 00010246 [ 934.933489] RAX: 4000000000800004 RBX: ffffea0001d82dc0 RCX: 00000003ffffffff [ 934.933504] RDX: 0000000000000001 RSI: ffffea0001d82dc0 RDI: 0000000000000000 [ 934.933519] RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff8133b0a0 [ 934.933534] R10: dead000000200200 R11: dead000000100100 R12: 0000000004cc4789 [ 934.933549] R13: 00000003ffffffff R14: 0000000000000000 R15: 0000000000000000 [ 934.933564] FS: 00007fdb397ff700(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000 [ 934.933581] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 934.933593] CR2: 00007fe29e575024 CR3: 00000000766b3000 CR4: 00000000000007f0 [ 934.933608] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 934.933677] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 934.933750] Process deliver (pid: 4301, threadinfo ffff880076c06000, task ffff88007d10a4e0) [ 934.935047] Stack: [ 934.935113] ffffea0001d82dc0 0000000000000001 0000000004cc4789 ffffffff8131bad6 [ 934.935236] 000000001725db78 ffffffff81ca03c8 ffff88007d10ab48 ffff880076c07c10 [ 934.935355] 0000000076c06010 0000000100000000 ffff8800172269b0 ffff88000d2a4000 [ 934.935471] Call Trace: [ 934.935534] [<ffffffff8131bad6>] ? ocfs2_read_blocks+0x176/0x6c0 [ 934.935600] [<ffffffff8133b0a0>] ? ocfs2_find_actor+0x120/0x120 [ 934.935665] [<ffffffff8133aac7>] ? ocfs2_read_inode_block_full+0x37/0x60 [ 934.935734] [<ffffffff8138ab1f>] ? ocfs2_fast_symlink_readpage+0x2f/0x160 [ 934.935803] [<ffffffff8110fff5>] ? do_read_cache_page+0x85/0x180 [ 934.935868] [<ffffffff8138aaf0>] ? ocfs2_fill_super+0x2500/0x2500 [ 934.935934] [<ffffffff81110149>] ? read_cache_page+0x9/0x20 [ 934.935999] [<ffffffff811565e5>] ? page_getlink+0x25/0x80 [ 934.936062] [<ffffffff8115665b>] ? page_follow_link_light+0x1b/0x30 [ 934.936128] [<ffffffff8115ab27>] ? path_lookupat+0x327/0x6c0 [ 934.936193] [<ffffffff8115aeec>] ? do_path_lookup+0x2c/0xc0 [ 934.936258] [<ffffffff8133b501>] ? ocfs2_inode_revalidate+0x71/0x160 [ 934.936325] [<ffffffff8113210a>] ? handle_pte_fault+0x8a/0x920 [ 934.936390] [<ffffffff8115bd8c>] ? user_path_at_empty+0x5c/0xb0 [ 934.936457] [<ffffffff81066fca>] ? do_page_fault+0x1aa/0x3c0 [ 934.936523] [<ffffffff81150ded>] ? cp_new_stat+0x10d/0x120 [ 934.936588] [<ffffffff81150ee1>] ? vfs_fstatat+0x41/0x80 [ 934.936653] [<ffffffff8115101f>] ? sys_newstat+0x1f/0x50 [ 934.936720] [<ffffffff817cb8a2>] ? system_call_fastpath+0x16/0x1bDoes it gives some better information ? Do you haev some ideas of thinks that could be tested ?
Vincent
Quoted 11 lines · Vincent ETIENNE
Hi
based on current git ( commit
1a9b4993b70fb1884716902774dc9025b457760d) and reverting commitea022dfb3c2a4680483b00eb2fecc9fc4f6091d1commit
ea022dfb3c2a4680483b00eb2fecc9fc4f6091d1Author: Al Viro <[email protected]> Date: Thu May 3 10:14:29 2012 -0400ocfs: simplify symlink handling
Quoted 9 lines
and adding the correction from
suppres the fallocate bug
and lead to no oops. At least immediatly. Will let it run some times
Apply this change and re-run:
fs/ocfs2/symlink.c +0 -0
@@ -57,7 +57,7 @@ static int ocfs2_fast_symlink_readpage(struct file *unused, struct page
*page)
{
struct inode *inode = page->mapping->host;
- struct buffer_head *bh;
+ struct buffer_head *bh = NULL;
int status = ocfs2_read_inode_block(inode, &bh);
struct ocfs2_dinode *fe;
const char *link;_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
Quoted 41 lines · Sunil Mushran
On Thu, Aug 2, 2012 at 12:28 PM, Vincent ETIENNE <[email protected] <mailto:[email protected]>> wrote:
Hi
based on current git ( commit
1a9b4993b70fb1884716902774dc9025b457760d)
and reverting commitea022dfb3c2a4680483b00eb2fecc9fc4f6091d1commit
ea022dfb3c2a4680483b00eb2fecc9fc4f6091d1
Author: Al Viro <[email protected] <mailto:[email protected]>>
Date: Thu May 3 10:14:29 2012 -0400ocfs: simplify symlink handling
and adding the correction from
suppres the fallocate bug
and lead to no oops. At least immediatly. Will let it run some times
Apply this change and re-run:
fs/ocfs2/symlink.c +1
-1@@ -57,7 +57,7 @@static int ocfs2_fast_symlink_readpage (struct file *unused,struct page *page) {struct inode *inode = page->mapping->host; -struct buffer_head *bh; +struct buffer_head *bh =NULL ;int status =ocfs2_read_inode_block (inode, &bh);struct ocfs2_dinode *fe;
Quoted 2 lines
const char *link;
latest head with only your two changes is working here, thanks a lot. Running now flowlessly since a few hours.
Vincent
_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
Thanks for your help.
Quoted 45 lines · Vincent ETIENEN
Le 02/08/2012 23:08, Sunil Mushran a ?crit :
Quoted 11 lines · Vincent ETIENNE
Hi
based on current git ( commit
1a9b4993b70fb1884716902774dc9025b457760d) and reverting commitea022dfb3c2a4680483b00eb2fecc9fc4f6091d1commit
ea022dfb3c2a4680483b00eb2fecc9fc4f6091d1Author: Al Viro <[email protected]> Date: Thu May 3 10:14:29 2012 -0400ocfs: simplify symlink handling
Quoted 9 lines
and adding the correction from
suppres the fallocate bug
and lead to no oops. At least immediatly. Will let it run some times
Apply this change and re-run:
diff --git
a/fs/ocfs2/symlink.cb/fs/ocfs2/symlink.c
index f1fbb4b..66edce7 100644fs/ocfs2/symlink.c +1
-1@@ -57,7 +57,7 @@static int ocfs2_fast_symlink_readpage (struct file *unused,struct page *page) {struct inode *inode = page->mapping->host; -struct buffer_head *bh; +struct buffer_head *bh =NULL ;int status =ocfs2_read_inode_block (inode, &bh);struct ocfs2_dinode *fe;const char *link;
Quoted 8 lines
latest head with only your two changes is working here, thanks a lot.
Running now flowlessly since a few hours.Vincent
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20120803/2fe695af/attachment.html