yet more slab corruption.
Garg. Is there no end to these ?
That kernel is based off 2.6.16rc5-git8
This brings the current count up to 8 different patterns filed against our 2.6.16rc tree in Fedora bugzilla.
(One of them doesn't count as it's against the out-of-tree bcm43xx driver).
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Please do not reply directly to this email. All additional comments should be made in the comments box of this bug report.
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=184310
Summary: Slab corruption in 2.6.15-1.2025_FC5smp
Product: Fedora Core
Version: devel
Platform: i686
OS/Version: Linux
Status: NEW
Severity: normal
Priority: normal
Component: kernel
AssignedTo: [email protected]
ReportedBy: [email protected]
QAContact: [email protected]
CC: [email protected]I am running 2.6.15-1.2025_FC5smp. When I was building gcc with the gcc source tree on a NFS server running RHEL 4 U2, I got
Slab corruption: (Not tainted) start=c574aeb8, len=2048
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02182d0>](release_mem+0xe6/0x1de)
[<c01602a5>] check_poison_obj+0x6a/0x15d [<c01603ba>] cache_alloc_debugcheck_after+0x22/0xf9 [<c02c26d5>] tcp_sendmsg+0x153/0x983 [<c0160bfa>] __kmalloc_track_caller+0xb5/0xbf [<c02c26d5>] tcp_sendmsg+0x153/0x983 [<c029c5db>] __alloc_skb+0x4d/0xf2 [<c02c26d5>] tcp_sendmsg+0x153/0x983 [<c02d980d>] inet_sendmsg+0x35/0x3f [<c02972da>] sock_sendmsg+0xd2/0xec [<c013465b>] autoremove_wake_function+0x0/0x2d [<c0298acb>] kernel_sendmsg+0x26/0x2c [<f8c4fd26>] xs_tcp_send_request+0x107/0x302 [sunrpc] [<f8c4f050>] xprt_transmit+0xd7/0x1c8 [sunrpc] [<f8d386dc>] nfs3_xdr_fhandle+0x0/0x21 [nfs] [<f8c4e1ba>] call_transmit+0x198/0x1cf [sunrpc] [<f8c51dd7>] __rpc_execute+0x7a/0x193 [sunrpc] [<f8c4d2e4>] rpc_call_sync+0x6b/0x91 [sunrpc] [<f8d35f3d>] nfs3_rpc_wrapper+0x1f/0x5b [nfs] [<f8d36939>] nfs3_proc_getattr+0x74/0x96 [nfs] [<f8d2f1f7>] __nfs_revalidate_inode+0x113/0x24e [nfs] [<c01780e5>] __d_lookup+0xbc/0xee [<c02f5136>] _read_unlock_irq+0x5/0x7 [<c014d5b6>] __do_page_cache_readahead+0x10f/0x212 [<f8d2b251>] nfs_lookup_revalidate+0x18a/0x318 [nfs] [<c017837a>] dput+0x31/0x164 [<f8d2b2d3>] nfs_lookup_revalidate+0x20c/0x318 [nfs] [<c0160420>] cache_alloc_debugcheck_after+0x88/0xf9 [<c01d765c>] vsnprintf+0x422/0x461 [<f8c52a58>] rpcauth_lookup_credcache+0x153/0x1ef [sunrpc] [<c01780e5>] __d_lookup+0xbc/0xee [<c016fbd7>] do_lookup+0x11d/0x14d [<c01716da>] __link_path_walk+0x834/0xc7d [<c017bfe2>] mntput_no_expire+0x11/0x6e [<c0171afd>] __link_path_walk+0xc57/0xc7d [<c0171b6c>] link_path_walk+0x49/0xbd [<c0171f07>] do_path_lookup+0x1e3/0x248 [<c01727bf>] __path_lookup_intent_open+0x42/0x72 [<c017283e>] path_lookup_open+0xf/0x13 [<c0172938>] open_namei+0x73/0x54c [<c0162c1f>] do_filp_open+0x1c/0x31 [<c0163ab8>] do_sys_open+0x3e/0xaa [<c0103d81>] syscall_call+0x7/0xb
0b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b ff ff ff ff 0c0: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Prev obj: start=c574a6ac, len=2048
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c029c35c>](kfree_skbmem+0x8/0x63) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Next obj: start=c574b6c4, len=2048
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02182d0>](release_mem+0xe6/0x1de) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Slab corruption: (Not tainted) start=c574b6c4, len=2048
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02182d0>](release_mem+0xe6/0x1de)
[<c01602a5>] check_poison_obj+0x6a/0x15d [<c0170dc4>] lookup_one_len+0x4b/0x59 [<c01603ba>] cache_alloc_debugcheck_after+0x22/0xf9 [<c02181d5>] alloc_tty_struct+0x10/0x25 [<c0160510>] kmem_cache_alloc+0x7f/0x89 [<c02181d5>] alloc_tty_struct+0x10/0x25 [<c02181d5>] alloc_tty_struct+0x10/0x25 [<c0218446>] init_dev+0x7e/0x47a [<c021ac1b>] ptmx_open+0xf8/0x1b6 [<c02f525b>] lock_kernel+0x25/0x34 [<c016ba4e>] chrdev_open+0x104/0x148 [<c016b94a>] chrdev_open+0x0/0x148 [<c0162aac>] __dentry_open+0xc7/0x1ab [<c0162bf4>] nameidata_to_filp+0x19/0x28 [<c0162c2e>] do_filp_open+0x2b/0x31 [<c0163ab8>] do_sys_open+0x3e/0xaa [<c0103d81>] syscall_call+0x7/0xb <3>0b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b ff ff ff ff
0c0: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Prev obj: start=c574aeb8, len=2048
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c029c35c>](kfree_skbmem+0x8/0x63) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Slab corruption: (Not tainted) start=c574a6ac, len=2048
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02182d0>](release_mem+0xe6/0x1de)
[<c01602a5>] check_poison_obj+0x6a/0x15d [<c01603ba>] cache_alloc_debugcheck_after+0x22/0xf9 [<c02c26d5>] tcp_sendmsg+0x153/0x983 [<c0160bfa>] __kmalloc_track_caller+0xb5/0xbf [<c02c26d5>] tcp_sendmsg+0x153/0x983 [<c029c5db>] __alloc_skb+0x4d/0xf2 [<c02c26d5>] tcp_sendmsg+0x153/0x983 [<c02d980d>] inet_sendmsg+0x35/0x3f [<c02972da>] sock_sendmsg+0xd2/0xec [<c013465b>] autoremove_wake_function+0x0/0x2d [<c0298acb>] kernel_sendmsg+0x26/0x2c [<f8c4fd26>] xs_tcp_send_request+0x107/0x302 [sunrpc] [<f8c4f050>] xprt_transmit+0xd7/0x1c8 [sunrpc] [<f8d38c69>] nfs3_xdr_diropargs+0x0/0x2e [nfs] [<f8c4e1ba>] call_transmit+0x198/0x1cf [sunrpc] [<f8c51dd7>] __rpc_execute+0x7a/0x193 [sunrpc] [<f8c4d2e4>] rpc_call_sync+0x6b/0x91 [sunrpc] [<f8d35f3d>] nfs3_rpc_wrapper+0x1f/0x5b [nfs] [<f8d36a59>] nfs3_proc_lookup+0xfe/0x1a6 [nfs] [<c01780e5>] __d_lookup+0xbc/0xee [<c02f5136>] _read_unlock_irq+0x5/0x7 [<c014d5b6>] __do_page_cache_readahead+0x10f/0x212 [<c017837a>] dput+0x31/0x164 [<f8d2b822>] nfs_lookup+0x96/0x104 [nfs] [<c0160420>] cache_alloc_debugcheck_after+0x88/0xf9 [<c01d765c>] vsnprintf+0x422/0x461 [<c0160459>] cache_alloc_debugcheck_after+0xc1/0xf9 [<c017816f>] d_alloc+0x1d/0x1c1 [<c0160510>] kmem_cache_alloc+0x7f/0x89 [<c017816f>] d_alloc+0x1d/0x1c1 [<c0178307>] d_alloc+0x1b5/0x1c1 [<c016fb6c>] do_lookup+0xb2/0x14d [<c01716da>] __link_path_walk+0x834/0xc7d [<c017837a>] dput+0x31/0x164 [<c0171b6c>] link_path_walk+0x49/0xbd [<c0171f07>] do_path_lookup+0x1e3/0x248 [<c0172661>] __user_walk_fd+0x29/0x3a [<c016c470>] vfs_stat_fd+0x15/0x3c [<c016c524>] sys_stat64+0xf/0x23 [<c02f5e9f>] do_page_fault+0x0/0x5e2 [<c0103d81>] syscall_call+0x7/0xb
0b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b ff ff ff ff 0c0: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Next obj: start=c574aeb8, len=2048
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c02181d5>](alloc_tty_struct+0x10/0x25) 000: 01 54 00 00 d4 a4 d5 f7 02 00 00 00 03 54 00 00 010: 7c 85 32 c0 00 00 00 00 01 00 00 00 4c b8 21 c0
Signature · 3 lines
Configure bugmail: https://bugzilla.redhat.com/bugzilla/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
Quoted 8 lines · Dave Jones
Garg. Is there no end to these ?
That kernel is based off 2.6.16rc5-git8This brings the current count up to 8 different patterns filed against our 2.6.16rc tree in Fedora bugzilla.
(One of them doesn't count as it's against the out-of-tree bcm43xx driver).
A use-after-free on size-2048. We wrote -1L and 0L apparently 0x6b8 bytes into the object. That's an awfully large offset for tty_struct - off the end. Sometimes the buffer was used as skb data too.
Unless it was a DMA scribble, CONFIG_DEBUG_PAGEALLOC should catch this.
Quoted 8 lines · Andrew Morton
Quoted 7 lines · Dave Jones
Garg. Is there no end to these ?
That kernel is based off 2.6.16rc5-git8This brings the current count up to 8 different patterns filed against our 2.6.16rc tree in Fedora bugzilla.
(One of them doesn't count as it's against the out-of-tree bcm43xx driver).
We have two similar bug reports to
https://bugzilla.redhat.com/bugzilla/184310, slab corruption in an object freed by release_mem:
https://bugzilla.novell.com/151111 (i386) https://bugzilla.novell.com/154601 (x86_64)
So this bug seems to trigger on different architectures, and with different hardware.
Quoted 3 lines
A use-after-free on size-2048. We wrote -1L and 0L apparently
0x6b8bytes into the object. That's an awfully large offset fortty_struct- off the end. Sometimes the buffer was used as skb data too.
I don't know about offset 0x6b8: 0x6b is POISON_FREE in mm/slab.c, so this is probably a misread. I think the correct offset is 0xbc. Bug 151111 has the corruption at the same offset. On x86_64, the offset is 0x124. This seems to be tty_struct->winsize in all cases, even though I can't tell for sure for most of the reports anymore.
So this could be a tty_struct locking bug -- it's surprising that we are also seeing filesystem corruption, but only in some cases. There was a recent locking fix in this area by Paul Fulghum <[email protected]> from Feb 15 which might be related, but the fix looks good to me:
http://www.kernel.org/hg/linux-2.6/?cs=623e3c38a511.
Quoted 1 line
Unless it was a DMA scribble,
CONFIG_DEBUG_PAGEALLOCshould catch this.
This didn't trigger anything in an overnight run. Any further ideas?
Thanks,
Andreas
Signature · 2 lines
Andreas Gruenbacher <[email protected]> Novell / SUSE Labs
Quoted 9 lines · Andreas Gruenbacher
We have two similar bug reports to
https://bugzilla.redhat.com/bugzilla/184310, slab corruption in an object freed byrelease_mem:https://bugzilla.novell.com/151111 (i386) https://bugzilla.novell.com/154601 (x86_64)So this bug seems to trigger on different architectures, and with different hardware.
I also got a few of these, just reported in a different thread, see http://lkml.org/lkml/2006/4/4/86 for details.
Jan