v2.6.38-rc3+ BUG when calling destroy_inodecache at module unload
Last good Kernel was 2.6.37
I'm doing a "mount" then "unmount". I think root is the only created inode. rmmod is called immediately after "unmount" within a script
if I only do unmount and manually call "modprobe --remove exofs" after a small while all is fine.
I get:
slab error in kmem_cache_destroy(): cache `exofs_inode_cache': Can't free all objects
Call Trace: 77dfde08: [<6007e9a6>] kmem_cache_destroy+0x82/0xca 77dfde38: [<7c1fa3da>] exit_exofs+0x1a/0x1c [exofs] 77dfde48: [<60054c10>] sys_delete_module+0x1b9/0x217 77dfdee8: [<60014d60>] handle_syscall+0x58/0x70 77dfdf08: [<60024163>] userspace+0x2dd/0x38a 77dfdfc8: [<600126af>] fork_handler+0x62/0x69
The UML Kernel also crashes after this message, with:
Modules linked in: nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc cryptomgr aead crc32c crypto_hash crypto_algapi iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod binfmt_misc [last unloaded: libosd] Pid: 6, comm: rcu_kthread Not tainted 2.6.38-rc3+ RIP: 0033:[<000000007c1fa0e7>] RSP: 000000007943be18 EFLAGS: 00010246 RAX: 000000007943a000 RBX: 000000007937bb80 RCX: 0000000000000095 RDX: 000000007937c8b8 RSI: 0000000077fb6c80 RDI: 000000007937bb80 RBP: 000000007943be40 R08: 000000007943be10 R09: 000000007943a000 R10: 0000000000000000 R11: 0000000000000000 R12: 00000000795123e0 R13: 0000000000000001 R14: 0000000000000000 R15: 000000000000000a Call Trace: 602678f8: [<600144ed>] segv+0x70/0x212 60267928: [<6001cd9e>] ubd_intr+0x72/0xdf 60267988: [<601b778e>] _raw_spin_unlock_irqrestore+0x18/0x1c 602679d8: [<600146ee>] segv_handler+0x5f/0x65 60267a08: [<60021488>] sig_handler_common+0x84/0x98 60267ab0: [<60130926>] strncpy+0xf/0x27 60267b38: [<600215ce>] sig_handler+0x30/0x3b 60267b58: [<60021800>] handle_signal+0x6d/0xa3 60267ba8: [<60023180>] hard_handler+0x10/0x14 Kernel panic - not syncing: Segfault with no mm Call Trace: 602677f8: [<601b52b1>] panic+0xea/0x1e6 60267818: [<6007e299>] kmem_cache_free+0x54/0x5f 60267850: [<6005342e>] __module_text_address+0xd/0x53 60267868: [<6005347d>] is_module_text_address+0x9/0x11 60267878: [<6004290c>] __kernel_text_address+0x65/0x6b 60267880: [<60023180>] hard_handler+0x10/0x14 60267898: [<6001345e>] show_trace+0x8e/0x95 602678c8: [<60026c40>] show_regs+0x2b/0x2f 602678f8: [<60014577>] segv+0xfa/0x212 60267928: [<6001cd9e>] ubd_intr+0x72/0xdf 60267988: [<601b778e>] _raw_spin_unlock_irqrestore+0x18/0x1c 602679d8: [<600146ee>] segv_handler+0x5f/0x65 60267a08: [<60021488>] sig_handler_common+0x84/0x98 60267ab0: [<60130926>] strncpy+0xf/0x27 60267b38: [<600215ce>] sig_handler+0x30/0x3b 60267b58: [<60021800>] handle_signal+0x6d/0xa3 60267ba8: [<60023180>] hard_handler+0x10/0x14 Modules linked in: nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc cryptomgr aead crc32c crypto_hash crypto_algapi iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod binfmt_misc [last unloaded: libosd] Pid: 6, comm: rcu_kthread Not tainted 2.6.38-rc3+ RIP: 0033:[<0000003ea3832ad7>] RSP: 00007fff63338e38 EFLAGS: 00000202 RAX: 0000000000000000 RBX: 0000000000000219 RCX: ffffffffffffffff RDX: 0000000000000000 RSI: 0000000000000013 RDI: 0000000000000219 RBP: 00007fff63338e70 R08: 0000000000000000 R09: 00007fff63338e70 R10: 00007fff63338be0 R11: 0000000000000202 R12: 0000000000000215 R13: 00007fe54ee756a8 R14: 00007fff63339090 R15: 00007fff63339928 Call Trace: 60267788: [<6001485b>] panic_exit+0x2f/0x45 602677a8: [<60048ad6>] notifier_call_chain+0x32/0x5e 602677e8: [<60048b24>] atomic_notifier_call_chain+0x13/0x15 602677f8: [<601b52cc>] panic+0x105/0x1e6 60267818: [<6007e299>] kmem_cache_free+0x54/0x5f 60267850: [<6005342e>] __module_text_address+0xd/0x53 60267868: [<6005347d>] is_module_text_address+0x9/0x11 60267878: [<6004290c>] __kernel_text_address+0x65/0x6b 60267880: [<60023180>] hard_handler+0x10/0x14 60267898: [<6001345e>] show_trace+0x8e/0x95 602678c8: [<60026c40>] show_regs+0x2b/0x2f 602678f8: [<60014577>] segv+0xfa/0x212 60267928: [<6001cd9e>] ubd_intr+0x72/0xdf 60267988: [<601b778e>] _raw_spin_unlock_irqrestore+0x18/0x1c 602679d8: [<600146ee>] segv_handler+0x5f/0x65 60267a08: [<60021488>] sig_handler_common+0x84/0x98 60267ab0: [<60130926>] strncpy+0xf/0x27 60267b38: [<600215ce>] sig_handler+0x30/0x3b 60267b58: [<60021800>] handle_signal+0x6d/0xa3 60267ba8: [<60023180>] hard_handler+0x10/0x14
Thanks
Boaz
Quoted 9 lines · Boaz Harrosh
Last good Kernel was 2.6.37
I'm doing a "mount" then "unmount". I think root is the only created inode. rmmod is called immediately after "unmount" within a scriptif I only do unmount and manually call "modprobe --remove exofs" after a small while all is fine.
I get:
slab error inkmem_cache_destroy(): cache `exofs_inode_cache': Can't free all objects
Quoted 8 lines
Call Trace: 77dfde08: [<6007e9a6>] kmem_cache_destroy+0x82/0xca 77dfde38: [<7c1fa3da>] exit_exofs+0x1a/0x1c [exofs] 77dfde48: [<60054c10>] sys_delete_module+0x1b9/0x217 77dfdee8: [<60014d60>] handle_syscall+0x58/0x70 77dfdf08: [<60024163>] userspace+0x2dd/0x38a 77dfdfc8: [<600126af>] fork_handler+0x62/0x69
I also get a similar error when testing ext4 and a bug is opened there.
https://bugzilla.kernel.org/show_bug.cgi?id=27652
And I have done some simple investigation for ext4 and It looks as if now with the new *fs_i_callback doesn't free the inode to *fs_inode_cache immediately. So the old logic will destroy the inode cache before we free all the inode object.
Since there are more than one fs affected by this, we may need to find a way in the VFS.
Regards,
Tao
Quoted 81 lines
The UML Kernel also crashes after this message, with:
Modules linked in: nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc cryptomgr aead crc32c crypto_hash crypto_algapi iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod binfmt_misc [last unloaded: libosd] Pid: 6, comm: rcu_kthread Not tainted 2.6.38-rc3+ RIP: 0033:[<000000007c1fa0e7>] RSP: 000000007943be18 EFLAGS: 00010246 RAX: 000000007943a000 RBX: 000000007937bb80 RCX: 0000000000000095 RDX: 000000007937c8b8 RSI: 0000000077fb6c80 RDI: 000000007937bb80 RBP: 000000007943be40 R08: 000000007943be10 R09: 000000007943a000 R10: 0000000000000000 R11: 0000000000000000 R12: 00000000795123e0 R13: 0000000000000001 R14: 0000000000000000 R15: 000000000000000a Call Trace: 602678f8: [<600144ed>] segv+0x70/0x212 60267928: [<6001cd9e>] ubd_intr+0x72/0xdf 60267988: [<601b778e>] _raw_spin_unlock_irqrestore+0x18/0x1c 602679d8: [<600146ee>] segv_handler+0x5f/0x65 60267a08: [<60021488>] sig_handler_common+0x84/0x98 60267ab0: [<60130926>] strncpy+0xf/0x27 60267b38: [<600215ce>] sig_handler+0x30/0x3b 60267b58: [<60021800>] handle_signal+0x6d/0xa3 60267ba8: [<60023180>] hard_handler+0x10/0x14 Kernel panic - not syncing: Segfault with no mm Call Trace: 602677f8: [<601b52b1>] panic+0xea/0x1e6 60267818: [<6007e299>] kmem_cache_free+0x54/0x5f 60267850: [<6005342e>] __module_text_address+0xd/0x53 60267868: [<6005347d>] is_module_text_address+0x9/0x11 60267878: [<6004290c>] __kernel_text_address+0x65/0x6b 60267880: [<60023180>] hard_handler+0x10/0x14 60267898: [<6001345e>] show_trace+0x8e/0x95 602678c8: [<60026c40>] show_regs+0x2b/0x2f 602678f8: [<60014577>] segv+0xfa/0x212 60267928: [<6001cd9e>] ubd_intr+0x72/0xdf 60267988: [<601b778e>] _raw_spin_unlock_irqrestore+0x18/0x1c 602679d8: [<600146ee>] segv_handler+0x5f/0x65 60267a08: [<60021488>] sig_handler_common+0x84/0x98 60267ab0: [<60130926>] strncpy+0xf/0x27 60267b38: [<600215ce>] sig_handler+0x30/0x3b 60267b58: [<60021800>] handle_signal+0x6d/0xa3 60267ba8: [<60023180>] hard_handler+0x10/0x14 Modules linked in: nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc cryptomgr aead crc32c crypto_hash crypto_algapi iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod binfmt_misc [last unloaded: libosd] Pid: 6, comm: rcu_kthread Not tainted 2.6.38-rc3+ RIP: 0033:[<0000003ea3832ad7>] RSP: 00007fff63338e38 EFLAGS: 00000202 RAX: 0000000000000000 RBX: 0000000000000219 RCX: ffffffffffffffff RDX: 0000000000000000 RSI: 0000000000000013 RDI: 0000000000000219 RBP: 00007fff63338e70 R08: 0000000000000000 R09: 00007fff63338e70 R10: 00007fff63338be0 R11: 0000000000000202 R12: 0000000000000215 R13: 00007fe54ee756a8 R14: 00007fff63339090 R15: 00007fff63339928 Call Trace: 60267788: [<6001485b>] panic_exit+0x2f/0x45 602677a8: [<60048ad6>] notifier_call_chain+0x32/0x5e 602677e8: [<60048b24>] atomic_notifier_call_chain+0x13/0x15 602677f8: [<601b52cc>] panic+0x105/0x1e6 60267818: [<6007e299>] kmem_cache_free+0x54/0x5f 60267850: [<6005342e>] __module_text_address+0xd/0x53 60267868: [<6005347d>] is_module_text_address+0x9/0x11 60267878: [<6004290c>] __kernel_text_address+0x65/0x6b 60267880: [<60023180>] hard_handler+0x10/0x14 60267898: [<6001345e>] show_trace+0x8e/0x95 602678c8: [<60026c40>] show_regs+0x2b/0x2f 602678f8: [<60014577>] segv+0xfa/0x212 60267928: [<6001cd9e>] ubd_intr+0x72/0xdf 60267988: [<601b778e>] _raw_spin_unlock_irqrestore+0x18/0x1c 602679d8: [<600146ee>] segv_handler+0x5f/0x65 60267a08: [<60021488>] sig_handler_common+0x84/0x98 60267ab0: [<60130926>] strncpy+0xf/0x27 60267b38: [<600215ce>] sig_handler+0x30/0x3b 60267b58: [<60021800>] handle_signal+0x6d/0xa3 60267ba8: [<60023180>] hard_handler+0x10/0x14Thanks
Boaz
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Excerpts from Tao Ma's message of 2011-02-04 03:36:59 -0500:
Quoted 18 lines · Boaz Harrosh
Quoted 9 lines
Last good Kernel was 2.6.37
I'm doing a "mount" then "unmount". I think root is the only created inode. rmmod is called immediately after "unmount" within a scriptif I only do unmount and manually call "modprobe --remove exofs" after a small while all is fine.
I get:
slab error inkmem_cache_destroy(): cache `exofs_inode_cache': Can't free all objectsQuoted 8 lines
Call Trace: 77dfde08: [<6007e9a6>] kmem_cache_destroy+0x82/0xca 77dfde38: [<7c1fa3da>] exit_exofs+0x1a/0x1c [exofs] 77dfde48: [<60054c10>] sys_delete_module+0x1b9/0x217 77dfdee8: [<60014d60>] handle_syscall+0x58/0x70 77dfdf08: [<60024163>] userspace+0x2dd/0x38a 77dfdfc8: [<600126af>] fork_handler+0x62/0x69
Quoted 7 lines
I also get a similar error when testing ext4 and a bug is opened there.
https://bugzilla.kernel.org/show_bug.cgi?id=27652
And I have done some simple investigation for ext4 and It looks as if now with the new
*fs_i_callbackdoesn't free the inode to*fs_inode_cacheimmediately. So the old logic will destroy the inode cache before we free all the inode object.Since there are more than one fs affected by this, we may need to find a way in the VFS.
Sounds like we just need a synchronize_rcu call before we delete the cache?
-chris
Quoted 23 lines · Chris Mason
Excerpts from Tao Ma's message of 2011-02-04 03:36:59 -0500:
Quoted 22 lines · Boaz Harrosh
Quoted 9 lines
Last good Kernel was 2.6.37
I'm doing a "mount" then "unmount". I think root is the only created inode. rmmod is called immediately after "unmount" within a scriptif I only do unmount and manually call "modprobe --remove exofs" after a small while all is fine.
I get:
slab error inkmem_cache_destroy(): cache `exofs_inode_cache': Can't free all objectsQuoted 8 lines
Call Trace: 77dfde08: [<6007e9a6>] kmem_cache_destroy+0x82/0xca 77dfde38: [<7c1fa3da>] exit_exofs+0x1a/0x1c [exofs] 77dfde48: [<60054c10>] sys_delete_module+0x1b9/0x217 77dfdee8: [<60014d60>] handle_syscall+0x58/0x70 77dfdf08: [<60024163>] userspace+0x2dd/0x38a 77dfdfc8: [<600126af>] fork_handler+0x62/0x69I also get a similar error when testing ext4 and a bug is opened there.
Quoted 8 lines
Quoted 3 lines
And I have done some simple investigation for ext4 and It looks as if now with the new
*fs_i_callbackdoesn't free the inode to*fs_inode_cacheimmediately. So the old logic will destroy the inode cache before we free all the inode object.Since there are more than one fs affected by this, we may need to find a way in the VFS.
Sounds like we just need a
synchronize_rcucall before we delete the cache?-chris
Hi Al, Nick.
Al please look into this issue. Absolutely all filesystems should be affected. Tao Ma has attempted the below fix, but it does not help. Exact same trace with his patch applied.
If you unmount and immediately rmmod the filesystem it will crash because of those RCU freed objects at umount, like the root inode. Nick is not responding, I'd try to fix it, but I don't know how.
---
Quoted 32 lines
From: Tao Ma <[email protected]>
In fa0d7e3, we use rcu free inode instead of freeing the inode directly. It causes a problem when we rmmod immediately after we umount the volume[1].
So we need to call
synchronize_rcuafter wekill_sbso that the inode is freed before we do rmmod. The idea is inspired by Chris Mason[2]. I tested with ext4 by umount+rmmod and it doesn't show any error by now.1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2Cc Nick Piggin <[email protected]>Cc Al Viro <[email protected]>Cc Chris Mason <[email protected]>Cc Boaz Harrosh <[email protected]>Signed-off-by Tao Ma <[email protected]>--- fs/super.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)fs/super.c +1
-0@@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)struct file_system_type *fs = s->s_type;if (atomic_dec_and_test (&s->s_active)) { fs->kill_sb (s); +/*
fs/super.c void deactivate_locked_super(struct super_block *s)
Quoted 10 lines
fs/super.c +6
-1+* We need to synchronize rcu here so that +* the delayed rcu inode free can be executed +* before we put_super. +* https://bugzilla.kernel.org/show_bug.cgi?id=27652 +*/ +synchronize_rcu ();put_filesystem (fs);put_super (s); }else { -- 1.6.3.GIT
Thanks
Boaz
Hi Boaz,
Quoted 35 lines · Boaz Harrosh
Quoted 33 lines · Chris Mason
Excerpts from Tao Ma's message of 2011-02-04 03:36:59 -0500:
Quoted 20 lines · Boaz Harrosh
Quoted 10 lines
Last good Kernel was 2.6.37
I'm doing a "mount" then "unmount". I think root is the only created inode. rmmod is called immediately after "unmount" within a scriptif I only do unmount and manually call "modprobe --remove exofs" after a small while all is fine.
I get:
slab error inkmem_cache_destroy(): cache `exofs_inode_cache': Can't free all objectsQuoted 9 lines
Call Trace: 77dfde08: [<6007e9a6>] kmem_cache_destroy+0x82/0xca 77dfde38: [<7c1fa3da>] exit_exofs+0x1a/0x1c [exofs] 77dfde48: [<60054c10>] sys_delete_module+0x1b9/0x217 77dfdee8: [<60014d60>] handle_syscall+0x58/0x70 77dfdf08: [<60024163>] userspace+0x2dd/0x38a 77dfdfc8: [<600126af>] fork_handler+0x62/0x69Quoted 8 lines
I also get a similar error when testing ext4 and a bug is opened there.
https://bugzilla.kernel.org/show_bug.cgi?id=27652
And I have done some simple investigation for ext4 and It looks as if now with the new
*fs_i_callbackdoesn't free the inode to*fs_inode_cacheimmediately. So the old logic will destroy the inode cache before we free all the inode object.Since there are more than one fs affected by this, we may need to find a way in the VFS.
Sounds like we just need a
synchronize_rcucall before we delete the cache?
Quoted 8 lines
Quoted 2 lines
-chris
Hi Al, Nick.
Al please look into this issue. Absolutely all filesystems should be affected. Tao Ma has attempted the below fix, but it does not help. Exact same trace with his patch applied.
I am in vacation so I could't reach my test box today. I have done some simple tracing yesterday, and it looked that although synchronize_rcu is called, I can still get ext4_i_callback after it. So the reason may be:
1. synchronize_rcu doesn't work as we expected.
2. the inode free rcu doesn't work as Nick expected.
I will go to the office tomorrow and do more test and debug there. Hope to find something more.
Quoted 4 lines
If you unmount and immediately rmmod the filesystem it will crash because of those RCU freed objects at umount, like the root inode. Nick is not responding, I'd try to fix it, but I don't know how.
I raised the error to Nick on Jan. 19, about 3 weeks ago.
http://marc.info/?l=linux-ext4&m=129542001031750&w=2
But it seems that he is quite busy these days. It is still rc3 and we have a lot of time before the final release. So no panic here. ;) Finally, I just tried to fix it recently. But it doesn't work. :( I will continue to work on it before Al or Nick respond with a perfect patch. :)
Regards,
Tao