VFS: call synchronize_rcu after kill_sb.
PATCH 1/1 VFS: call synchronize_rcu after kill_sb.
From: Tao Ma <[email protected]>
In fa0d7e3, we use rcu free inode instead of freeing the inode directly. It causes a problem when we rmmod immediately after we umount the volume[1].
So we need to call synchronize_rcu after we kill_sb so that the inode is freed before we do rmmod. The idea is inspired by Chris Mason[2]. I tested with ext4 by umount+rmmod and it doesn't show any error by now.
1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2
--- fs/super.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)
fs/super.c +7 -0
@@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)struct file_system_type *fs = s->s_type;if (atomic_dec_and_test (&s->s_active)) { fs->kill_sb (s); +/* +* We need to synchronize rcu here so that +* the delayed rcu inode free can be executed +* before we put_super. +* https://bugzilla.kernel.org/show_bug.cgi?id=27652 +*/ +synchronize_rcu ();put_filesystem (fs);put_super (s); }else {
Signature · 1 line
1.6.3.GIT
Quoted 31 lines · Tao Ma
From: Tao Ma <[email protected]>
In fa0d7e3, we use rcu free inode instead of freeing the inode directly. It causes a problem when we rmmod immediately after we umount the volume[1].
So we need to call
synchronize_rcuafter wekill_sbso that the inode is freed before we do rmmod. The idea is inspired by Chris Mason[2]. I tested with ext4 by umount+rmmod and it doesn't show any error by now.1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2Cc Nick Piggin <[email protected]>Cc Al Viro <[email protected]>Cc Chris Mason <[email protected]>Cc Boaz Harrosh <[email protected]>Signed-off-by Tao Ma <[email protected]>--- fs/super.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)fs/super.c +0
-0@@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)struct file_system_type *fs = s->s_type;if (atomic_dec_and_test (&s->s_active)) { fs->kill_sb (s);
fs/super.c void deactivate_locked_super(struct super_block *s) · line 189
Quoted 10 lines
fs/super.c +7
-0+/* +* We need to synchronize rcu here so that +* the delayed rcu inode free can be executed +* before we put_super. +* https://bugzilla.kernel.org/show_bug.cgi?id=27652 +*/ +synchronize_rcu ();put_filesystem (fs);put_super (s); }else {
Sorry for not testing sooner.
The above does not work I still get the exact same crash!!
Looking at the code for synchronize_rcu() it looks like it might not be enough. It looks like all it does is a memory barrier. But we need something that will actually pump these pending releases.
(I might be way off here)
BTW after I get the Warning from the kmem_cache_destroy:
slab error in kmem_cache_destroy(): cache `exofs_inode_cache': Can't free all objects
Call Trace: 754efe08: [<6007e9a6>] kmem_cache_destroy+0x82/0xca 754efe38: [<7a9296ba>] exit_exofs+0x1a/0x1c [exofs] 754efe48: [<60054c10>] sys_delete_module+0x1b9/0x217 754efee8: [<60014d60>] handle_syscall+0x58/0x70 754eff08: [<60024163>] userspace+0x2dd/0x38a 754effc8: [<600126af>] fork_handler+0x62/0x69
I also get a Kernel crash. I suspect it's when finally these free_rcu come and the module (and kmem_cache) are no longer there.
What to do? Nick?
Boaz
Quoted 64 lines · Boaz Harrosh
Quoted 31 lines · Tao Ma
From: Tao Ma <[email protected]>
In fa0d7e3, we use rcu free inode instead of freeing the inode directly. It causes a problem when we rmmod immediately after we umount the volume[1].
So we need to call
synchronize_rcuafter wekill_sbso that the inode is freed before we do rmmod. The idea is inspired by Chris Mason[2]. I tested with ext4 by umount+rmmod and it doesn't show any error by now.1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2Cc Nick Piggin <[email protected]>Cc Al Viro <[email protected]>Cc Chris Mason <[email protected]>Cc Boaz Harrosh <[email protected]>Signed-off-by Tao Ma <[email protected]>--- fs/super.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)fs/super.c +0
-0@@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)struct file_system_type *fs = s->s_type;if (atomic_dec_and_test (&s->s_active)) { fs->kill_sb (s);fs/super.c void deactivate_locked_super(struct super_block *s) · line 189
Quoted 10 lines
fs/super.c +7
-0+/* +* We need to synchronize rcu here so that +* the delayed rcu inode free can be executed +* before we put_super. +* https://bugzilla.kernel.org/show_bug.cgi?id=27652 +*/ +synchronize_rcu ();put_filesystem (fs);put_super (s); }else {Sorry for not testing sooner.
The above does not work I still get the exact same crash!!
Looking at the code for
synchronize_rcu()it looks like it might not be enough. It looks like all it does is a memory barrier. But we need something that will actually pump these pending releases.
(I might be way off here)BTW after I get the Warning from the
kmem_cache_destroy:
slab error inkmem_cache_destroy(): cache `exofs_inode_cache': Can't free all objectsCall Trace: 754efe08: [<6007e9a6>] kmem_cache_destroy+0x82/0xca 754efe38: [<7a9296ba>] exit_exofs+0x1a/0x1c [exofs] 754efe48: [<60054c10>] sys_delete_module+0x1b9/0x217 754efee8: [<60014d60>] handle_syscall+0x58/0x70 754eff08: [<60024163>] userspace+0x2dd/0x38a 754effc8: [<600126af>] fork_handler+0x62/0x69
Quoted 4 lines
I also get a Kernel crash. I suspect it's when finally these
free_rcucome and the module (andkmem_cache) are no longer there.What to do? Nick?
http://lwn.net/Articles/217484/ explains how to wait for rcu callback to finish
-aneesh
Quoted 37 lines · Aneesh Kumar K. V
Quoted 36 lines · Boaz Harrosh
Quoted 35 lines · Tao Ma
From: Tao Ma <[email protected]>
In fa0d7e3, we use rcu free inode instead of freeing the inode directly. It causes a problem when we rmmod immediately after we umount the volume[1].
So we need to call
synchronize_rcuafter wekill_sbso that the inode is freed before we do rmmod. The idea is inspired by Chris Mason[2]. I tested with ext4 by umount+rmmod and it doesn't show any error by now.1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2Cc Nick Piggin <[email protected]>Cc Al Viro <[email protected]>Cc Chris Mason <[email protected]>Cc Boaz Harrosh <[email protected]>Signed-off-by Tao Ma <[email protected]>--- fs/super.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)fs/super.c +4
-0@@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)struct file_system_type *fs = s->s_type;if (atomic_dec_and_test (&s->s_active)) { fs->kill_sb (s); +/* +* We need to synchronize rcu here so that +* the delayed rcu inode free can be executed +* before we put_super.
Quoted 8 lines
Quoted 8 lines
fs/super.c void deactivate_locked_super(struct super_block *s) · line 189
Quoted 6 lines
fs/super.c +3
-0+* https://bugzilla.kernel.org/show_bug.cgi?id=27652 +*/ +synchronize_rcu ();put_filesystem (fs);put_super (s); }else {
<>
Quoted 4 lines
http://lwn.net/Articles/217484/ explains how to wait for rcu callback to finish
-aneesh
Yes thanks Aneesh, rcu_barrier does the trick
---
From: Boaz Harrosh <[email protected]>
In fa0d7e3, we use rcu free inode instead of freeing the inode directly. It causes a problem when we rmmod immediately after we umount the volume[1].
So we need to call rcu_barrier after we kill_sb so that the inode is freed before we do rmmod. The idea is inspired by Aneesh Kumar. rcu_barrier will wait for all callbacks to end before preceding. The original patch was done by Tao Ma, but synchronize_rcu() is not enough here.
1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2
---
git diff --stat -p -M fs/super.c
fs/super.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
fs/super.c +7 -0
@@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)struct file_system_type *fs = s->s_type;if (atomic_dec_and_test (&s->s_active)) { fs->kill_sb (s); +/* +* We need to synchronize rcu here so that +* the delayed rcu inode free can be executed +* before we put_super. +* https://bugzilla.kernel.org/show_bug.cgi?id=27652 +*/ +rcu_barrier ();put_filesystem (fs);put_super (s); }else {
Tao Ma
Tested-byQuoted 70 lines · Boaz Harrosh
Quoted 40 lines · Aneesh Kumar K. V
Quoted 38 lines · Boaz Harrosh
Quoted 36 lines · Tao Ma
From: Tao Ma<[email protected]>
In fa0d7e3, we use rcu free inode instead of freeing the inode directly. It causes a problem when we rmmod immediately after we umount the volume[1].
So we need to call
synchronize_rcuafter wekill_sbso that the inode is freed before we do rmmod. The idea is inspired by Chris Mason[2]. I tested with ext4 by umount+rmmod and it doesn't show any error by now.1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2Cc Nick Piggin<[email protected]>Cc Al Viro<[email protected]>Cc Chris Mason<[email protected]>Cc Boaz Harrosh<[email protected]>Signed-off-by Tao Ma<[email protected]>--- fs/super.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)fs/super.c +5
-0@@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)struct file_system_type *fs = s->s_type;if (atomic_dec_and_test (&s->s_active)) { fs->kill_sb (s); +/* +* We need to synchronize rcu here so that +* the delayed rcu inode free can be executed +* before we put_super. +* https://bugzilla.kernel.org/show_bug.cgi?id=27652 Quoted 8 lines
Quoted 8 lines
fs/super.c void deactivate_locked_super(struct super_block *s) · line 189
Quoted 6 lines
fs/super.c +2
-0+*/ +synchronize_rcu ();put_filesystem (fs);put_super (s); }else {<>
Quoted 4 lines
http://lwn.net/Articles/217484/ explains how to wait for rcu callback to finish
-aneesh
Yes thanks Aneesh,
rcu_barrierdoes the trick
---
From: Boaz Harrosh<[email protected]>In fa0d7e3, we use rcu free inode instead of freeing the inode directly. It causes a problem when we rmmod immediately after we umount the volume[1].
So we need to call
rcu_barrierafter wekill_sbso that the inode is freed before we do rmmod. The idea is inspired by Aneesh Kumar.rcu_barrierwill wait for all callbacks to end before preceding. The original patch was done by Tao Ma, butsynchronize_rcu()is not enough here.
Quoted 9 lines
1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2Cc Nick Piggin<[email protected]>Cc Al Viro<[email protected]>Cc Chris Mason<[email protected]>Cc Tao Ma<[email protected]>Signed-off-by Boaz Harrosh<[email protected]>
It works now in my ext4 test box. Thanks for your work.
Quoted 28 lines
---
git diff --stat -p -Mfs/super.cfs/super.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)fs/super.c +7
-1@@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)struct file_system_type *fs = s->s_type;if (atomic_dec_and_test (&s->s_active)) { fs->kill_sb (s); +/* +* We need to synchronize rcu here so that +* the delayed rcu inode free can be executed +* before we put_super. +* https://bugzilla.kernel.org/show_bug.cgi?id=27652 +*/ +rcu_barrier ();put_filesystem (fs);put_super (s); }else { --To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
fs/super.c void deactivate_locked_super(struct super_block *s) · line 185
Quoted 6 lines
fs/super.c +6
-0+/* +* We need to synchronize rcu here so that +* the delayed rcu inode free can be executed +* before we put_super. +* https://bugzilla.kernel.org/show_bug.cgi?id=27652 +*/
URLs in comments are not useful, descriptions of the issues in comments should be complete enough to understand the issue. I think the comment without the url is enough, though.
PATCH 1/1 VFS: call rcu_barrier after kill_sb.
In fa0d7e3, we use rcu free inode instead of freeing the inode directly. It causes a crash when we rmmod immediately after we umount the volume[1].
So we need to call rcu_barrier after we kill_sb so that the inode is freed before we do rmmod. The idea is inspired by Aneesh Kumar. rcu_barrier will wait for all callbacks to end before preceding. The original patch was done by Tao Ma, but synchronize_rcu() is not enough here.
1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
--- fs/super.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-)
fs/super.c +5 -0
@@ -177,6 +177,11 @@ void deactivate_locked_super(struct super_block *s)struct file_system_type *fs = s->s_type;if (atomic_dec_and_test (&s->s_active)) { fs->kill_sb (s); +/* +* We need to call rcu_barrier so all the delayed rcu free +* inodes are flushed before we release the fs module. +*/ +rcu_barrier ();put_filesystem (fs);put_super (s); }else {
Signature · 1 line
1.7.2.3