crash in filesytem during reboot . (and proposed patch)

DISCUSSION Andrew Morton 2 participants 2 messages Jun 22, 2012 – Jun 23, 2012 1 message not in the archive

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Andrew Morton

author

2012-06-22 21:29 UTC #1 lore raw

Quoted 4 lines · On Fri, 15 Jun 2012 11:12:09 -0700 Sadasivan Shaiju <[email protected]> wrote:

Hi

Your email is quadruple-spaced. Please, fix that.

Quoted 59 lines

I  am  getting  the  following  crashes during  a  reboot of   the  system
.  It  looks  like  a  race  condition   during  unmount .
<4>Call Trace:
<4>[] clear_inode+0x28/0xe8
<4>[] generic_drop_inode+0x3c/0xa8
<4>[] d_kill+0x4c/0x78
<4>[] __shrink_dcache_sb+0x258/0x360
<4>[] shrink_dcache_parent+0x140/0x190
<4>[] proc_flush_task+0xac/0x2e8
<4>[] release_task+0x80/0x4c0
<4>[] wait_consider_task+0x608/0xa80
<4>[] do_wait+0x10c/0x2b8
<4>[] SyS_wait4+0x88/0x120
<4>[] compat_sys_wait4+0xc8/0xd0
<4>[] handle_sysn32+0x44/0x84
Call Trace:
[] file_ra_state_init+0x0/0x20
[] __dentry_open+0x26c/0x3d0
[] do_filp_open+0x70c/0xbc8
[] do_sys_open+0x78/0x1e0
[] handle_sysn32+0x44/0x84

Call Trace:
[<ffffffff812ae3e4>] iput+0x3c/0x88
[<ffffffff812aaa84>] d_kill+0x4c/0x78
[<ffffffff812aad08>] __shrink_dcache_sb+0x258/0x360
[<ffffffff812ab300>] shrink_dcache_parent+0x140/0x190
[<ffffffff812eea14>] proc_flush_task+0xac/0x2e8
[<ffffffff811e6538>] release_task+0x80/0x4c0
[<ffffffff811e80c8>] do_exit+0x6f8/0x908
[<ffffffff8121dee8>] unregister_module_notifier+0x0/0x10

Call Trace:
[<ffffffff812ae3e4>] iput+0x3c/0x88
[<ffffffff812aaa84>] d_kill+0x4c/0x78
[<ffffffff812ab6b8>] dput+0x120/0x220
[<ffffffff812a0f1c>] do_lookup+0xdc/0x210 [<ffffffff812a33e8>]
__link_path_walk+0x910/0x1408 [<ffffffff812a4194>] path_walk+0x64/0x108
[<ffffffff812a4350>] do_path_lookup+0x60/0x68 [<ffffffff812a519c>]
do_filp_open+0xdc/0xbc8 [<ffffffff81293768>] do_sys_open+0x78/0x1e0
[<ffffffff81103844>] handle_sysn32+0x44/0x84
...
I am thinking of putting the following fix in
shrink_dcache_parent() . Please let me know is there any problem
with this fix .
...
fs/dcache.c 2012-06-11 17:10:33.000000000 -0700 renamed from fs/dcache.c 2012-05-30 15:59:18.000000000 -0700 +0 -1
@@ -881,8 +881,14 @@
        struct super_block *sb = parent->d_sb;
        int found;
 
-       while ((found = select_parent(parent)) != 0)

fs/dcache.c 2012-06-11 17:10:33.000000000 -0700 line 892

Quoted 10 lines

fs/dcache.c 2012-06-11 17:10:33.000000000 -0700 renamed from fs/dcache.c 2012-05-30 15:59:18.000000000 -0700 +8 -1

-               __shrink_dcache_sb(sb, &found, 0);
+       while ((found = select_parent(parent)) != 0) {
+               if (down_read_trylock(&sb->s_umount)) {
+                        if ((sb->s_root != NULL)) {
+                               __shrink_dcache_sb(sb, &found, 0);
+                }
+                  up_read(&sb->s_umount);
+             }
+        }
 }

Please fully describe the race which you believe you have found. What races against what?

Please also confirm that the bug is still present in current kernels - 2.6.32 is rather old.

Thanks.

Sadasivan Shaiju

2012-06-23 00:53 UTC #2 lore raw

Hi Andrew,

Please see inline .
-----Original Message-----
From: Andrew Morton [mailto:[email protected]]
Sent: Friday, June 22, 2012 2:30 PM
To: Sadasivan Shaiju
Cc: [email protected]
Subject: Re: crash in filesytem during reboot . (and proposed patch)

Quoted 4 lines · On Fri, 15 Jun 2012 11:12:09 -0700 Sadasivan Shaiju <[email protected]> wrote:

Hi

Your email is quadruple-spaced. Please, fix that.

Sure I will fix this .

Quoted 1 line

I am getting the following crashes during a reboot of the

system

Quoted 40 lines

. It looks like a race condition during unmount .

<4>Call Trace:
<4>[] clear_inode+0x28/0xe8
<4>[] generic_drop_inode+0x3c/0xa8
<4>[] d_kill+0x4c/0x78
<4>[] __shrink_dcache_sb+0x258/0x360
<4>[] shrink_dcache_parent+0x140/0x190 <4>[]

proc_flush_task+0xac/0x2e8 <4>[] release_task+0x80/0x4c0 <4>[]
wait_consider_task+0x608/0xa80 <4>[] do_wait+0x10c/0x2b8 <4>[]
SyS_wait4+0x88/0x120 <4>[] compat_sys_wait4+0xc8/0xd0 <4>[]
handle_sysn32+0x44/0x84

Call Trace:
[] file_ra_state_init+0x0/0x20
[] __dentry_open+0x26c/0x3d0
[] do_filp_open+0x70c/0xbc8
[] do_sys_open+0x78/0x1e0
[] handle_sysn32+0x44/0x84

Call Trace:
[<ffffffff812ae3e4>] iput+0x3c/0x88
[<ffffffff812aaa84>] d_kill+0x4c/0x78
[<ffffffff812aad08>] __shrink_dcache_sb+0x258/0x360
[<ffffffff812ab300>] shrink_dcache_parent+0x140/0x190
[<ffffffff812eea14>] proc_flush_task+0xac/0x2e8 [<ffffffff811e6538>]
release_task+0x80/0x4c0 [<ffffffff811e80c8>] do_exit+0x6f8/0x908
[<ffffffff8121dee8>] unregister_module_notifier+0x0/0x10

Call Trace:
[<ffffffff812ae3e4>] iput+0x3c/0x88
[<ffffffff812aaa84>] d_kill+0x4c/0x78
[<ffffffff812ab6b8>] dput+0x120/0x220
[<ffffffff812a0f1c>] do_lookup+0xdc/0x210 [<ffffffff812a33e8>]
__link_path_walk+0x910/0x1408 [<ffffffff812a4194>]
path_walk+0x64/0x108 [<ffffffff812a4350>] do_path_lookup+0x60/0x68
[<ffffffff812a519c>]
do_filp_open+0xdc/0xbc8 [<ffffffff81293768>] do_sys_open+0x78/0x1e0
[<ffffffff81103844>] handle_sysn32+0x44/0x84

Quoted 4 lines

...
I am thinking of putting the following fix in
shrink_dcache_parent() . Please let me know is there any

problem

Quoted 5 lines

with this fix .
...
--- linux-2.6.32.orig/fs/dcache.c 2012-05-30 15:59:18.000000000

-0700

Quoted 6 lines

+++ linux-2.6.32/fs/dcache.c 2012-06-11 17:10:33.000000000 -0700

(quoted diff) +0 -1

@@ -881,8 +881,14 @@
        struct super_block *sb = parent->d_sb;
        int found;
 
-       while ((found = select_parent(parent)) != 0)

Review line 892

Quoted 10 lines

(quoted diff) +8 -1

-               __shrink_dcache_sb(sb, &found, 0);
+       while ((found = select_parent(parent)) != 0) {
+               if (down_read_trylock(&sb->s_umount)) {
+                        if ((sb->s_root != NULL)) {
+                               __shrink_dcache_sb(sb, &found, 0);
+                }
+                  up_read(&sb->s_umount);
+             }
+        }
 }

Please fully describe the race which you believe you have found. What races against what?

The race is between generic_shutdown_super() and __shrink_dcache_sb () . Under high memory pressure one
Of our user process crashed and the parent was trying to do a clean up with the following stack flow

<4>[] clear_inode+0x28/0xe8

<4>[] generic_drop_inode+0x3c/0xa8

<4>[] d_kill+0x4c/0x78

<4>[] __shrink_dcache_sb+0x258/0x360

<4>[] shrink_dcache_parent+0x140/0x190

<4>[] proc_flush_task+0xac/0x2e8

<4>[] release_task+0x80/0x4c0

<4>[] wait_consider_task+0x608/0xa80

<4>[] do_wait+0x10c/0x2b8

<4>[] SyS_wait4+0x88/0x120

<4>[] compat_sys_wait4+0xc8/0xd0

<4>[] handle_sysn32+0x44/0x84

During that time the system get rebooted and unmounting starts . Meanwhile the parent process is trying to clean up
The child' dentry's and clear_inode will reference to a stale inode and it will crash . So I try to grab the s_umount lock
So that __shrink_dcache_sb() won't be called during unmounts . This prevents accessing the stale inode in clear_inode .

A similar race condition is already prevented in prune_dcache() (between generic_shutdown_super ()and __shrink_dcache_sb () ) .

Please also confirm that the bug is still present in current kernels - 2.6.32 is rather old.

I am not sure whether the bug is still present in current kernels. But I do see some rcu locks in this area in the current kernel .

We are moving to 3.4 kernel . But the current product is still based on 2.6.32 .
So we need to fix this issue in 2.6.32 .

Regards,
Shaiju.

Thanks.