Skip to contentSkip to conversation

f2fs: Avoid print false deadlock messages.

RFC PATCH In review majianpeng 3 participants 8 messages May 15, 2013 – May 16, 2013 F2FS file system

lore
Share & tools
lore

majianpeng

author
#0 lore raw

RFC PATCH 1/1 f2fs: Avoid print false deadlock messages.

When mounted the f2fs, kernel will print the following messages:

[  105.533038] =============================================
[  105.533065] [ INFO: possible recursive locking detected ]
[  105.533088] 3.10.0-rc1+ #101 Not tainted
[  105.533105] ---------------------------------------------
[  105.533127] mount/5833 is trying to acquire lock:
[  105.533147]  (&sbi->fs_lock[i]){+.+...}, at: [<ffffffffa02017a6>] write_checkpoint+0xb6/0xaf0 [f2fs]
[  105.533204]
[  105.533204] but task is already holding lock:
[  105.533228]  (&sbi->fs_lock[i]){+.+...}, at: [<ffffffffa02017a6>] write_checkpoint+0xb6/0xaf0 [f2fs]
[  105.533278]
[  105.533278] other info that might help us debug this:
[  105.533305]  Possible unsafe locking scenario:
[  105.533305]
[  105.533329]        CPU0
[  105.533341]        ----
[  105.533353]   lock(&sbi->fs_lock[i]);
[  105.533373]   lock(&sbi->fs_lock[i]);
[  105.533394]
[  105.533394]  *** DEADLOCK ***
[  105.533394]
[  105.533419]  May be due to missing lock nesting notation
[  105.533419]
[  105.533447] 3 locks held by mount/5833:
[  105.533463]  #0:  (&type->s_umount_key#23/1){+.+.+.}, at: [<ffffffff81167b2f>] sget+0x2af/0x620
[  105.533519]  #1:  (&sbi->cp_mutex){+.+...}, at: [<ffffffffa0201745>] write_checkpoint+0x55/0xaf0 [f2fs]
[  105.533573]  #2:  (&sbi->fs_lock[i]){+.+...}, at: [<ffffffffa02017a6>] write_checkpoint+0xb6/0xaf0 [f2fs]
[  105.533626]
[  105.533626] stack backtrace:
[  105.533647] CPU: 0 PID: 5833 Comm: mount Not tainted 3.10.0-rc1+ #101
[  105.533673] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./To be filled by O.E.M., BIOS 080015  11/09/2011
[  105.533714]  ffffffff82240690 ffff8800a6f759d0 ffffffff8168286a ffff8800a6f75ac0
[  105.533754]  ffffffff810a03e3 0000000000000002 ffff8800a1b7a3f0 0000000000000000
[  105.533794]  ffff8800a6f75b28 ffffffff812c376e ffff8800a1b7a3f0 0000000000000000
[  105.533833] Call Trace:
[  105.533848]  [<ffffffff8168286a>] dump_stack+0x19/0x1b
[  105.533872]  [<ffffffff810a03e3>] __lock_acquire+0x733/0x1ee0
[  105.533898]  [<ffffffff812c376e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[  105.533926]  [<ffffffff8168c65c>] ? retint_restore_args+0xe/0xe
[  105.533951]  [<ffffffff810a2205>] lock_acquire+0x95/0x130
[  105.533977]  [<ffffffffa02017a6>] ? write_checkpoint+0xb6/0xaf0 [f2fs]
[  105.534011]  [<ffffffff816878da>] mutex_lock_nested+0x7a/0x3c0
[  105.534011]  [<ffffffffa02017a6>] ? write_checkpoint+0xb6/0xaf0 [f2fs]
[  105.534011]  [<ffffffffa02017a6>] ? write_checkpoint+0xb6/0xaf0 [f2fs]
[  105.534011]  [<ffffffffa02017a6>] write_checkpoint+0xb6/0xaf0 [f2fs]
[  105.534011]  [<ffffffff8109e35b>] ? mark_held_locks+0xbb/0x140
[  105.534011]  [<ffffffff81688907>] ?__mutex_unlock_slowpath+0xc7/0x170
[  105.534011]  [<ffffffffa020f43c>] recover_fsync_data+0x43c/0x4f0 [f2fs]
[  105.534011]  [<ffffffff812673bb>] ? security_d_instantiate+0x1b/0x30
[  105.534011]  [<ffffffffa0200795>] f2fs_fill_super+0x845/0x870 [f2fs]
[  105.534011]  [<ffffffff81168104>] mount_bdev+0x1b4/0x1f0
[  105.534011]  [<ffffffffa01fff50>] ? validate_superblock+0x170/0x170 [f2fs]
[  105.534011]  [<ffffffffa01fe745>] f2fs_mount+0x15/0x20 [f2fs]
[  105.534011]  [<ffffffff81168e03>] mount_fs+0x43/0x1b0
[  105.534011]  [<ffffffff8112d9d0>] ? __alloc_percpu+0x10/0x20
[  105.534011]  [<ffffffff81183be4>] vfs_kern_mount+0x74/0x120
[  105.534011]  [<ffffffff811862b9>] do_mount+0x259/0xa20
[  105.534011]  [<ffffffff8112890b>] ? strndup_user+0x5b/0x80
[  105.534011]  [<ffffffff81186b0e>] SyS_mount+0x8e/0xe0
[  105.534011]  [<ffffffff81694b02>] system_call_fastpath+0x16/0x1b

By adding some messages, i found this problem because the gcc optimizing. For those codes:

Quoted 2 lines
       for (i = 0; i < NR_GLOBAL_LOCKS; i++)
               mutex_init(&sbi->fs_lock[i]);

The defination of mutex_init is:

Quoted 9 lines

#define mutex_init(mutex)
do {

       static struct lock_class_key __key;

                                                                     
       __mutex_init((mutex), #mutex, &__key);

} while (0)

Because the optimizing of gcc, there are only one __key rather than NR_GLOBAL_LOCKS times.

Add there is other problems about lockname.Using 'for()' the lockname is the same which is '&sbi->fs_lock[i]'.If it met problem about mutex-operation, it can't find which one.

Although my patch can work,i think it's not best.Because if NR_GLOBAL_LOCKS changed, we may leak to change this.

BTY, if who know how to avoid optimize, please tell me. Thanks!

---
 fs/f2fs/super.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
fs/f2fs/super.c +10 -3
index 8555f7d..ce08b96 100644
@@ -520,7 +520,6 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
     struct buffer_head *raw_super_buf;
     struct inode *root;
     long err = -EINVAL;
-    int i;
 
     /* allocate memory for f2fs-specific super block info */
     sbi = kzalloc(sizeof(struct f2fs_sb_info), GFP_KERNEL);
@@ -578,8 +577,16 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
     mutex_init(&sbi->gc_mutex);
     mutex_init(&sbi->writepages);
     mutex_init(&sbi->cp_mutex);
-    for (i = 0; i < NR_GLOBAL_LOCKS; i++)
-        mutex_init(&sbi->fs_lock[i]);
+
+    mutex_init(&sbi->fs_lock[0]);
+    mutex_init(&sbi->fs_lock[1]);
+    mutex_init(&sbi->fs_lock[2]);
+    mutex_init(&sbi->fs_lock[3]);
+    mutex_init(&sbi->fs_lock[4]);
+    mutex_init(&sbi->fs_lock[5]);
+    mutex_init(&sbi->fs_lock[6]);
+    mutex_init(&sbi->fs_lock[7]);
+
     mutex_init(&sbi->node_write);
     sbi->por_doing = 0;
     spin_lock_init(&sbi->stat_lock);
Signature · 1 line
1.8.3.rc1.44.gb387c77
#1 lore raw
Quoted 44 lines · majianpeng

By adding some messages, i found this problem because the gcc optimizing. For those codes:

Quoted 2 lines
Quoted 2 lines
       for (i = 0; i < NR_GLOBAL_LOCKS; i++)
               mutex_init(&sbi->fs_lock[i]);

The defination of mutex_init is:

Quoted 9 lines
Quoted 9 lines

#define mutex_init(mutex)
do {

       static struct lock_class_key __key;

                                                                     
       __mutex_init((mutex), #mutex, &__key);

} while (0)

Because the optimizing of gcc, there are only one __key rather than NR_GLOBAL_LOCKS times.

Add there is other problems about lockname.Using 'for()' the lockname is the same which is '&sbi->fs_lock[i]'.If it met problem about mutex-operation, it can't find which one.

Although my patch can work,i think it's not best.Because if NR_GLOBAL_LOCKS changed, we may leak to change this.

BTY, if who know how to avoid optimize, please tell me. Thanks!

---
 fs/f2fs/super.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
fs/f2fs/super.c +0 -1
index 8555f7d..ce08b96 100644
@@ -520,7 +520,6 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
     struct buffer_head *raw_super_buf;
     struct inode *root;
     long err = -EINVAL;
-    int i;
 
     /* allocate memory for f2fs-specific super block info */
     sbi = kzalloc(sizeof(struct f2fs_sb_info), GFP_KERNEL);

fs/f2fs/super.c static int f2fs_fill_super(struct super_block *sb, void *data, int silent) · line 580

Quoted 6 lines
fs/f2fs/super.c +1 -2
@@ -578,8 +577,16 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
     mutex_init(&sbi->gc_mutex);
     mutex_init(&sbi->writepages);
     mutex_init(&sbi->cp_mutex);
-    for (i = 0; i < NR_GLOBAL_LOCKS; i++)
-        mutex_init(&sbi->fs_lock[i]);
+

you can try barrier() or mb() to avoid compile optimization.

author
#2 lore raw
Quoted 42 lines · Libo Chen
Quoted 41 lines · majianpeng

By adding some messages, i found this problem because the gcc optimizing. For those codes:

Quoted 2 lines
Quoted 2 lines
       for (i = 0; i < NR_GLOBAL_LOCKS; i++)
               mutex_init(&sbi->fs_lock[i]);

The defination of mutex_init is:

Quoted 9 lines
Quoted 9 lines

#define mutex_init(mutex)
do {

       static struct lock_class_key __key;

                                                                     
       __mutex_init((mutex), #mutex, &__key);

} while (0)

Because the optimizing of gcc, there are only one __key rather than NR_GLOBAL_LOCKS times.

Add there is other problems about lockname.Using 'for()' the lockname is the same which is '&sbi->fs_lock[i]'.If it met problem about mutex-operation, it can't find which one.

Although my patch can work,i think it's not best.Because if NR_GLOBAL_LOCKS changed, we may leak to change this.

BTY, if who know how to avoid optimize, please tell me. Thanks!

---
 fs/f2fs/super.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
fs/f2fs/super.c +0 -1
index 8555f7d..ce08b96 100644
@@ -520,7 +520,6 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
     struct buffer_head *raw_super_buf;
     struct inode *root;
     long err = -EINVAL;
-    int i;
 
Quoted 11 lines

fs/f2fs/super.c static int f2fs_fill_super(struct super_block *sb, void *data, int silent) · line 580

Quoted 9 lines
fs/f2fs/super.c +1 -2
     /* allocate memory for f2fs-specific super block info */
     sbi = kzalloc(sizeof(struct f2fs_sb_info), GFP_KERNEL);
@@ -578,8 +577,16 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
     mutex_init(&sbi->gc_mutex);
     mutex_init(&sbi->writepages);
     mutex_init(&sbi->cp_mutex);
-    for (i = 0; i < NR_GLOBAL_LOCKS; i++)
-        mutex_init(&sbi->fs_lock[i]);
+

you can try barrier() or mb() to avoid compile optimization.

Hi,
They aren't work!But thanks very much!

Thanks!
Jianpeng Ma

#3 lore raw
In reply to majianpeng
Quoted 34 lines · majianpeng

When mounted the f2fs, kernel will print the following messages:

[  105.533038] =============================================
[  105.533065] [ INFO: possible recursive locking detected ]
[  105.533088] 3.10.0-rc1+ #101 Not tainted
[  105.533105] ---------------------------------------------
[  105.533127] mount/5833 is trying to acquire lock:
[  105.533147]  (&sbi->fs_lock[i]){+.+...}, at: [<ffffffffa02017a6>] write_checkpoint+0xb6/0xaf0 [f2fs]
[  105.533204]
[  105.533204] but task is already holding lock:
[  105.533228]  (&sbi->fs_lock[i]){+.+...}, at: [<ffffffffa02017a6>] write_checkpoint+0xb6/0xaf0 [f2fs]
[  105.533278]
[  105.533278] other info that might help us debug this:
[  105.533305]  Possible unsafe locking scenario:
[  105.533305]
[  105.533329]        CPU0
[  105.533341]        ----
[  105.533353]   lock(&sbi->fs_lock[i]);
[  105.533373]   lock(&sbi->fs_lock[i]);
[  105.533394]
[  105.533394]  *** DEADLOCK ***
[  105.533394]

By adding some messages, i found this problem because the gcc optimizing. For those codes:

Quoted 2 lines
       for (i = 0; i < NR_GLOBAL_LOCKS; i++)
               mutex_init(&sbi->fs_lock[i]);

The defination of mutex_init is:

Quoted 6 lines

#define mutex_init(mutex)
do {

       static struct lock_class_key __key;
Quoted 6 lines
Quoted 3 lines
       __mutex_init((mutex), #mutex, &__key);

} while (0)

Because the optimizing of gcc, there are only one __key rather than NR_GLOBAL_LOCKS times.

Its not a gcc specific optimization, any C compiler would. Its also very much on purpose.

Quoted 8 lines

Add there is other problems about lockname.Using 'for()' the lockname is the same which is '&sbi->fs_lock[i]'.If it met problem about mutex-operation, it can't find which one.

Although my patch can work,i think it's not best.Because if NR_GLOBAL_LOCKS changed, we may leak to change this.

BTY, if who know how to avoid optimize, please tell me. Thanks!

There isn't. What you typically want to do is annotate the lock site.

In particular it looks like mutex_lock_all() is the offensive piece of code (horrible function name though; the only redeeming thing being that f2fs.h isn't likely to be included elsewhere).

One thing you can do here is modify it to look like:

static inline void mutex_lock_all(struct f2fs_sb_info *sbi)

{
	int i;

	for (i = 0; i < NR_GLOBAL_LOCKS; i++) {
		/*
		 * This is the only time we take multiple fs_lock[]
		 * instances; the order is immaterial since we
		 * always hold cp_mutex, which serializes multiple
		 * such operations.
		 */
		mutex_lock_nest_lock(&sbi->fs_lock[i], &sbi->cp_mutex);
	}
}

That tells the lock validator that it is ok to lock multiple instances of the fs_lock[i] class because the lock order is guarded by cp_mutex.

While your patch also works, it has multiple down-sides; its easy to get out of sync when you modify NR_GLOBAL_LOCKS; also it consumes more static lockdep resources (lockdep has to allocate all its resources from static arrays since allocating memory also uses locks -- recursive problem).

author
#4 lore raw
Quoted 34 lines · Peter Zijlstra
Quoted 33 lines · majianpeng

When mounted the f2fs, kernel will print the following messages:

[  105.533038] =============================================
[  105.533065] [ INFO: possible recursive locking detected ]
[  105.533088] 3.10.0-rc1+ #101 Not tainted
[  105.533105] ---------------------------------------------
[  105.533127] mount/5833 is trying to acquire lock:
[  105.533147]  (&sbi->fs_lock[i]){+.+...}, at: [<ffffffffa02017a6>] write_checkpoint+0xb6/0xaf0 [f2fs]
[  105.533204]
[  105.533204] but task is already holding lock:
[  105.533228]  (&sbi->fs_lock[i]){+.+...}, at: [<ffffffffa02017a6>] write_checkpoint+0xb6/0xaf0 [f2fs]
[  105.533278]
[  105.533278] other info that might help us debug this:
[  105.533305]  Possible unsafe locking scenario:
[  105.533305]
[  105.533329]        CPU0
[  105.533341]        ----
[  105.533353]   lock(&sbi->fs_lock[i]);
[  105.533373]   lock(&sbi->fs_lock[i]);
[  105.533394]
[  105.533394]  *** DEADLOCK ***
[  105.533394]

By adding some messages, i found this problem because the gcc optimizing. For those codes:

Quoted 2 lines
       for (i = 0; i < NR_GLOBAL_LOCKS; i++)
               mutex_init(&sbi->fs_lock[i]);

The defination of mutex_init is:

Quoted 6 lines

#define mutex_init(mutex)
do {

       static struct lock_class_key __key;
Quoted 16 lines
Quoted 5 lines
Quoted 3 lines
       __mutex_init((mutex), #mutex, &__key);

} while (0)

Because the optimizing of gcc, there are only one __key rather than NR_GLOBAL_LOCKS times.

Its not a gcc specific optimization, any C compiler would. Its also very much on purpose.

Quoted 8 lines

Add there is other problems about lockname.Using 'for()' the lockname is the same which is '&sbi->fs_lock[i]'.If it met problem about mutex-operation, it can't find which one.

Although my patch can work,i think it's not best.Because if NR_GLOBAL_LOCKS changed, we may leak to change this.

BTY, if who know how to avoid optimize, please tell me. Thanks!

Thanks your answer! Your patch looks good.

Quoted 22 lines

There isn't. What you typically want to do is annotate the lock site. In particular it looks like mutex_lock_all() is the offensive piece of code (horrible function name though; the only redeeming thing being that f2fs.h isn't likely to be included elsewhere).

One thing you can do here is modify it to look like:

static inline void mutex_lock_all(struct f2fs_sb_info *sbi)

{
	int i;

	for (i = 0; i < NR_GLOBAL_LOCKS; i++) {
		/*
		 * This is the only time we take multiple fs_lock[]
		 * instances; the order is immaterial since we
		 * always hold cp_mutex, which serializes multiple
		 * such operations.
		 */
		mutex_lock_nest_lock(&sbi->fs_lock[i], &sbi->cp_mutex);
	}
}
Quoted 8 lines

That tells the lock validator that it is ok to lock multiple instances of the fs_lock[i] class because the lock order is guarded by cp_mutex. While your patch also works, it has multiple down-sides; its easy to get out of sync when you modify NR_GLOBAL_LOCKS; also it consumes more static lockdep resources (lockdep has to allocate all its resources from static arrays since allocating memory also uses locks -- recursive problem).

Yes, but there is a problem if fs_block[] met deadlock. How to find which one? Because the lock->name is the same.

Thanks!
Jianpeng Ma

#5 lore raw
Quoted 30 lines · majianpeng
Quoted 22 lines

There isn't. What you typically want to do is annotate the lock site. In particular it looks like mutex_lock_all() is the offensive piece of code (horrible function name though; the only redeeming thing being that f2fs.h isn't likely to be included elsewhere).

One thing you can do here is modify it to look like:

static inline void mutex_lock_all(struct f2fs_sb_info *sbi)

{
	int i;

	for (i = 0; i < NR_GLOBAL_LOCKS; i++) {
		/*
		 * This is the only time we take multiple fs_lock[]
		 * instances; the order is immaterial since we
		 * always hold cp_mutex, which serializes multiple
		 * such operations.
		 */
		mutex_lock_nest_lock(&sbi->fs_lock[i], &sbi->cp_mutex);
	}
}
Quoted 8 lines

That tells the lock validator that it is ok to lock multiple instances of the fs_lock[i] class because the lock order is guarded by cp_mutex. While your patch also works, it has multiple down-sides; its easy to get out of sync when you modify NR_GLOBAL_LOCKS; also it consumes more static lockdep resources (lockdep has to allocate all its resources from static arrays since allocating memory also uses locks -- recursive problem).

Quoted 2 lines

Yes, but there is a problem if fs_block[] met deadlock. How to find which one? Because the lock->name is the same.

The most useful part of the lockdep report are the call traces that tell you where locks where acquired; the names are secondary. That is, while they are at times helpful in finding the right lock site, they're rarely _that_ important.

Remember, your code will very likely not have the exact number hardcoded either. It'll be a variable. So having the number in the lockdep output will not help you find the offending code any sooner.

Suppose there's another site that acquires two fs_block[] locks; currently this would generate another such warning as this thread started with -- lockdep doesn't look at lock instances but at classes; and it cannot differentiate between two locks of the same class and thus reports the possible deadlock.

The way to find the offending code is to look at the "locks held" section of the lockdep report along with the call traces.

Once you find the way in which the two locks nest the specific numbers are irrelevant. Your aim then is to prove your locking scheme is indeed deadlock free and then tell lockdep about it.

author
#6 lore raw
Quoted 47 lines · Peter Zijlstra
Quoted 30 lines · majianpeng
Quoted 22 lines

There isn't. What you typically want to do is annotate the lock site. In particular it looks like mutex_lock_all() is the offensive piece of code (horrible function name though; the only redeeming thing being that f2fs.h isn't likely to be included elsewhere).

One thing you can do here is modify it to look like:

static inline void mutex_lock_all(struct f2fs_sb_info *sbi)

{
	int i;

	for (i = 0; i < NR_GLOBAL_LOCKS; i++) {
		/*
		 * This is the only time we take multiple fs_lock[]
		 * instances; the order is immaterial since we
		 * always hold cp_mutex, which serializes multiple
		 * such operations.
		 */
		mutex_lock_nest_lock(&sbi->fs_lock[i], &sbi->cp_mutex);
	}
}
Quoted 8 lines

That tells the lock validator that it is ok to lock multiple instances of the fs_lock[i] class because the lock order is guarded by cp_mutex. While your patch also works, it has multiple down-sides; its easy to get out of sync when you modify NR_GLOBAL_LOCKS; also it consumes more static lockdep resources (lockdep has to allocate all its resources from static arrays since allocating memory also uses locks -- recursive problem).

Quoted 2 lines

Yes, but there is a problem if fs_block[] met deadlock. How to find which one? Because the lock->name is the same.

The most useful part of the lockdep report are the call traces that tell you where locks where acquired; the names are secondary. That is, while they are at times helpful in finding the right lock site, they're rarely _that_ important.

Remember, your code will very likely not have the exact number hardcoded either. It'll be a variable. So having the number in the lockdep output will not help you find the offending code any sooner.

Suppose there's another site that acquires two fs_block[] locks; currently this would generate another such warning as this thread started with -- lockdep doesn't look at lock instances but at classes; and it cannot differentiate between two locks of the same class and thus reports the possible deadlock.

Quoted 8 lines

The way to find the offending code is to look at the "locks held" section of the lockdep report along with the call traces.

Once you find the way in which the two locks nest the specific numbers are irrelevant. Your aim then is to prove your locking scheme is indeed deadlock free and then tell lockdep about it.

Thanks very much! I'll take times to understand.
Can you send a patch about this?

Thanks!
Jianpeng Ma

#7 lore raw
Quoted 2 lines · majianpeng

Thanks very much! I'll take times to understand.
Can you send a patch about this?

You mean the below or something else?

---
Subject: f2fs, lockdep: Annotate mutex_lock_all()

Majianpeng reported a lockdep splat for f2fs. It turns out mutex_lock_all() acquires an array of locks (in global/local lock style).

Any such operation is always serialized using cp_mutex, therefore there is no fs_lock[] lock-order issue; tell lockdep about this using the mutex_lock_nest_lock() primitive.

---
 fs/f2fs/f2fs.h | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
fs/f2fs/f2fs.h +11 -3
index 20aab02..8454277 100644
@@ -495,9 +495,17 @@ static inline void clear_ckpt_flags(struct f2fs_checkpoint *cp, unsigned int f)
 
 static inline void mutex_lock_all(struct f2fs_sb_info *sbi)
 {
-	int i = 0;
-	for (; i < NR_GLOBAL_LOCKS; i++)
-		mutex_lock(&sbi->fs_lock[i]);
+	int i;
+
+	for (i = 0; i < NR_GLOBAL_LOCKS; i++) {
+		/*
+		 * This is the only time we take multiple fs_lock[]
+		 * instances; the order is immaterial since we
+		 * always hold cp_mutex, which serializes multiple
+		 * such operations.
+		 */
+		mutex_lock_nest_lock(&sbi->fs_lock[i], &sbi->cp_mutex);
+	}
 }
 
 static inline void mutex_unlock_all(struct f2fs_sb_info *sbi)