Skip to contentSkip to conversation

f2fs: optimize fs_lock for better performance

PATCH Chao Yu 2 participants 3 messages Sep 11, 2013 – Sep 12, 2013

Subsystems (2)
lore
Share & tools
lore

Chao Yu

author
#0 lore raw

Hi Kim,

I did some tests as you mention of using random instead of spin_lock. The test model is as following:
eight threads race to grab one of eight locks for one thousand times, and I used four methods to generate lock num:

1.atomic_add_return(1, &sbi->next_lock_num) % NR_GLOBAL_LOCKS; 2.spin_lock(); next_lock_num++ % NR_GLOBAL_LOCKS; spin_unlock(); 3.ktime_get().tv64 % NR_GLOBAL_LOCKS; 4.get_random_bytes(&next_lock, sizeof(unsigned int));

the result indicate that:
max count of collide continuously: 4 > 3 > 2 = 1
max-min count of lock is grabbed: 4 > 3 > 2 = 1
elapsed time of generating: 3 > 2 > 4 > 1

So I think it's better to use atomic_add_return in round-robin method to cost less time and reduce collide.
What's your opinion?

thanks

------- Original Message -------
Sender : ???<[email protected]> S5(??)/??/?????????(???)/???? Date : 九月 10, 2013 09:52 (GMT+09:00)
Title : Re: [f2fs-dev][PATCH] f2fs: optimize fs_lock for better performance

Hi,

At first, thank you for the report and please follow the email writing rules. :)

Anyway, I agree to the below issue.
One thing that I can think of is that we don't need to use the spin_lock, since we don't care about the exact lock number, but just need to get any not-collided number.

So, how about removing the spin_lock?
And how about using a random number?
Thanks,

2013-09-06 (?), 09:48 +0000, Chao Yu:

Quoted 138 lines

Hi Kim:

     I think there is a performance problem: when all sbi->fs_lock is

holded,

then all other threads may get the same next_lock value from sbi->next_lock_num in function mutex_lock_op,

and wait to get the same lock at position fs_lock[next_lock], it unbalance the fs_lock usage.

It may lost performance when we do the multithread test.

Here is the patch to fix this problem:

Signed-off-by Yu Chao
fs/f2fs/f2fs.h +0 -0

old mode 100644

new mode 100755

index 467d42d..983bb45

--- a/fs/f2fs/f2fs.h

+++ b/fs/f2fs/f2fs.h

fs/f2fs/f2fs.h +0 -0
@@ -371,6 +371,7 @@ struct f2fs_sb_info {
 
        struct mutex fs_lock[NR_GLOBAL_LOCKS];  /* blocking FS

operations */

struct mutex node_write; /* locking node writes
*/

struct mutex writepages; /* mutex for
writepages() */

+ spinlock_t spin_lock; /* lock for
next_lock_num */

unsigned char next_lock_num; /* round-robin global
locks */

int por_doing; /* recovery is doing
or not */

int on_build_free_nids; /* build_free_nids is
doing */

fs/f2fs/f2fs.h +6 -2
@@ -533,15 +534,19 @@ static inline void mutex_unlock_all(struct
 f2fs_sb_info *sbi)
 
 
 
 static inline int mutex_lock_op(struct f2fs_sb_info *sbi)
 
 {
 
-       unsigned char next_lock = sbi->next_lock_num %
 NR_GLOBAL_LOCKS;
 
+       unsigned char next_lock;
 
        int i = 0;
 
 
 
        for (; i < NR_GLOBAL_LOCKS; i++)
 
                if (mutex_trylock(&sbi->fs_lock[i]))
 
                        return i;
 
 
 
-       mutex_lock(&sbi->fs_lock[next_lock]);
 
+       spin_lock(&sbi->spin_lock);
 
+       next_lock = sbi->next_lock_num % NR_GLOBAL_LOCKS;
 
        sbi->next_lock_num++;
 
+       spin_unlock(&sbi->spin_lock);
 
+
 
+       mutex_lock(&sbi->fs_lock[next_lock]);
 
        return next_lock;
 
 }
fs/f2fs/super.c +0 -0

old mode 100644

new mode 100755

index 75c7dc3..4f27596

--- a/fs/f2fs/super.c

+++ b/fs/f2fs/super.c

fs/f2fs/super.c +1 -0
@@ -657,6 +657,7 @@ static int f2fs_fill_super(struct super_block *sb,
 void *data, int silent)
 
        mutex_init(&sbi->cp_mutex);
 
        for (i = 0; i < NR_GLOBAL_LOCKS; i++)
 
                mutex_init(&sbi->fs_lock[i]);
 
+       spin_lock_init(&sbi->spin_lock);
 
        mutex_init(&sbi->node_write);
 
        sbi->por_doing = 0;
 
        spin_lock_init(&sbi->stat_lock);

(END)

Signature · 2 lines
Jaegeuk Kim
Samsung
#1 lore raw

Hi,

2013/9/11 Chao Yu <[email protected]>

Quoted 13 lines

Hi Kim,

I did some tests as you mention of using random instead of spin_lock. The test model is as following:
eight threads race to grab one of eight locks for one thousand times, and I used four methods to generate lock num:

1.atomic_add_return(1, &sbi->next_lock_num) % NR_GLOBAL_LOCKS; 2.spin_lock(); next_lock_num++ % NR_GLOBAL_LOCKS; spin_unlock(); 3.ktime_get().tv64 % NR_GLOBAL_LOCKS; 4.get_random_bytes(&next_lock, sizeof(unsigned int));

Quoted 8 lines

the result indicate that:
max count of collide continuously: 4 > 3 > 2 = 1
max-min count of lock is grabbed: 4 > 3 > 2 = 1
elapsed time of generating: 3 > 2 > 4 > 1

So I think it's better to use atomic_add_return in round-robin method to cost less time and reduce collide.
What's your opinion?

Could you test with sbi->next_lock_num++ only instead of using atomic_add_return?
IMO, this is just an integer value and still I don't think this value should be covered by any kind of locks.
Thanks,

Quoted 154 lines

thanks

------- Original Message -------
Sender : ???<[email protected]> S5(??)/??/?????????(???)/???? Date : 九月 10, 2013 09:52 (GMT+09:00)
Title : Re: [f2fs-dev][PATCH] f2fs: optimize fs_lock for better performance

Hi,

At first, thank you for the report and please follow the email writing rules. :)

Anyway, I agree to the below issue.
One thing that I can think of is that we don't need to use the spin_lock, since we don't care about the exact lock number, but just need to get any not-collided number.

So, how about removing the spin_lock?
And how about using a random number?
Thanks,

2013-09-06 (?), 09:48 +0000, Chao Yu:

Quoted 131 lines

Hi Kim:

     I think there is a performance problem: when all sbi->fs_lock is

holded,

then all other threads may get the same next_lock value from sbi->next_lock_num in function mutex_lock_op,

and wait to get the same lock at position fs_lock[next_lock], it unbalance the fs_lock usage.

It may lost performance when we do the multithread test.

Here is the patch to fix this problem:

Signed-off-by Yu Chao
fs/f2fs/f2fs.h +0 -0

old mode 100644

new mode 100755

index 467d42d..983bb45

--- a/fs/f2fs/f2fs.h

+++ b/fs/f2fs/f2fs.h

fs/f2fs/f2fs.h +0 -0
@@ -371,6 +371,7 @@ struct f2fs_sb_info {
 
        struct mutex fs_lock[NR_GLOBAL_LOCKS];  /* blocking FS

operations */

struct mutex node_write; /* locking node writes
*/

struct mutex writepages; /* mutex for
writepages() */

+ spinlock_t spin_lock; /* lock for
next_lock_num */

unsigned char next_lock_num; /* round-robin global
locks */

int por_doing; /* recovery is doing
or not */

int on_build_free_nids; /* build_free_nids is
doing */

fs/f2fs/f2fs.h +6 -2
@@ -533,15 +534,19 @@ static inline void mutex_unlock_all(struct
 f2fs_sb_info *sbi)
 
 
 
 static inline int mutex_lock_op(struct f2fs_sb_info *sbi)
 
 {
 
-       unsigned char next_lock = sbi->next_lock_num %
 NR_GLOBAL_LOCKS;
 
+       unsigned char next_lock;
 
        int i = 0;
 
 
 
        for (; i < NR_GLOBAL_LOCKS; i++)
 
                if (mutex_trylock(&sbi->fs_lock[i]))
 
                        return i;
 
 
 
-       mutex_lock(&sbi->fs_lock[next_lock]);
 
+       spin_lock(&sbi->spin_lock);
 
+       next_lock = sbi->next_lock_num % NR_GLOBAL_LOCKS;
 
        sbi->next_lock_num++;
 
+       spin_unlock(&sbi->spin_lock);
 
+
 
+       mutex_lock(&sbi->fs_lock[next_lock]);
 
        return next_lock;
 
 }
fs/f2fs/super.c +0 -0

old mode 100644

new mode 100755

index 75c7dc3..4f27596

--- a/fs/f2fs/super.c

+++ b/fs/f2fs/super.c

fs/f2fs/super.c +1 -0
@@ -657,6 +657,7 @@ static int f2fs_fill_super(struct super_block *sb,
 void *data, int silent)
 
        mutex_init(&sbi->cp_mutex);
 
        for (i = 0; i < NR_GLOBAL_LOCKS; i++)
 
                mutex_init(&sbi->fs_lock[i]);
 
+       spin_lock_init(&sbi->spin_lock);
 
        mutex_init(&sbi->node_write);
 
        sbi->por_doing = 0;
 
        spin_lock_init(&sbi->stat_lock);
Quoted 11 lines
Quoted 7 lines

(END)

--
Jaegeuk Kim
Samsung

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#2 lore raw

Hi Kim

Quoted 5 lines

-----Original Message-----
From: Kim Jaegeuk [mailto:[email protected]]
Sent: Wednesday, September 11, 2013 9:15 PM
To: [email protected]
Cc: ???; 谭姝; [email protected];

[email protected];

Quoted 2 lines

[email protected]
Subject: Re: Re: [f2fs-dev][PATCH] f2fs: optimize fs_lock for better

performance

Quoted 21 lines

Hi,

2013/9/11 Chao Yu <[email protected]>

Quoted 17 lines

Hi Kim,

I did some tests as you mention of using random instead of spin_lock. The test model is as following:
eight threads race to grab one of eight locks for one thousand times, and I used four methods to generate lock num:

1.atomic_add_return(1, &sbi->next_lock_num) % NR_GLOBAL_LOCKS; 2.spin_lock(); next_lock_num++ % NR_GLOBAL_LOCKS; spin_unlock(); 3.ktime_get().tv64 % NR_GLOBAL_LOCKS; 4.get_random_bytes(&next_lock, sizeof(unsigned int));

the result indicate that:
max count of collide continuously: 4 > 3 > 2 = 1 max-min count of lock is grabbed: 4 > 3 > 2 = 1 elapsed time of generating: 3 > 2 > 4 > 1

Quoted 7 lines
Quoted 3 lines

So I think it's better to use atomic_add_return in round-robin method to cost less time and reduce collide.
What's your opinion?

Could you test with sbi->next_lock_num++ only instead of using atomic_add_return?
IMO, this is just an integer value and still I don't think this value

should be

Quoted 2 lines

covered by any kind of locks.
Thanks,

Thanks for the advice, I have tested sbi->next_lock_num++.
The time cost of it is a little bit lower than the atomic one's. for running 8 thread for 1000000 times test.
the performance of it's balance and collide play quit well than I expected.

Can we modify this patch as following?

root@virtaulmachine:/home/yuchao/git/linux-next/fs/f2fs# git diff --stat
 fs/f2fs/f2fs.h |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
root@virtaulmachine:/home/yuchao/git/linux-next/fs/f2fs# git diff
fs/f2fs/f2fs.h +0 -0
index 608f0df..7fd99d8 100644
@@ -544,15 +544,15 @@ static inline void mutex_unlock_all(struct

f2fs_sb_info *sbi)

static inline int mutex_lock_op(struct f2fs_sb_info *sbi) {
- unsigned char next_lock = sbi->next_lock_num % NR_GLOBAL_LOCKS;

+       unsigned char next_lock;
        int i = 0;
 
        for (; i < NR_GLOBAL_LOCKS; i++)
                if (mutex_trylock(&sbi->fs_lock[i]))
                        return i;
+       next_lock = sbi->next_lock_num++ % NR_GLOBAL_LOCKS;
        mutex_lock(&sbi->fs_lock[next_lock]);
-       sbi->next_lock_num++;
        return next_lock;

}

Quoted 170 lines
Quoted 56 lines

thanks

------- Original Message -------
Sender : ???<[email protected]> S5(??)/??/?????????(???)/???? Date : 九月 10, 2013 09:52 (GMT+09:00)
Title : Re: [f2fs-dev][PATCH] f2fs: optimize fs_lock for better performance

Hi,

At first, thank you for the report and please follow the email writing rules. :)

Anyway, I agree to the below issue.
One thing that I can think of is that we don't need to use the spin_lock, since we don't care about the exact lock number, but just need to get any not-collided number.

So, how about removing the spin_lock?
And how about using a random number?
Thanks,

2013-09-06 (?), 09:48 +0000, Chao Yu:

Quoted 31 lines

Hi Kim:

     I think there is a performance problem: when all sbi->fs_lock

is holded,

then all other threads may get the same next_lock value from sbi->next_lock_num in function mutex_lock_op,

and wait to get the same lock at position fs_lock[next_lock], it unbalance the fs_lock usage.

It may lost performance when we do the multithread test.

Here is the patch to fix this problem:

Signed-off-by Yu Chao
fs/f2fs/f2fs.h +0 -0

old mode 100644

new mode 100755

index 467d42d..983bb45

--- a/fs/f2fs/f2fs.h

Quoted 8 lines
Quoted 8 lines

+++ b/fs/f2fs/f2fs.h

fs/f2fs/f2fs.h +0 -0
@@ -371,6 +371,7 @@ struct f2fs_sb_info {
 
        struct mutex fs_lock[NR_GLOBAL_LOCKS];  /* blocking FS

operations */

struct mutex node_write; /* locking node

writes

Quoted 9 lines
Quoted 9 lines

*/

struct mutex writepages; /* mutex for
writepages() */

+ spinlock_t spin_lock; /* lock for
next_lock_num */

unsigned char next_lock_num; /* round-robin

global

Quoted 3 lines
Quoted 3 lines

locks */

int por_doing; /* recovery is

doing

Quoted 91 lines
Quoted 80 lines

or not */

int on_build_free_nids; /* build_free_nids is
doing */

fs/f2fs/f2fs.h +6 -2
@@ -533,15 +534,19 @@ static inline void mutex_unlock_all(struct
 f2fs_sb_info *sbi)
 
 
 
 static inline int mutex_lock_op(struct f2fs_sb_info *sbi)
 
 {
 
-       unsigned char next_lock = sbi->next_lock_num %
 NR_GLOBAL_LOCKS;
 
+       unsigned char next_lock;
 
        int i = 0;
 
 
 
        for (; i < NR_GLOBAL_LOCKS; i++)
 
                if (mutex_trylock(&sbi->fs_lock[i]))
 
                        return i;
 
 
 
-       mutex_lock(&sbi->fs_lock[next_lock]);
 
+       spin_lock(&sbi->spin_lock);
 
+       next_lock = sbi->next_lock_num % NR_GLOBAL_LOCKS;
 
        sbi->next_lock_num++;
 
+       spin_unlock(&sbi->spin_lock);
 
+
 
+       mutex_lock(&sbi->fs_lock[next_lock]);
 
        return next_lock;
 
 }
fs/f2fs/super.c +0 -0

old mode 100644

new mode 100755

index 75c7dc3..4f27596

--- a/fs/f2fs/super.c

+++ b/fs/f2fs/super.c

fs/f2fs/super.c +1 -0
@@ -657,6 +657,7 @@ static int f2fs_fill_super(struct super_block
 *sb, void *data, int silent)
 
        mutex_init(&sbi->cp_mutex);
 
        for (i = 0; i < NR_GLOBAL_LOCKS; i++)
 
                mutex_init(&sbi->fs_lock[i]);
 
+       spin_lock_init(&sbi->spin_lock);
 
        mutex_init(&sbi->node_write);
 
        sbi->por_doing = 0;
 
        spin_lock_init(&sbi->stat_lock);
Quoted 7 lines

(END)

--
Jaegeuk Kim
Samsung