fs: use RCU for free_super() vs. __sb_start_write()
RFC PATCH 1/2 fs: use RCU for free_super() vs. __sb_start_write()
Currently, __sb_start_write() and freeze_super() can race with each other. __sb_start_write() uses a smp_mb() to ensure that freeze_super() can see its write to sb->s_writers.counter and that it can see freeze_super()'s update to sb->s_writers.frozen. This all seems to work fine.
But, this smp_mb() makes __sb_start_write() the single hottest function in the kernel if I sit in a loop and do tiny write()s to tmpfs over and over. This is on a very small 2-core system, so it will only get worse on larger systems.
This _seems_ like an ideal case for RCU. __sb_start_write() is the RCU read-side and is in a very fast, performance-sensitive path. freeze_super() is the RCU writer and is in an extremely rare non-performance-sensitive path.
Instead of doing and smp_wmb() in __sb_start_write(), we do rcu_read_lock(). This ensures that a CPU doing freeze_super() can not proceed past its synchronize_rcu() until the grace period has ended and the 's_writers.frozen = SB_FREEZE_WRITE' is visible to __sb_start_write().
One question here: Does the work that __sb_start_write() does in a previous grace period becomes visible to freeze_super() after its call to synchronize_rcu()? It _seems_ like it should, but it seems backwards to me since __sb_start_write() is the "reader" in this case.
This patch increases the number of writes/second that I can do by 10.4%.
Does anybody see any holes with this?
---
b/fs/super.c | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-)
fs/super.c renamed from fs/super.c~rcu-__sb_start_write +21 -17
@@ -1190,27 +1190,25 @@ static void acquire_freeze_lock(struct s*/ int __sb_start_write (struct super_block *sb,int level,bool wait) { -retry: -if (unlikely (sb->s_writers.frozen >= level)) { +/* +* RCU keeps freeze_super() from proceeding +* while we are in here. +*/ +rcu_read_lock (); +while (unlikely (sb->s_writers.frozen >= level)) { +rcu_read_unlock ();if (!wait) -return 0 ; +return 0 ;wait_event (sb->s_writers.wait_unfrozen, sb->s_writers.frozen < level); +rcu_read_lock (); }#ifdef CONFIG_LOCKDEP acquire_freeze_lock (sb, level, !wait, _RET_IP_);#endif percpu_counter_inc (&sb->s_writers.counter[level-1 ]); -/* -* Make sure counter is updated before we check for frozen. -* freeze_super() first sets frozen and then checks the counter. -*/ -smp_mb (); -if (unlikely (sb->s_writers.frozen >= level)) { -__sb_end_write (sb, level); - goto retry; - } +rcu_read_unlock ();return 1 ; }EXPORT_SYMBOL (__sb_start_write);
@@ -1312,7 +1310,13 @@ int freeze_super(struct super_block *sb)/* From now on, no new normal writers can start */ sb->s_writers.frozen =SB_FREEZE_WRITE ; -smp_wmb (); +/* +* After we synchronize_rcu(), we have ensured that everyone +* who reads sb->s_writers.frozen under rcu_read_lock() can +* now see our update. This pretty much means that +* __sb_start_write() will not allow any new writers. +*/ +synchronize_rcu ();/* Release s_umount to preserve sb_start_write -> s_umount ordering */ up_write (&sb->s_umount);
@@ -1322,7 +1326,7 @@ int freeze_super(struct super_block *sb)/* Now we go and block page faults... */ down_write (&sb->s_umount); sb->s_writers.frozen =SB_FREEZE_PAGEFAULT ; -smp_wmb (); +synchronize_rcu ();sb_wait_write (sb,SB_FREEZE_PAGEFAULT );
@@ -1331,7 +1335,7 @@ int freeze_super(struct super_block *sb)/* Now wait for internal filesystem counter */ sb->s_writers.frozen =SB_FREEZE_FS ; -smp_wmb (); +synchronize_rcu ();sb_wait_write (sb,SB_FREEZE_FS );if (sb->s_op->freeze_fs) {
@@ -1340,7 +1344,7 @@ int freeze_super(struct super_block *sb)printk (KERN_ERR "VFS:Filesystem freeze failed\n" ); sb->s_writers.frozen =SB_UNFROZEN ; -smp_wmb (); +synchronize_rcu ();wake_up (&sb->s_writers.wait_unfrozen);deactivate_locked_super (sb);return ret;
@@ -1387,7 +1391,7 @@ int thaw_super(struct super_block *sb) out: sb->s_writers.frozen =SB_UNFROZEN ; -smp_wmb (); +synchronize_rcu ();wake_up (&sb->s_writers.wait_unfrozen);deactivate_locked_super (sb);
_
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
RFC PATCH 2/2 fs: conditionally do memory barrier in __sb_end_write()
If I sit in a loop and do write()s to small tmpfs files, __sb_end_write() is third-hottest kernel function due to its smp_mb().
The stated purpose for the smp_mb() in __sb_end_write() is to ensure "s_writers are updated before we wake up waiters". We only wake up waiters if waitqueue_active(), but we do the smp_mb() unconditionally.
It seems like we should be able to avoid it unless we are actually doing the wake_up().
---
b/fs/super.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
fs/super.c renamed from fs/super.c~selectively-do-barriers-in-__sb_end_write +7 -6
@@ -1147,13 +1147,14 @@ out:void __sb_end_write (struct super_block *sb,int level) {percpu_counter_dec (&sb->s_writers.counter[level-1 ]); -/* -* Make sure s_writers are updated before we wake up waiters in -* freeze_super(). -*/ -smp_mb (); -if (waitqueue_active (&sb->s_writers.wait)) +if (waitqueue_active (&sb->s_writers.wait)) { +/* +* Make sure other CPUs can see our s_writers update +* before we wake up waiters in freeze_super(). +*/ +smp_mb ();wake_up (&sb->s_writers.wait); + }rwsem_release (&sb->s_writers.lock_map[level-1 ],1 , _RET_IP_); }EXPORT_SYMBOL (__sb_end_write);
_
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
Jan Kara
reviewerRe: [RFC][PATCH 2/2] fs: conditionally do memory barrier in __sb_end_write()
Quoted 11 lines · Dave Hansen
If I sit in a loop and do write()s to small tmpfs files,
__sb_end_write()is third-hottest kernel function due to itssmp_mb().The stated purpose for the
smp_mb()in__sb_end_write()is to ensure "s_writersare updated before we wake up waiters". We only wake up waiters ifwaitqueue_active(), but we do thesmp_mb()unconditionally.It seems like we should be able to avoid it unless we are actually doing the
wake_up().
...
Quoted 9 lines
diff -puN
fs/super.c~selectively-do-barriers-in-__sb_end_writefs/super.cfs/super.c renamed from fs/super.c~selectively-do-barriers-in-__sb_end_write +0
-2@@ -1147,13 +1147,14 @@ out:void __sb_end_write (struct super_block *sb,int level) {percpu_counter_dec (&sb->s_writers.counter[level-1 ]); -/* -* Make sure s_writers are updated before we wake up waiters in
fs/super.c out: · line 1155
Quoted 10 lines
fs/super.c renamed from fs/super.c~selectively-do-barriers-in-__sb_end_write +6
-4-* freeze_super(). -*/ -smp_mb (); -if (waitqueue_active (&sb->s_writers.wait)) +if (waitqueue_active (&sb->s_writers.wait)) { +/* +* Make sure other CPUs can see our s_writers update +* before we wake up waiters in freeze_super(). +*/ +smp_mb ();
I think this is actually wrong. The barrier has to be before the waitqueue_active() check. Otherwise that read can be reordered before the percpu counter increment and a race window opens...
But we could make things faster by something like:
__sb_end_write()
rcu_read_lock(); percpu_counter_dec(&sb->s_writers.counter[level-1]); if (unlikely(sb->s_writers.frozen >= level)) wake_up(&sb->s_writers.wait); rcu_read_unlock();
So the synchronize_rcu() calls you've added in the first patch will make sure that all __sb_end_write() calls after we've started the freeze procedure will end up calling wake_up() and so the process waiting in sb_wait_write() will be woken as necessary. But please add a detailed comment about the synchronization because its tricky and uncommon...
Honza
Quoted 6 lines
fs/super.c renamed from fs/super.c~selectively-do-barriers-in-__sb_end_write +1
-0wake_up (&sb->s_writers.wait); + }rwsem_release (&sb->s_writers.lock_map[level-1 ],1 , _RET_IP_); }EXPORT_SYMBOL (__sb_end_write);_
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Jan Kara
reviewerQuoted 29 lines · Dave Hansen
Currently,
__sb_start_write()andfreeze_super()can race with each other.__sb_start_write()uses asmp_mb()to ensure thatfreeze_super()can see its write tosb->s_writers.counterand that it can see freeze_super()'s update tosb->s_writers.frozen. This all seems to work fine.But, this
smp_mb()makes__sb_start_write()the single hottest function in the kernel if I sit in a loop and do tiny write()s to tmpfs over and over. This is on a very small 2-core system, so it will only get worse on larger systems.This _seems_ like an ideal case for RCU.
__sb_start_write()is the RCU read-side and is in a very fast, performance-sensitive path.freeze_super()is the RCU writer and is in an extremely rare non-performance-sensitive path.Instead of doing and
smp_wmb()in__sb_start_write(), we dorcu_read_lock(). This ensures that a CPU doingfreeze_super()can not proceed past itssynchronize_rcu()until the grace period has ended and the 's_writers.frozen =SB_FREEZE_WRITE' is visible to__sb_start_write().One question here: Does the work that
__sb_start_write()does in a previous grace period becomes visible tofreeze_super()after its call tosynchronize_rcu()? It _seems_ like it should, but it seems backwards to me since__sb_start_write()is the "reader" in this case.
Quoted 4 lines
This patch increases the number of writes/second that I can do by 10.4%.
Does anybody see any holes with this?
Nice speed up and looks good to me. Just one question below.
fs/super.c int freeze_super(struct super_block *sb) · line 1347
Quoted 6 lines
fs/super.c renamed from fs/super.c~rcu-__sb_start_write +1
-1@@ -1340,7 +1344,7 @@ int freeze_super(struct super_block *sb)printk (KERN_ERR "VFS:Filesystem freeze failed\n" ); sb->s_writers.frozen =SB_UNFROZEN ; -smp_wmb (); +synchronize_rcu ();
Do we really need synchronize_rcu() here? We just need to make sure write to sb->s_writers.frozen happens before we start waking processes...
fs/super.c int thaw_super(struct super_block *sb) · line 1395
Quoted 10 lines
fs/super.c renamed from fs/super.c~rcu-__sb_start_write +1
-1wake_up (&sb->s_writers.wait_unfrozen);deactivate_locked_super (sb);return ret;@@ -1387,7 +1391,7 @@ int thaw_super(struct super_block *sb) out: sb->s_writers.frozen =SB_UNFROZEN ; -smp_wmb (); +synchronize_rcu ();wake_up (&sb->s_writers.wait_unfrozen);
And here as well...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 9 lines · Jan Kara
fs/super.c int freeze_super(struct super_block *sb) · line 1347
Quoted 6 lines
fs/super.c renamed from fs/super.c~rcu-__sb_start_write +1
-1@@ -1340,7 +1344,7 @@ int freeze_super(struct super_block *sb)printk (KERN_ERR "VFS:Filesystem freeze failed\n" ); sb->s_writers.frozen =SB_UNFROZEN ; -smp_wmb (); +synchronize_rcu ();Do we really need
synchronize_rcu()here? We just need to make sure write tosb->s_writers.frozenhappens before we start waking processes...
I don't think it is necessary. We only need to be concerned in practice if someone could be inside a critical section when we are executing this. I *think* the only case that we have that really matters will be taken care of by the _first_ synchronize_rcu().
It's definitely worth adding a comment.