srcu: API for barrier after srcu read unlock
Subsystems (4)
PATCH 1/2 srcu: API for barrier after srcu read unlock
srcu read lock/unlock include a full memory barrier but that's an implementation detail.
Add an API for make memory fencing explicit for users that need this barrier, to make sure we can change it as needed without breaking all users.
--- include/linux/srcu.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
include/linux/srcu.h +14 -0
@@ -237,4 +237,18 @@ static inline void srcu_read_unlock(struct srcu_struct *sp, int idx) __srcu_read_unlock(sp, idx); } +/** +* smp_mb__after_srcu_read_unlock - ensure full ordering after srcu_read_unlock +* +* Converts the preceding srcu_read_unlock into a two-way memory barrier. +* +* Call this after srcu_read_unlock, to guarantee that all memory operations +* that occur after smp_mb__after_srcu_read_unlock will appear to happen after +* the preceding srcu_read_unlock. +*/ +static inline void smp_mb__after_srcu_read_unlock (void ) +{ +/* __srcu_read_unlock has smp_mb() internally so nothing to do here. */ +} +#endif
Signature · 1 line
MST
PATCH 2/2 kvm: optimize out smp_mb after srcu_read_unlock
I noticed that srcu_read_lock/unlock both have a memory barrier, so just by moving srcu_read_unlock earlier we can get rid of one call to smp_mb() using smp_mb__after_srcu_read_unlock instead.
Unsurprisingly, the gain is small but measureable using the unit test microbenchmark:
before
vmcall in the ballpark of 1410 cycles
after
vmcall in the ballpark of 1360 cycles
--- arch/x86/kvm/x86.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
arch/x86/kvm/x86.c +4 -3
@@ -5945,10 +5945,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) vcpu->mode =IN_GUEST_MODE ; +srcu_read_unlock (&vcpu->kvm->srcu, vcpu->srcu_idx); +/* We should set ->mode before check ->requests, * see the comment in make_all_cpus_request. */ -smp_mb (); +smp_mb__after_srcu_read_unlock ();local_irq_disable ();
@@ -5958,12 +5960,11 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)smp_wmb ();local_irq_enable ();preempt_enable (); + vcpu->srcu_idx =srcu_read_lock (&vcpu->kvm->srcu); r =1 ; goto cancel_injection; } -srcu_read_unlock (&vcpu->kvm->srcu, vcpu->srcu_idx); -if (req_immediate_exit)smp_send_reschedule (vcpu->cpu);
Signature · 1 line
MST
Quoted 8 lines · Michael S. Tsirkin
srcu read lock/unlock include a full memory barrier but that's an implementation detail.
Add an API for make memory fencing explicit for users that need this barrier, to make sure we can change it as needed without breaking all users.Signed-off-by Michael S. Tsirkin <[email protected]>
Quoted 30 lines
--- include/linux/srcu.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+)include/linux/srcu.h +14
-0@@ -237,4 +237,18 @@ static inline void srcu_read_unlock(struct srcu_struct *sp, int idx) __srcu_read_unlock(sp, idx); } +/** +* smp_mb__after_srcu_read_unlock - ensure full ordering after srcu_read_unlock +* +* Converts the preceding srcu_read_unlock into a two-way memory barrier. +* +* Call this after srcu_read_unlock, to guarantee that all memory operations +* that occur after smp_mb__after_srcu_read_unlock will appear to happen after +* the preceding srcu_read_unlock. +*/ +static inline void smp_mb__after_srcu_read_unlock (void ) +{ +/* __srcu_read_unlock has smp_mb() internally so nothing to do here. */ +} +#endif Signature · 1 line
MST
Lai Jiangshan
Acked-byQuoted 11 lines · Paul E. McKenney
Quoted 8 lines · Michael S. Tsirkin
srcu read lock/unlock include a full memory barrier but that's an implementation detail.
Add an API for make memory fencing explicit for users that need this barrier, to make sure we can change it as needed without breaking all users.Signed-off-by Michael S. Tsirkin <[email protected]>Reviewed-by Paul E. McKenney <[email protected]>
Quoted 38 lines
Quoted 31 lines
--- include/linux/srcu.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+)include/linux/srcu.h +14
-0@@ -237,4 +237,18 @@ static inline void srcu_read_unlock(struct srcu_struct *sp, int idx) __srcu_read_unlock(sp, idx); } +/** +* smp_mb__after_srcu_read_unlock - ensure full ordering after srcu_read_unlock +* +* Converts the preceding srcu_read_unlock into a two-way memory barrier. +* +* Call this after srcu_read_unlock, to guarantee that all memory operations +* that occur after smp_mb__after_srcu_read_unlock will appear to happen after +* the preceding srcu_read_unlock. +*/ +static inline void smp_mb__after_srcu_read_unlock (void ) +{ +/* __srcu_read_unlock has smp_mb() internally so nothing to do here. */ +} +#endif Signature · 1 line
MST--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 8 lines · Lai Jiangshan
Quoted 7 lines · Paul E. McKenney
Quoted 6 lines · Michael S. Tsirkin
srcu read lock/unlock include a full memory barrier but that's an implementation detail.
Add an API for make memory fencing explicit for users that need this barrier, to make sure we can change it as needed without breaking all users.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Signed-off-by Michael S. Tsirkin <[email protected]>Reviewed-by Paul E. McKenney <[email protected]>Acked-by "Lai Jiangshan" <[email protected]>
Very good, queued for 3.14.
Thanx, Paul
Quoted 38 lines
Quoted 37 lines
Quoted 30 lines
--- include/linux/srcu.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+)include/linux/srcu.h +14
-0@@ -237,4 +237,18 @@ static inline void srcu_read_unlock(struct srcu_struct *sp, int idx) __srcu_read_unlock(sp, idx); } +/** +* smp_mb__after_srcu_read_unlock - ensure full ordering after srcu_read_unlock +* +* Converts the preceding srcu_read_unlock into a two-way memory barrier. +* +* Call this after srcu_read_unlock, to guarantee that all memory operations +* that occur after smp_mb__after_srcu_read_unlock will appear to happen after +* the preceding srcu_read_unlock. +*/ +static inline void smp_mb__after_srcu_read_unlock (void ) +{ +/* __srcu_read_unlock has smp_mb() internally so nothing to do here. */ +} +#endif Signature · 1 line
MST--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/