Skip to contentSkip to conversation

srcu: API for barrier after srcu read unlock

PATCH In review Michael S. Tsirkin 3 participants 5 messages Nov 4, 2013 – Nov 6, 2013

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 srcu: API for barrier after srcu read unlock

srcu read lock/unlock include a full memory barrier but that's an implementation detail.
Add an API for make memory fencing explicit for users that need this barrier, to make sure we can change it as needed without breaking all users.

---
 include/linux/srcu.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
include/linux/srcu.h +14 -0
index c114614..9b058ee 100644
@@ -237,4 +237,18 @@ static inline void srcu_read_unlock(struct srcu_struct *sp, int idx)
 	__srcu_read_unlock(sp, idx);
 }
 
+/**
+ * smp_mb__after_srcu_read_unlock - ensure full ordering after srcu_read_unlock
+ *
+ * Converts the preceding srcu_read_unlock into a two-way memory barrier.
+ *
+ * Call this after srcu_read_unlock, to guarantee that all memory operations
+ * that occur after smp_mb__after_srcu_read_unlock will appear to happen after
+ * the preceding srcu_read_unlock.
+ */
+static inline void smp_mb__after_srcu_read_unlock(void)
+{
+	/* __srcu_read_unlock has smp_mb() internally so nothing to do here. */
+}
+
 #endif
Signature · 1 line
MST
#1 lore raw

PATCH 2/2 kvm: optimize out smp_mb after srcu_read_unlock

I noticed that srcu_read_lock/unlock both have a memory barrier, so just by moving srcu_read_unlock earlier we can get rid of one call to smp_mb() using smp_mb__after_srcu_read_unlock instead.

Unsurprisingly, the gain is small but measureable using the unit test microbenchmark:
before
vmcall in the ballpark of 1410 cycles
after
vmcall in the ballpark of 1360 cycles

---
 arch/x86/kvm/x86.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
arch/x86/kvm/x86.c +4 -3
index e5ca72a..d609dce 100644
@@ -5945,10 +5945,12 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 
 	vcpu->mode = IN_GUEST_MODE;
 
+	srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx);
+
 	/* We should set ->mode before check ->requests,
 	 * see the comment in make_all_cpus_request.
 	 */
-	smp_mb();
+	smp_mb__after_srcu_read_unlock();
 
 	local_irq_disable();
 
@@ -5958,12 +5960,11 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 		smp_wmb();
 		local_irq_enable();
 		preempt_enable();
+		vcpu->srcu_idx = srcu_read_lock(&vcpu->kvm->srcu);
 		r = 1;
 		goto cancel_injection;
 	}
 
-	srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx);
-
 	if (req_immediate_exit)
 		smp_send_reschedule(vcpu->cpu);
 
Signature · 1 line
MST
Reviewed-by on v1 1/2
#2 lore raw
In reply to Michael S. Tsirkin
Quoted 8 lines · Michael S. Tsirkin

srcu read lock/unlock include a full memory barrier but that's an implementation detail.
Add an API for make memory fencing explicit for users that need this barrier, to make sure we can change it as needed without breaking all users.

Quoted 30 lines
---
 include/linux/srcu.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
include/linux/srcu.h +14 -0
index c114614..9b058ee 100644
@@ -237,4 +237,18 @@ static inline void srcu_read_unlock(struct srcu_struct *sp, int idx)
 	__srcu_read_unlock(sp, idx);
 }
 
+/**
+ * smp_mb__after_srcu_read_unlock - ensure full ordering after srcu_read_unlock
+ *
+ * Converts the preceding srcu_read_unlock into a two-way memory barrier.
+ *
+ * Call this after srcu_read_unlock, to guarantee that all memory operations
+ * that occur after smp_mb__after_srcu_read_unlock will appear to happen after
+ * the preceding srcu_read_unlock.
+ */
+static inline void smp_mb__after_srcu_read_unlock(void)
+{
+	/* __srcu_read_unlock has smp_mb() internally so nothing to do here. */
+}
+
 #endif
Signature · 1 line
MST
Acked-by
#3 lore raw
Quoted 11 lines · Paul E. McKenney
Quoted 8 lines · Michael S. Tsirkin

srcu read lock/unlock include a full memory barrier but that's an implementation detail.
Add an API for make memory fencing explicit for users that need this barrier, to make sure we can change it as needed without breaking all users.

Quoted 38 lines
Quoted 31 lines
---
 include/linux/srcu.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
include/linux/srcu.h +14 -0
index c114614..9b058ee 100644
@@ -237,4 +237,18 @@ static inline void srcu_read_unlock(struct srcu_struct *sp, int idx)
 	__srcu_read_unlock(sp, idx);
 }
 
+/**
+ * smp_mb__after_srcu_read_unlock - ensure full ordering after srcu_read_unlock
+ *
+ * Converts the preceding srcu_read_unlock into a two-way memory barrier.
+ *
+ * Call this after srcu_read_unlock, to guarantee that all memory operations
+ * that occur after smp_mb__after_srcu_read_unlock will appear to happen after
+ * the preceding srcu_read_unlock.
+ */
+static inline void smp_mb__after_srcu_read_unlock(void)
+{
+	/* __srcu_read_unlock has smp_mb() internally so nothing to do here. */
+}
+
 #endif
Signature · 1 line
MST

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#4 lore raw
Quoted 8 lines · Lai Jiangshan
Quoted 7 lines · Paul E. McKenney
Quoted 6 lines · Michael S. Tsirkin

srcu read lock/unlock include a full memory barrier but that's an implementation detail.
Add an API for make memory fencing explicit for users that need this barrier, to make sure we can change it as needed without breaking all users.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Very good, queued for 3.14.

Thanx, Paul

Quoted 38 lines
Quoted 37 lines
Quoted 30 lines
---
 include/linux/srcu.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
include/linux/srcu.h +14 -0
index c114614..9b058ee 100644
@@ -237,4 +237,18 @@ static inline void srcu_read_unlock(struct srcu_struct *sp, int idx)
 	__srcu_read_unlock(sp, idx);
 }
 
+/**
+ * smp_mb__after_srcu_read_unlock - ensure full ordering after srcu_read_unlock
+ *
+ * Converts the preceding srcu_read_unlock into a two-way memory barrier.
+ *
+ * Call this after srcu_read_unlock, to guarantee that all memory operations
+ * that occur after smp_mb__after_srcu_read_unlock will appear to happen after
+ * the preceding srcu_read_unlock.
+ */
+static inline void smp_mb__after_srcu_read_unlock(void)
+{
+	/* __srcu_read_unlock has smp_mb() internally so nothing to do here. */
+}
+
 #endif
Signature · 1 line
MST

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/