Skip to contentSkip to conversation

Fix memory order issue inside pv spinlock

PATCH In review Yang, Xiaowei 2 participants 6 messages Sep 7, 2009 – Sep 10, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Fix memory order issue inside pv spinlock

barrier() can't prevent reads after it not being reordered with older writes to different locates before it. Because of it, I can't bring up > 4 HVM guests on one SMP machine. Use mb() instead.

arch/x86/xen/spinlock.c +1 -1
index 5601506..9dee5f8 100644
@@ -324,7 +325,7 @@ static void xen_spin_unlock(struct raw_spinlock *lock)
        xl->lock = 0;           /* release lock */
 
        /* make sure unlock happens before kick */
-       barrier();
+       mb();
 
        if (unlikely(xl->spinners))
                xen_spin_unlock_slow(xl);

Thanks,
Xiaowei

#1 lore raw
Quoted 3 lines · Xiaowei

barrier() can't prevent reads after it not being reordered with older writes to different locates before it. Because of it, I can't bring up

Quoted 1 line

4 HVM guests on one SMP machine. Use mb() instead.

Which read is happening too early? Is it "xl->spinners"? How does it fail?

Thanks,
J

Quoted 21 lines
arch/x86/xen/spinlock.c +1 -1
index 5601506..9dee5f8 100644
@@ -324,7 +325,7 @@ static void xen_spin_unlock(struct raw_spinlock
 *lock)
        xl->lock = 0;           /* release lock */
 
        /* make sure unlock happens before kick */
-       barrier();
+       mb();
 
        if (unlikely(xl->spinners))
                xen_spin_unlock_slow(xl);

Thanks,
Xiaowei

#2 lore raw
Quoted 6 lines · Jeremy Fitzhardinge
Quoted 3 lines · Xiaowei

barrier() can't prevent reads after it not being reordered with older writes to different locates before it. Because of it, I can't bring up

Quoted 1 line

4 HVM guests on one SMP machine. Use mb() instead.

Which read is happening too early? Is it "xl->spinners"? How does it fail?

Yes. If read of xl->spinners happens earlier than write 0 to xl->lock, notifications to wake up other spinners can be omitted incorrectly, resulting in others polling indefinitely (because of poll evtchn not pending) with the lock is uncontended.

Thanks,
xiaowei

Quoted 25 lines

Thanks,
J

Quoted 20 lines
arch/x86/xen/spinlock.c +1 -1
index 5601506..9dee5f8 100644
@@ -324,7 +325,7 @@ static void xen_spin_unlock(struct raw_spinlock
 *lock)
        xl->lock = 0;           /* release lock */
 
        /* make sure unlock happens before kick */
-       barrier();
+       mb();
 
        if (unlikely(xl->spinners))
                xen_spin_unlock_slow(xl);

Thanks,
Xiaowei

#3 lore raw
Quoted 6 lines · Xiaowei
Quoted 5 lines · Jeremy Fitzhardinge
Quoted 3 lines · Xiaowei

barrier() can't prevent reads after it not being reordered with older writes to different locates before it. Because of it, I can't bring up

Quoted 1 line

4 HVM guests on one SMP machine. Use mb() instead.

Quoted 7 lines
Quoted 2 lines

Which read is happening too early? Is it "xl->spinners"? How does it fail?

Yes. If read of xl->spinners happens earlier than write 0 to xl->lock, notifications to wake up other spinners can be omitted incorrectly, resulting in others polling indefinitely (because of poll evtchn not pending) with the lock is uncontended.

OK. And the CPU only guarantees that, without explicit barriers, write-read ordering is only maintained between accesses to the same memory location, not separate locations?

J

#4 lore raw
Quoted 8 lines · Jeremy Fitzhardinge
Quoted 7 lines · Xiaowei
Quoted 6 lines · Jeremy Fitzhardinge
Quoted 3 lines · Xiaowei

barrier() can't prevent reads after it not being reordered with older writes to different locates before it. Because of it, I can't bring up

Quoted 1 line

4 HVM guests on one SMP machine. Use mb() instead.

Which read is happening too early? Is it "xl->spinners"? How does it fail?

Quoted 9 lines
Quoted 4 lines

Yes. If read of xl->spinners happens earlier than write 0 to xl->lock, notifications to wake up other spinners can be omitted incorrectly, resulting in others polling indefinitely (because of poll evtchn not pending) with the lock is uncontended.

OK. And the CPU only guarantees that, without explicit barriers, write-read ordering is only maintained between accesses to the same memory location, not separate locations?

Exactly! For more details please refer to Chapter 8.2 of Intel SDM 3A.

Thanks,
Xiaowei

#5 lore raw
Quoted 16 lines · Xiaowei
Quoted 14 lines · Jeremy Fitzhardinge
Quoted 12 lines · Xiaowei
Quoted 7 lines · Jeremy Fitzhardinge
Quoted 4 lines · Xiaowei

barrier() can't prevent reads after it not being reordered with older writes to different locates before it. Because of it, I can't bring up

Quoted 1 line

4 HVM guests on one SMP machine. Use mb() instead.

Which read is happening too early? Is it "xl->spinners"? How does it fail?

Yes. If read of xl->spinners happens earlier than write 0 to xl->lock, notifications to wake up other spinners can be omitted incorrectly, resulting in others polling indefinitely (because of poll evtchn not pending) with the lock is uncontended.

Quoted 6 lines
Quoted 4 lines

OK. And the CPU only guarantees that, without explicit barriers, write-read ordering is only maintained between accesses to the same memory location, not separate locations?

Exactly! For more details please refer to Chapter 8.2 of Intel SDM 3A.

Thanks. I just submitted both fixes to upstream.

J