Skip to contentSkip to conversation

qemu/virtio: make wmb compiler barrier + comments

PATCH In review Michael S. Tsirkin 4 participants 17 messages Oct 26, 2009 – Nov 13, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 qemu/virtio: make wmb compiler barrier + comments

wmb must be at least a compiler barrier, even without SMP. Further, we likely need some rmb()/mb() as well:
I have not audited the code but lguest has mb(), add a comment for now.

---
 hw/virtio.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)
hw/virtio.c +4 -1
index 337ff27..1f92171 100644
@@ -23,8 +23,11 @@
 /* QEMU doesn't strictly need write barriers since everything runs in
  * lock-step.  We'll leave the calls to wmb() in though to make it obvious for
  * KVM or if kqemu gets SMP support.
+ * In any case, we must prevent the compiler from reordering the code.
+ * TODO: we likely need some rmb()/mb() as well.
  */
-#define wmb() do { } while (0)
+
+#define wmb() __asm__ __volatile__("": : :"memory")
 
 typedef struct VRingDesc
 {
Signature · 1 line
1.6.5.rc2
#1 lore raw
Quoted 1 line · Michael S. Tsirkin

wmb must be at least a compiler barrier, even without SMP.

Why?

Paul

#2 lore raw
Quoted 4 lines · Paul Brook
Quoted 1 line · Michael S. Tsirkin

wmb must be at least a compiler barrier, even without SMP.

Why?

Because virtio code might run on a separate thread from guest. If compiler reorders writes, guest might see inconsistent data.

Signature · 1 line
MST
#3 lore raw
Quoted 8 lines · Michael S. Tsirkin
Quoted 4 lines · Paul Brook
Quoted 1 line · Michael S. Tsirkin

wmb must be at least a compiler barrier, even without SMP.

Why?

Because virtio code might run on a separate thread from guest. If compiler reorders writes, guest might see inconsistent data.

If you've got threads running in parallel (which may be running on separate CPUs) then you need an actual memory barrier to prevent the hardware reordering things behind your back.

If you've already used locking to avoid simultaneous execution then the locking routines already include memory barriers.

A "compiler memory barrier" provides absolutely no guarantees in a multithreaded environment. They are sometimes useful in a single threaded interruptable system (i.e. UNIX signals), but that's definitely not the case here.

Paul

#4 lore raw
Quoted 5 lines · Paul Brook
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Paul Brook
Quoted 1 line · Michael S. Tsirkin

wmb must be at least a compiler barrier, even without SMP.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Why?

Because virtio code might run on a separate thread from guest. If compiler reorders writes, guest might see inconsistent data.

If you've got threads running in parallel (which may be running on separate CPUs)

Yes, but you asked what happens without SMP (single CPU).

Quoted 5 lines

then you need an actual memory barrier to prevent the hardware reordering things behind your back.

If you've already used locking to avoid simultaneous execution then the locking routines already include memory barriers.

You can not share a lock with guest.

Quoted 6 lines

A "compiler memory barrier" provides absolutely no guarantees in a multithreaded environment. They are sometimes useful in a single threaded interruptable system (i.e. UNIX signals), but that's definitely not the case here.

Paul

"absolutely no guarantees" is surely wrong. On intel CPUs, regular memory writes are never re-ordered by the CPU. Only compiler can reorder such writes. So yes, on this platform a "compiler barrier" does provide necessary and sufficient guarantees agains write reordering in a multithreaded environment, both with and without SMP.

Signature · 1 line
MST
#16 lore raw
In reply to Paul Brook
Quoted 2 lines · Paul Brook

A "compiler memory barrier" provides absolutely no guarantees in a multithreaded environment.

Claims including words "absolutely no" have absolutely no chance to be correct.

Signature · 1 line
MST
#5 lore raw
In reply to Michael S. Tsirkin
Quoted 11 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

wmb must be at least a compiler barrier, even without SMP.

Why?

Because virtio code might run on a separate thread from guest. If compiler reorders writes, guest might see inconsistent data.

If you've got threads running in parallel (which may be running on separate CPUs)

Yes, but you asked what happens without SMP (single CPU).

I assumed you meant guest SMP where guest code can run in parallel with host code. My understanding is that qemu currently does not implement this, and everything always runs in lockstep.

Quoted 7 lines
Quoted 5 lines

then you need an actual memory barrier to prevent the hardware reordering things behind your back.

If you've already used locking to avoid simultaneous execution then the locking routines already include memory barriers.

You can not share a lock with guest.

No, but you can prevent the guest code running at the same time as host code.

Quoted 7 lines
Quoted 4 lines

A "compiler memory barrier" provides absolutely no guarantees in a multithreaded environment. They are sometimes useful in a single threaded interruptable system (i.e. UNIX signals), but that's definitely not the case here.

"absolutely no guarantees" is surely wrong. On intel CPUs, regular memory writes are never re-ordered by the CPU.

Technically true, however reads and writes may be reordered.
If you don't need real barriers, then why does the kvm code have them? Also, there's no indication that your code is specific to Intel CPUs.

Paul

#6 lore raw
Quoted 1 line · Paul Brook

If you don't need real barriers, then why does the kvm code have them?

We need real barriers but AFAIK kvm does not have them :(
IOW: virtio is currently broken with kvm, and my patch did not fix this. The comment that I added says as much.

Signature · 1 line
MST
#7 lore raw
Quoted 6 lines · Michael S. Tsirkin
Quoted 1 line · Paul Brook

If you don't need real barriers, then why does the kvm code have them?

We need real barriers but AFAIK kvm does not have them :(
IOW: virtio is currently broken with kvm, and my patch did not fix this. The comment that I added says as much.

So your code just makes the bug harder to reproduce? Doesn't sound like a good thing to me.

Paul

#8 lore raw
Quoted 4 lines · Paul Brook
Quoted 3 lines · Michael S. Tsirkin
Quoted 1 line · Paul Brook

If you don't need real barriers, then why does the kvm code have them?

Quoted 8 lines
Quoted 3 lines

We need real barriers but AFAIK kvm does not have them :(
IOW: virtio is currently broken with kvm, and my patch did not fix this. The comment that I added says as much.

So your code just makes the bug harder to reproduce? Doesn't sound like a good thing to me.

Paul

There are multiple bugs, I can't fix all of them,
for all architectures, in one patch.

Signature · 1 line
MST
#9 lore raw
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Paul Brook
Quoted 7 lines · Michael S. Tsirkin
Quoted 2 lines · Paul Brook

If you don't need real barriers, then why does the kvm code have them?

We need real barriers but AFAIK kvm does not have them :(
IOW: virtio is currently broken with kvm, and my patch did not fix this. The comment that I added says as much.

Quoted 7 lines
Quoted 4 lines

So your code just makes the bug harder to reproduce? Doesn't sound like a good thing to me.

Paul

There are multiple bugs, I can't fix all of them,
for all architectures, in one patch.

Maybe not, but the associated comment now appears to be bogus. Either we execute everything in lockstep (as the comment says, in which case no barriers are required), or we need proper SMP barriers.
Having a half-harted barrier that maybe works some of the time on some hosts, and is completely unnecessary in most cases, and doesn't match the comment is IMO significantly worse than what we had before.

Paul

#10 lore raw
In reply to Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 1 line · Paul Brook

If you don't need real barriers, then why does the kvm code have them?

We need real barriers but AFAIK kvm does not have them :(
IOW: virtio is currently broken with kvm, and my patch did not fix this. The comment that I added says as much.

How about just using GCC's __sync__synchronize atomic builtin (if detected as available by configure)?
It's a full memory barrier instead of just a write barrier, for x86, it generates the same code as the current Linux mb() implementation:
"mfence" on x86_64
"lock orl $0x0,(%esp)" on x86 unless -march is specified to a processor with "mfence".
PPC could continue to use "eieio" while other architectures could just default to __sync_synchronize

I do have a newbie question, when exactly would vrtio have to handle concurrent access from multiple threads?
My current reading of the code suggests:
1. when CONFIG_IOTHREAD is true
2. when CONFIG_KVM is true and the guest machine has multiple CPUs

#11 lore raw
Quoted 8 lines · Scott Tsai
Quoted 6 lines · Michael S. Tsirkin
Quoted 1 line · Paul Brook

If you don't need real barriers, then why does the kvm code have them?

We need real barriers but AFAIK kvm does not have them :(
IOW: virtio is currently broken with kvm, and my patch did not fix this. The comment that I added says as much.

Quoted 9 lines

How about just using GCC's __sync__synchronize atomic builtin (if detected as available by configure)?
It's a full memory barrier instead of just a write barrier, for x86, it generates the same code as the current Linux mb() implementation:
"mfence" on x86_64
"lock orl $0x0,(%esp)" on x86 unless -march is specified to a processor with "mfence".
PPC could continue to use "eieio" while other architectures could just default to __sync_synchronize

Hmm, on x86 that's more expensive than it needs to be... We can also ifdef platforms that we care about ...

Quoted 5 lines

I do have a newbie question, when exactly would vrtio have to handle concurrent access from multiple threads?
My current reading of the code suggests:
1. when CONFIG_IOTHREAD is true
2. when CONFIG_KVM is true and the guest machine has multiple CPUs

#12 lore raw
In reply to Scott Tsai
Quoted 18 lines · Scott Tsai
Quoted 6 lines · Michael S. Tsirkin
Quoted 1 line · Paul Brook

If you don't need real barriers, then why does the kvm code have them?

We need real barriers but AFAIK kvm does not have them :(
IOW: virtio is currently broken with kvm, and my patch did not fix this. The comment that I added says as much.

How about just using GCC's __sync__synchronize atomic builtin (if detected as available by configure)?
It's a full memory barrier instead of just a write barrier, for x86, it generates the same code as the current Linux mb() implementation:
"mfence" on x86_64
"lock orl $0x0,(%esp)" on x86 unless -march is specified to a processor with "mfence".
PPC could continue to use "eieio" while other architectures could just default to __sync_synchronize

Quoted 5 lines

I do have a newbie question, when exactly would vrtio have to handle concurrent access from multiple threads?
My current reading of the code suggests:
1. when CONFIG_IOTHREAD is true
2. when CONFIG_KVM is true and the guest machine has multiple CPUs

Right. I don't think CONFIG_IOTHREAD can work correctly without kvm though: how would atomics be handled?

Signature · 1 line
MST
#13 lore raw
Quoted 8 lines · Michael S. Tsirkin
Quoted 5 lines

I do have a newbie question, when exactly would vrtio have to handle concurrent access from multiple threads?
My current reading of the code suggests:
1. when CONFIG_IOTHREAD is true
2. when CONFIG_KVM is true and the guest machine has multiple CPUs

Right. I don't think CONFIG_IOTHREAD can work correctly without kvm though: how would atomics be handled?

I naively imagined it to work like this:
When CONFIG_IOTHREAD is true and CONFIG_KVM is false,
all the tcg CPUs run in the tcg_cpu_thread and device emulation code runs in io_thread,
so if the tcg translators generate suitable memory barrier instructions when it sees a "lfence", "sfence", "mfence" instruction while emulating a x86 or "sync" while emulating a MIPS everything should work but a quick look at target-*/translate.c suggests memory barrier instructions are treated as nops.

So maybe --enable-io-thread while --disable-kvm should not be allowed at configure time.
Does anyone actually ship qemu with CONFIG_IOTHREAD enabled?

#14 lore raw
Quoted 10 lines · Scott Tsai
Quoted 8 lines · Michael S. Tsirkin
Quoted 5 lines

I do have a newbie question, when exactly would vrtio have to handle concurrent access from multiple threads?
My current reading of the code suggests:
1. when CONFIG_IOTHREAD is true
2. when CONFIG_KVM is true and the guest machine has multiple CPUs

Right. I don't think CONFIG_IOTHREAD can work correctly without kvm though: how would atomics be handled?

Quoted 8 lines

I naively imagined it to work like this:
When CONFIG_IOTHREAD is true and CONFIG_KVM is false,
all the tcg CPUs run in the tcg_cpu_thread and device emulation code runs in io_thread,
so if the tcg translators generate suitable memory barrier instructions when it sees a "lfence", "sfence", "mfence" instruction while emulating a x86 or "sync" while emulating a MIPS everything should work

that might not be enough. guest can do e.g. atomics on the same memory with iothread. In parctice, with virtio it doesn't.

Quoted 6 lines

but a quick look at target-*/translate.c suggests memory barrier instructions are treated as nops.

So maybe --enable-io-thread while --disable-kvm should not be allowed at configure time.
Does anyone actually ship qemu with CONFIG_IOTHREAD enabled?

#15 lore raw
In reply to Scott Tsai
Quoted 5 lines · Scott Tsai

I do have a newbie question, when exactly would vrtio have to handle concurrent access from multiple threads?
My current reading of the code suggests:
1. when CONFIG_IOTHREAD is true
2. when CONFIG_KVM is true and the guest machine has multiple CPUs

It's enough to have CONFIG_KVM with a single guest CPU, isn't it?

Because the guest CPU can still run concurrently with the I/O thread on kvm, and the host may have multiple CPUs even though the guest does not.

-- Jamie