qemu/virtio: make wmb compiler barrier + comments
PATCH 1/1 qemu/virtio: make wmb compiler barrier + comments
wmb must be at least a compiler barrier, even without SMP. Further, we likely need some rmb()/mb() as well:
I have not audited the code but lguest has mb(), add a comment for now.
--- hw/virtio.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)
hw/virtio.c +4 -1
@@ -23,8 +23,11 @@/* QEMU doesn't strictly need write barriers since everything runs in * lock-step. We'll leave the calls to wmb() in though to make it obvious for * KVM or if kqemu gets SMP support. +* In any case, we must prevent the compiler from reordering the code. +* TODO: we likely need some rmb()/mb() as well. */ -#define wmb () do { } while (0) + +#define wmb () __asm__ __volatile__("": : :"memory") typedef struct VRingDesc {
Signature · 1 line
1.6.5.rc2
Quoted 1 line · Michael S. Tsirkin
wmb must be at least a compiler barrier, even without SMP.
Why?
Paul
Quoted 4 lines · Paul Brook
Quoted 1 line · Michael S. Tsirkin
wmb must be at least a compiler barrier, even without SMP.
Why?
Because virtio code might run on a separate thread from guest. If compiler reorders writes, guest might see inconsistent data.
Signature · 1 line
MST
Quoted 8 lines · Michael S. Tsirkin
Quoted 4 lines · Paul Brook
Quoted 1 line · Michael S. Tsirkin
wmb must be at least a compiler barrier, even without SMP.
Why?
Because virtio code might run on a separate thread from guest. If compiler reorders writes, guest might see inconsistent data.
If you've got threads running in parallel (which may be running on separate CPUs) then you need an actual memory barrier to prevent the hardware reordering things behind your back.
If you've already used locking to avoid simultaneous execution then the locking routines already include memory barriers.
A "compiler memory barrier" provides absolutely no guarantees in a multithreaded environment. They are sometimes useful in a single threaded interruptable system (i.e. UNIX signals), but that's definitely not the case here.
Paul
Quoted 5 lines · Paul Brook
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Paul Brook
Quoted 1 line · Michael S. Tsirkin
wmb must be at least a compiler barrier, even without SMP.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Why?
Because virtio code might run on a separate thread from guest. If compiler reorders writes, guest might see inconsistent data.
If you've got threads running in parallel (which may be running on separate CPUs)
Yes, but you asked what happens without SMP (single CPU).
Quoted 5 lines
then you need an actual memory barrier to prevent the hardware reordering things behind your back.
If you've already used locking to avoid simultaneous execution then the locking routines already include memory barriers.
You can not share a lock with guest.
Quoted 6 lines
A "compiler memory barrier" provides absolutely no guarantees in a multithreaded environment. They are sometimes useful in a single threaded interruptable system (i.e. UNIX signals), but that's definitely not the case here.
Paul
"absolutely no guarantees" is surely wrong. On intel CPUs, regular memory writes are never re-ordered by the CPU. Only compiler can reorder such writes. So yes, on this platform a "compiler barrier" does provide necessary and sufficient guarantees agains write reordering in a multithreaded environment, both with and without SMP.
Signature · 1 line
MST
Quoted 11 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
wmb must be at least a compiler barrier, even without SMP.
Why?
Because virtio code might run on a separate thread from guest. If compiler reorders writes, guest might see inconsistent data.
If you've got threads running in parallel (which may be running on separate CPUs)
Yes, but you asked what happens without SMP (single CPU).
I assumed you meant guest SMP where guest code can run in parallel with host code. My understanding is that qemu currently does not implement this, and everything always runs in lockstep.
Quoted 7 lines
Quoted 5 lines
then you need an actual memory barrier to prevent the hardware reordering things behind your back.
If you've already used locking to avoid simultaneous execution then the locking routines already include memory barriers.
You can not share a lock with guest.
No, but you can prevent the guest code running at the same time as host code.
Quoted 7 lines
Quoted 4 lines
A "compiler memory barrier" provides absolutely no guarantees in a multithreaded environment. They are sometimes useful in a single threaded interruptable system (i.e. UNIX signals), but that's definitely not the case here.
"absolutely no guarantees" is surely wrong. On intel CPUs, regular memory writes are never re-ordered by the CPU.
Technically true, however reads and writes may be reordered.
If you don't need real barriers, then why does the kvm code have them? Also, there's no indication that your code is specific to Intel CPUs.
Paul
Quoted 1 line · Paul Brook
If you don't need real barriers, then why does the kvm code have them?
We need real barriers but AFAIK kvm does not have them :(
IOW: virtio is currently broken with kvm, and my patch did not fix this. The comment that I added says as much.
Signature · 1 line
MST
Quoted 6 lines · Michael S. Tsirkin
Quoted 1 line · Paul Brook
If you don't need real barriers, then why does the kvm code have them?
We need real barriers but AFAIK kvm does not have them :(
IOW: virtio is currently broken with kvm, and my patch did not fix this. The comment that I added says as much.
So your code just makes the bug harder to reproduce? Doesn't sound like a good thing to me.
Paul
Quoted 4 lines · Paul Brook
Quoted 3 lines · Michael S. Tsirkin
Quoted 1 line · Paul Brook
If you don't need real barriers, then why does the kvm code have them?
Quoted 8 lines
Quoted 3 lines
We need real barriers but AFAIK kvm does not have them :(
IOW: virtio is currently broken with kvm, and my patch did not fix this. The comment that I added says as much.So your code just makes the bug harder to reproduce? Doesn't sound like a good thing to me.
Paul
There are multiple bugs, I can't fix all of them,
for all architectures, in one patch.
Signature · 1 line
MST
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Paul Brook
Quoted 7 lines · Michael S. Tsirkin
Quoted 2 lines · Paul Brook
If you don't need real barriers, then why does the kvm code have them?
We need real barriers but AFAIK kvm does not have them :(
IOW: virtio is currently broken with kvm, and my patch did not fix this. The comment that I added says as much.
Quoted 7 lines
Quoted 4 lines
So your code just makes the bug harder to reproduce? Doesn't sound like a good thing to me.
Paul
There are multiple bugs, I can't fix all of them,
for all architectures, in one patch.
Maybe not, but the associated comment now appears to be bogus. Either we execute everything in lockstep (as the comment says, in which case no barriers are required), or we need proper SMP barriers.
Having a half-harted barrier that maybe works some of the time on some hosts, and is completely unnecessary in most cases, and doesn't match the comment is IMO significantly worse than what we had before.
Paul
Quoted 6 lines · Michael S. Tsirkin
Quoted 1 line · Paul Brook
If you don't need real barriers, then why does the kvm code have them?
We need real barriers but AFAIK kvm does not have them :(
IOW: virtio is currently broken with kvm, and my patch did not fix this. The comment that I added says as much.
How about just using GCC's __sync__synchronize atomic builtin (if detected as available by configure)?
It's a full memory barrier instead of just a write barrier, for x86, it generates the same code as the current Linux mb() implementation:
"mfence" on x86_64
"lock orl $0x0,(%esp)" on x86 unless -march is specified to a processor with "mfence".
PPC could continue to use "eieio" while other architectures could just default to __sync_synchronize
I do have a newbie question, when exactly would vrtio have to handle concurrent access from multiple threads?
My current reading of the code suggests:
1. when CONFIG_IOTHREAD is true
2. when CONFIG_KVM is true and the guest machine has multiple CPUs
Quoted 8 lines · Scott Tsai
Quoted 6 lines · Michael S. Tsirkin
Quoted 1 line · Paul Brook
If you don't need real barriers, then why does the kvm code have them?
We need real barriers but AFAIK kvm does not have them :(
IOW: virtio is currently broken with kvm, and my patch did not fix this. The comment that I added says as much.
Quoted 9 lines
How about just using GCC's
__sync__synchronizeatomic builtin (if detected as available by configure)?
It's a full memory barrier instead of just a write barrier, for x86, it generates the same code as the current Linuxmb()implementation:
"mfence" onx86_64
"lock orl $0x0,(%esp)" on x86 unless -march is specified to a processor with "mfence".
PPC could continue to use "eieio" while other architectures could just default to__sync_synchronize
Hmm, on x86 that's more expensive than it needs to be... We can also ifdef platforms that we care about ...
Quoted 5 lines
I do have a newbie question, when exactly would vrtio have to handle concurrent access from multiple threads?
My current reading of the code suggests:
1. whenCONFIG_IOTHREADis true
2. whenCONFIG_KVMis true and the guest machine has multiple CPUs
Quoted 18 lines · Scott Tsai
Quoted 6 lines · Michael S. Tsirkin
Quoted 1 line · Paul Brook
If you don't need real barriers, then why does the kvm code have them?
We need real barriers but AFAIK kvm does not have them :(
IOW: virtio is currently broken with kvm, and my patch did not fix this. The comment that I added says as much.How about just using GCC's
__sync__synchronizeatomic builtin (if detected as available by configure)?
It's a full memory barrier instead of just a write barrier, for x86, it generates the same code as the current Linuxmb()implementation:
"mfence" onx86_64
"lock orl $0x0,(%esp)" on x86 unless -march is specified to a processor with "mfence".
PPC could continue to use "eieio" while other architectures could just default to__sync_synchronize
Quoted 5 lines
I do have a newbie question, when exactly would vrtio have to handle concurrent access from multiple threads?
My current reading of the code suggests:
1. whenCONFIG_IOTHREADis true
2. whenCONFIG_KVMis true and the guest machine has multiple CPUs
Right. I don't think CONFIG_IOTHREAD can work correctly without kvm though: how would atomics be handled?
Signature · 1 line
MST
Quoted 8 lines · Michael S. Tsirkin
Quoted 5 lines
I do have a newbie question, when exactly would vrtio have to handle concurrent access from multiple threads?
My current reading of the code suggests:
1. whenCONFIG_IOTHREADis true
2. whenCONFIG_KVMis true and the guest machine has multiple CPUsRight. I don't think
CONFIG_IOTHREADcan work correctly without kvm though: how would atomics be handled?
I naively imagined it to work like this:
When CONFIG_IOTHREAD is true and CONFIG_KVM is false,
all the tcg CPUs run in the tcg_cpu_thread and device emulation code runs in io_thread,
so if the tcg translators generate suitable memory barrier instructions when it sees a "lfence", "sfence", "mfence" instruction while emulating a x86 or "sync" while emulating a MIPS everything should work but a quick look at target-*/translate.c suggests memory barrier instructions are treated as nops.
So maybe --enable-io-thread while --disable-kvm should not be allowed at configure time.
Does anyone actually ship qemu with CONFIG_IOTHREAD enabled?
Quoted 10 lines · Scott Tsai
Quoted 8 lines · Michael S. Tsirkin
Quoted 5 lines
I do have a newbie question, when exactly would vrtio have to handle concurrent access from multiple threads?
My current reading of the code suggests:
1. whenCONFIG_IOTHREADis true
2. whenCONFIG_KVMis true and the guest machine has multiple CPUsRight. I don't think
CONFIG_IOTHREADcan work correctly without kvm though: how would atomics be handled?
Quoted 8 lines
I naively imagined it to work like this:
WhenCONFIG_IOTHREADis true andCONFIG_KVMis false,
all the tcg CPUs run in thetcg_cpu_threadand device emulation code runs inio_thread,
so if the tcg translators generate suitable memory barrier instructions when it sees a "lfence", "sfence", "mfence" instruction while emulating a x86 or "sync" while emulating a MIPS everything should work
that might not be enough. guest can do e.g. atomics on the same memory with iothread. In parctice, with virtio it doesn't.
Quoted 6 lines
but a quick look at
target-*/translate.csuggests memory barrier instructions are treated as nops.So maybe --enable-io-thread while --disable-kvm should not be allowed at configure time.
Does anyone actually ship qemu withCONFIG_IOTHREADenabled?
Quoted 5 lines · Scott Tsai
I do have a newbie question, when exactly would vrtio have to handle concurrent access from multiple threads?
My current reading of the code suggests:
1. whenCONFIG_IOTHREADis true
2. whenCONFIG_KVMis true and the guest machine has multiple CPUs
It's enough to have CONFIG_KVM with a single guest CPU, isn't it?
Because the guest CPU can still run concurrently with the I/O thread on kvm, and the host may have multiple CPUs even though the guest does not.
-- Jamie
Quoted 2 lines · Paul Brook
A "compiler memory barrier" provides absolutely no guarantees in a multithreaded environment.
Claims including words "absolutely no" have absolutely no chance to be correct.
Signature · 1 line
MST