system/memory: Make ram device region directly accessible
v3 cover system/memory: Make ram device region directly accessible
All ram device regions was turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to a hanged guest where a NVidia GH100 GPU is passed from host. The memory in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take DMA bounce buffer in address_space_map() to cover the DMA request. However, the bounce buffer size is 4096 bytes and we're overrunning it easily when the guest has significant disk activities on compiling 'cuda-samples'. The full log and problem description can be found from PATCH[1/2]'s commit log.
Try to fix the issue handled in commit 4a2e242bbb by replacing memcopy()/ memmove() with newly added helpers qemu_ram_{copy, move}() that works on top of __builtin_{memcpy, memmove} or unaligned access friendly memory movement in the accessors to the ram device regions. With this, we can basically revert that commit to make ram device region directly accessible again and bypass the bounce buffer in address_space_map() where the guest hang is caused.
PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors PATCH[2] makes ram device region directly accessible again
Changelog
=========
v2 -> v3:
* https://lore.kernel.org/qemu-arm/[email protected]/
* Documentation for qemu_ram_{copy, move} (Peter/Michael)
* Support qemu_ram_move() for overlapped src/dest (Richard)
* Use {memcpy, memmove} if step is 16-bytes or more (Michael)
* Code improvements (Richard/Michael)
v1 -> v2:
* https://lore.kernel.org/qemu-arm/[email protected]/
* Rename address_space_{memcpy, memmove}() to qemu_ram_{copy, move}()
and move them to physmem.c and memory.h (Philippe)
* Use memcpy() and memmove() in qemu_ram_{copy, move}() for the variable
length case (Miachel)
* Handle unaligned access in qemu_ram_{copy, move}() for all archs
except i386 and x86_64 (Richard/Michael)
RFCv1 -> v1:
* https://lists.nongnu.org/archive/html/qemu-arm/2026-06/msg00307.html
* Reworked solution based on suggestions from Peter Xu, Peter Maydell and Michael S. Tsirkin
Gavin Shan (2):
system/memory: Use qemu_ram_{copy, move}() in ram device region
accessors
system/memory: Make ram device region directly accessible
hw/remote/vfio-user-obj.c | 4 +- include/system/memory.h | 43 ++++++--- system/memory.c | 41 +-------- system/physmem.c | 178 +++++++++++++++++++++++++++++++++++++- system/trace-events | 2 - 5 files changed, 210 insertions(+), 58 deletions(-)
Signature · 1 line
2.54.0
PATCH v3 1/2 system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
All ram device regions were turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.
host$ lspci | grep GH100
0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1)
host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64 \
-machine virt,gic-version=host,ras=on,highmem-mmio-size=4T \
-accel kvm -cpu host -smp cpus=48 -m size=8G \
-drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0 \
-device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4 \
-device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0
:
guest$ cd cuda-samples/build
guest$ make -j 20 clean
guest$ make -j 20
:
[ 54%] Linking CUDA executable graphMemoryNodes
[ 54%] Built target graphMemoryNodes
<no more output afterwards, guest becomes frozen here>guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources
[ 555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)
When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 of the GH100 GPU card can be presented to the guest through memory hot-add. The page cache can then be allocated from the hot added memory blocks when cuda-samples is being built. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is an indirectly accessible ram device region in qemu. However, the max bounce bufer size is only 4096 bytes by default and that is exhausted quickly, leading to a reset on the virtio-blk device and frozen guest eventually.
QEMU
====
virtio_blk_handle_output
virtio_blk_handle_vq
virtio_blk_get_request
virtqueue_pop
virtqueue_split_pop
virtqueue_map_desc
address_space_map
memory_access_is_direct # Return false
memory_region_supports_direct_access
(qemu) info mtree
memory-region: pci_bridge_pci
0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci
0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4
0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4
0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]This adds qemu_ram_{copy, move}() and replaces {memcpy, memmove}() with them in the ram device memory region accessors, similar to what's done in commit 4a2e242bbb so that the issue (MMIO access instructions were optimized to SSE instructions) covered by that commit is fixed. This makes 'ram_device_mem_ops' redundant, paving the way to revert that commit to make ram device region directly accessible again in the next patch.
---
v3: Documentation for qemu_ram_{copy, move} (Peter/Michael)
Support qemu_ram_move() for overlapped src/dest (Richard)
Use {memcpy, memmove} if step is 16-bytes or more (Michael)
Code improvements (Richard/Michael)
--- hw/remote/vfio-user-obj.c | 4 +- include/system/memory.h | 32 ++++++- system/physmem.c | 178 +++++++++++++++++++++++++++++++++++++- 3 files changed, 207 insertions(+), 7 deletions(-)
hw/remote/vfio-user-obj.c +2 -2
@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset, ram_ptr =memory_region_get_ram_ptr (mr);if (is_write) { -memcpy ((ram_ptr + offset), buf, size); +qemu_ram_copy (ram_ptr + offset, buf, size); }else { -memcpy (buf, (ram_ptr + offset), size); +qemu_ram_copy (buf, ram_ptr + offset, size); }return 0 ;
include/system/memory.h +31 -1
@@ -2897,6 +2897,36 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);void address_space_unregister_map_client (AddressSpace *as,QEMUBH *bh);/* Internal functions, part of the implementation of address_space_read. */ + +/** +* qemu_ram_copy: copy data to ramblock +* +* @dst: destination where the data is copied to +* @src: source where the data is copied from +* @n: length of data to be copied +* +* +* Copy @n bytes from @src to @dst with the assumption that @src and @dst +* do not overlap. Handles special cases such as uncacheable ramblocks +* correctly. Use this for accessing ramblock in response to DMA/VCPU IO, +* in preference to memcpy(). +*/ +void qemu_ram_copy (void *dest,const void *src,size_t n); + +/** +* qemu_ram_move: move data to ramblock +* +* @dst: destination where the data is moved to +* @src: source where the data is moved from +* @n: length of data to be moved +* +* Move @n bytes from @src to @dst with the assumption that @src and @dst +* can overlap. Handles special cases such as uncacheable ramblocks +* correctly. Use this for accessing ramblock in response to DMA/VCPU IO, +* in preference to memmove(). +*/ +void qemu_ram_move (void *dest,const void *src,size_t n); +MemTxResult address_space_read_full (AddressSpace *as,hwaddr addr,MemTxAttrs attrs,void *buf,hwaddr len); MemTxResult flatview_read_continue(FlatView *fv, hwaddr addr,
@@ -2970,7 +3000,7 @@ MemTxResult address_space_read(AddressSpace *as, hwaddr addr, mr =flatview_translate (fv, addr, &addr1, &l, false, attrs);if (len == l &&memory_access_is_direct (mr, false, attrs)) { ptr =qemu_map_ram_ptr (mr->ram_block, addr1); -memcpy (buf, ptr, len); +qemu_ram_copy (buf, ptr, len); }else { result =flatview_read_continue (fv, addr, attrs, buf, len, addr1, l, mr);
system/physmem.c +174 -4
202 lines not on this page · raw message
Signature · 1 line
2.54.0
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 185 lines · Gavin Shan
All ram device regions were turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.host$ lspci | grep GH100 0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1) host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64 \ -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T \ -accel kvm -cpu host -smp cpus=48 -m size=8G \ -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0 \ -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4 \ -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0 : guest$ cd cuda-samples/build guest$ make -j 20 clean guest$ make -j 20 : [ 54%] Linking CUDA executable graphMemoryNodes [ 54%] Built target graphMemoryNodes <no more output afterwards, guest becomes frozen here>guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources
[ 555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 of the GH100 GPU card can be presented to the guest through memory hot-add. The page cache can then be allocated from the hot added memory blocks when cuda-samples is being built. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is an indirectly accessible ram device region in qemu. However, the max bounce bufer size is only 4096 bytes by default and that is exhausted quickly, leading to a reset on the virtio-blk device and frozen guest eventually.
QEMU
====
virtio_blk_handle_output
virtio_blk_handle_vq
virtio_blk_get_request
virtqueue_pop
virtqueue_split_pop
virtqueue_map_desc
address_space_map
memory_access_is_direct# Return false
memory_region_supports_direct_access(qemu) info mtree
memory-region:pci_bridge_pci0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci 0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4 0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4 0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]This adds qemu_ram_{copy, move}() and replaces {memcpy, memmove}() with them in the ram device memory region accessors, similar to what's done in commit
4a2e242bbbso that the issue (MMIO access instructions were optimized to SSE instructions) covered by that commit is fixed. This makes 'ram_device_mem_ops' redundant, paving the way to revert that commit to make ram device region directly accessible again in the next patch.Reported-by Julia Graham <[email protected]>Suggested-by Michael S. Tsirkin <[email protected]>Suggested-by Peter Xu <[email protected]>Suggested-by Richard Henderson <[email protected]>Suggested-by Peter Maydell <[email protected]>Signed-off-by Gavin Shan <[email protected]>---
v3: Documentation for qemu_ram_{copy, move} (Peter/Michael)
Supportqemu_ram_move()for overlapped src/dest (Richard)
Use {memcpy, memmove} if step is 16-bytes or more (Michael)
Code improvements (Richard/Michael)--- hw/remote/vfio-user-obj.c | 4 +- include/system/memory.h | 32 ++++++- system/physmem.c | 178 +++++++++++++++++++++++++++++++++++++- 3 files changed, 207 insertions(+), 7 deletions(-)hw/remote/vfio-user-obj.c +2
-2@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset, ram_ptr =memory_region_get_ram_ptr (mr);if (is_write) { -memcpy ((ram_ptr + offset), buf, size); +qemu_ram_copy (ram_ptr + offset, buf, size); }else { -memcpy (buf, (ram_ptr + offset), size); +qemu_ram_copy (buf, ram_ptr + offset, size); }return 0 ;include/system/memory.h +31
-1@@ -2897,6 +2897,36 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);void address_space_unregister_map_client (AddressSpace *as,QEMUBH *bh);/* Internal functions, part of the implementation of address_space_read. */ + +/** +* qemu_ram_copy: copy data to ramblock +* +* @dst: destination where the data is copied to +* @src: source where the data is copied from +* @n: length of data to be copied +* +* +* Copy @n bytes from @src to @dst with the assumption that @src and @dst +* do not overlap. Handles special cases such as uncacheable ramblocks +* correctly. Use this for accessing ramblock in response to DMA/VCPU IO, +* in preference to memcpy(). +*/ +void qemu_ram_copy (void *dest,const void *src,size_t n); + +/** +* qemu_ram_move: move data to ramblock +* +* @dst: destination where the data is moved to +* @src: source where the data is moved from +* @n: length of data to be moved +* +* Move @n bytes from @src to @dst with the assumption that @src and @dst +* can overlap. Handles special cases such as uncacheable ramblocks +* correctly. Use this for accessing ramblock in response to DMA/VCPU IO, +* in preference to memmove(). +*/ +void qemu_ram_move (void *dest,const void *src,size_t n); +MemTxResult address_space_read_full (AddressSpace *as,hwaddr addr,MemTxAttrs attrs,void *buf,hwaddr len); MemTxResult flatview_read_continue(FlatView *fv, hwaddr addr,@@ -2970,7 +3000,7 @@ MemTxResult address_space_read(AddressSpace *as, hwaddr addr, mr =flatview_translate (fv, addr, &addr1, &l, false, attrs);if (len == l &&memory_access_is_direct (mr, false, attrs)) { ptr =qemu_map_ram_ptr (mr->ram_block, addr1); -memcpy (buf, ptr, len); +qemu_ram_copy (buf, ptr, len); }else { result =flatview_read_continue (fv, addr, attrs, buf, len, addr1, l, mr);system/physmem.c +29
-0@@ -3160,6 +3160,177 @@ void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size)invalidate_and_set_dirty (mr, addr, size); } +static void qemu_ram_copy_aligned (void *dst,const void *src,size_t n) +{ +switch (n) { +case 1 : +__builtin_memcpy (dst, src,1 ); +break ; +case 2 : +__builtin_memcpy (dst, src,2 ); +break ; +case 4 : +__builtin_memcpy (dst, src,4 ); +break ; +case 8 : +__builtin_memcpy (dst, src,8 ); +break ; +default : +memcpy (dst, src, n); + } +} + +static void qemu_ram_move_aligned (void *dst,const void *src,size_t n) +{ +switch (n) { +case 1 : +__builtin_memmove (dst, src,1 ); +break ; +case 2 : +__builtin_memmove (dst, src,2 ); +break ;
system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3201
Quoted 10 lines
system/physmem.c +10
-0+case 4 : +__builtin_memmove (dst, src,4 ); +break ; +case 8 : +__builtin_memmove (dst, src,8 ); +break ; +default : +memmove (dst, src, n); + } +}
A weird name for this. You call it for both aligned and unaligned.
maybe this is _unsafe and the "unaligned" below is _safe ?
Quoted 90 lines
system/physmem.c +90
-0+ +static void qemu_ram_copy_unaligned (void *dst,const void *src, +size_t n,size_t max_step) +{ +uintptr_t test, step; + +/* Aligned maximal step */ + max_step =pow2floor (max_step); + +while (n) { + test = (uintptr_t )src | (uintptr_t )dst | n | max_step; + step = test & -test; + +switch (step) { +case 1 : +qatomic_set ((uint8_t *)dst,qatomic_read ((uint8_t *)src)); + src +=1 ; + dst +=1 ; + n -=1 ; +break ; +case 2 : +qatomic_set ((uint16_t *)dst,qatomic_read ((uint16_t *)src)); + src +=2 ; + dst +=2 ; + n -=2 ; +break ; +case 4 : +qatomic_set ((uint32_t *)dst,qatomic_read ((uint32_t *)src)); + src +=4 ; + dst +=4 ; + n -=4 ; +break ; +case 8 : +qatomic_set ((uint64_t *)dst,qatomic_read ((uint64_t *)src)); + src +=8 ; + dst +=8 ; + n -=8 ; +break ; +default : +memcpy (dst, src, step); + src += step; + dst += step; + n -= step; + } + } +} + +static void qemu_ram_backwards_copy_unaligned (void *dst,const void *src, +size_t n,size_t max_step) +{ +uintptr_t test, step; + +/* Aligned maximal step */ + max_step =pow2floor (max_step); + +/* End of the blocks */ + src += n; + dst += n; + +while (n) { + test = (uintptr_t )src | (uintptr_t )dst | n | max_step; + step = test & -test; + +switch (step) { +case 1 : + src -=1 ; + dst -=1 ; + n -=1 ; +qatomic_set ((uint8_t *)dst,qatomic_read ((uint8_t *)src)); +break ; +case 2 : + src -=2 ; + dst -=2 ; + n -=2 ; +qatomic_set ((uint16_t *)dst,qatomic_read ((uint16_t *)src)); +break ; +case 4 : + src -=4 ; + dst -=4 ; + n -=4 ; +qatomic_set ((uint32_t *)dst,qatomic_read ((uint32_t *)src)); +break ; +case 8 : + src -=8 ; + dst -=8 ; + n -=8 ; +qatomic_set ((uint64_t *)dst,qatomic_read ((uint64_t *)src)); +break ; +default : + src -= step;
system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3301
Quoted 10 lines
system/physmem.c +10
-0+ dst -= step; + n -= step; +memmove (dst, src, step); + } + } +} + +/* x86 should work with __builtin_{memcpy, memmove}() for IO access */ +#if defined(__i386__) || defined(__x86_64__) +#define HOST_UNALIGNED_MMIO_OK 1
maybe HOST_UNALIGNED_VECTOR_MMIO_OK ? We also care that vector stores are safe.
Quoted 65 lines
system/physmem.c +35
-4+#else +#define HOST_UNALIGNED_MMIO_OK 0 +#endif + +void qemu_ram_copy (void *dst,const void *src,size_t n) +{ +if (dst == src || n ==0 ) { +return ; + } + +if (HOST_UNALIGNED_MMIO_OK ) { +qemu_ram_copy_aligned (dst, src, n); + }else { +qemu_ram_copy_unaligned (dst, src, n,8 ); + } +} + +void qemu_ram_move (void *dst,const void *src,size_t n) +{ +if (src == dst || n ==0 ) { +return ; + } + +if (HOST_UNALIGNED_MMIO_OK ) { +qemu_ram_move_aligned (dst, src, n); + }else if (dst < src) { +qemu_ram_copy_unaligned (dst, src, n, src - dst); + }else { +qemu_ram_backwards_copy_unaligned (dst, src, n, dst - src); + } +} +int memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr) {unsigned access_size_max = mr->ops->valid.max_access_size;@@ -3272,7 +3443,7 @@ static MemTxResult flatview_write_continue_step(MemTxAttrs attrs,uint8_t *ram_ptr =qemu_ram_ptr_length (mr->ram_block, mr_addr, l, false, true); -memmove (ram_ptr, buf, *l); +qemu_ram_move (ram_ptr, buf, *l);invalidate_and_set_dirty (mr, mr_addr, *l);return MEMTX_OK ;@@ -3365,7 +3536,7 @@ static MemTxResult flatview_read_continue_step(MemTxAttrs attrs, uint8_t *buf,uint8_t *ram_ptr =qemu_ram_ptr_length (mr->ram_block, mr_addr, l, false, false); -memcpy (buf, ram_ptr, *l); +qemu_ram_copy (buf, ram_ptr, *l);return MEMTX_OK ; }@@ -3503,8 +3674,7 @@ MemTxResult address_space_write_rom(AddressSpace *as, hwaddr addr, l =memory_access_size (mr, l, addr1); }else {/* ROM/RAM case */ -void *ram_ptr =qemu_map_ram_ptr (mr->ram_block, addr1); -memcpy (ram_ptr, buf, l); +qemu_ram_copy (qemu_map_ram_ptr (mr->ram_block, addr1), buf, l);invalidate_and_set_dirty (mr, addr1, l); } len -= l;Signature · 1 line
2.54.0
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 199 lines · Michael S. Tsirkin
Quoted 185 lines · Gavin Shan
All ram device regions were turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.host$ lspci | grep GH100 0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1) host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64 \ -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T \ -accel kvm -cpu host -smp cpus=48 -m size=8G \ -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0 \ -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4 \ -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0 : guest$ cd cuda-samples/build guest$ make -j 20 clean guest$ make -j 20 : [ 54%] Linking CUDA executable graphMemoryNodes [ 54%] Built target graphMemoryNodes <no more output afterwards, guest becomes frozen here> guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources [ 555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 of the GH100 GPU card can be presented to the guest through memory hot-add. The page cache can then be allocated from the hot added memory blocks when cuda-samples is being built. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is an indirectly accessible ram device region in qemu. However, the max bounce bufer size is only 4096 bytes by default and that is exhausted quickly, leading to a reset on the virtio-blk device and frozen guest eventually.
QEMU ==== virtio_blk_handle_output virtio_blk_handle_vq virtio_blk_get_request virtqueue_pop virtqueue_split_pop virtqueue_map_desc address_space_map memory_access_is_direct # Return false memory_region_supports_direct_access (qemu) info mtree memory-region: pci_bridge_pci 0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci 0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4 0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4 0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]This adds qemu_ram_{copy, move}() and replaces {memcpy, memmove}() with them in the ram device memory region accessors, similar to what's done in commit
4a2e242bbbso that the issue (MMIO access instructions were optimized to SSE instructions) covered by that commit is fixed. This makes 'ram_device_mem_ops' redundant, paving the way to revert that commit to make ram device region directly accessible again in the next patch.Reported-by Julia Graham <[email protected]>Suggested-by Michael S. Tsirkin <[email protected]>Suggested-by Peter Xu <[email protected]>Suggested-by Richard Henderson <[email protected]>Suggested-by Peter Maydell <[email protected]>Signed-off-by Gavin Shan <[email protected]>---
v3: Documentation for qemu_ram_{copy, move} (Peter/Michael)
Supportqemu_ram_move()for overlapped src/dest (Richard)
Use {memcpy, memmove} if step is 16-bytes or more (Michael)
Code improvements (Richard/Michael)--- hw/remote/vfio-user-obj.c | 4 +- include/system/memory.h | 32 ++++++- system/physmem.c | 178 +++++++++++++++++++++++++++++++++++++- 3 files changed, 207 insertions(+), 7 deletions(-)hw/remote/vfio-user-obj.c +2
-2@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset, ram_ptr =memory_region_get_ram_ptr (mr);if (is_write) { -memcpy ((ram_ptr + offset), buf, size); +qemu_ram_copy (ram_ptr + offset, buf, size); }else { -memcpy (buf, (ram_ptr + offset), size); +qemu_ram_copy (buf, ram_ptr + offset, size); }return 0 ;include/system/memory.h +31
-1@@ -2897,6 +2897,36 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);void address_space_unregister_map_client (AddressSpace *as,QEMUBH *bh);/* Internal functions, part of the implementation of address_space_read. */ + +/** +* qemu_ram_copy: copy data to ramblock +* +* @dst: destination where the data is copied to +* @src: source where the data is copied from +* @n: length of data to be copied +* +* +* Copy @n bytes from @src to @dst with the assumption that @src and @dst +* do not overlap. Handles special cases such as uncacheable ramblocks +* correctly. Use this for accessing ramblock in response to DMA/VCPU IO, +* in preference to memcpy(). +*/ +void qemu_ram_copy (void *dest,const void *src,size_t n); + +/** +* qemu_ram_move: move data to ramblock +* +* @dst: destination where the data is moved to +* @src: source where the data is moved from +* @n: length of data to be moved +* +* Move @n bytes from @src to @dst with the assumption that @src and @dst +* can overlap. Handles special cases such as uncacheable ramblocks +* correctly. Use this for accessing ramblock in response to DMA/VCPU IO, +* in preference to memmove(). +*/ +void qemu_ram_move (void *dest,const void *src,size_t n); +MemTxResult address_space_read_full (AddressSpace *as,hwaddr addr,MemTxAttrs attrs,void *buf,hwaddr len); MemTxResult flatview_read_continue(FlatView *fv, hwaddr addr,@@ -2970,7 +3000,7 @@ MemTxResult address_space_read(AddressSpace *as, hwaddr addr, mr =flatview_translate (fv, addr, &addr1, &l, false, attrs);if (len == l &&memory_access_is_direct (mr, false, attrs)) { ptr =qemu_map_ram_ptr (mr->ram_block, addr1); -memcpy (buf, ptr, len); +qemu_ram_copy (buf, ptr, len); }else { result =flatview_read_continue (fv, addr, attrs, buf, len, addr1, l, mr);system/physmem.c +29
-0@@ -3160,6 +3160,177 @@ void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size)invalidate_and_set_dirty (mr, addr, size); } +static void qemu_ram_copy_aligned (void *dst,const void *src,size_t n) +{ +switch (n) { +case 1 : +__builtin_memcpy (dst, src,1 ); +break ; +case 2 : +__builtin_memcpy (dst, src,2 ); +break ; +case 4 : +__builtin_memcpy (dst, src,4 ); +break ; +case 8 : +__builtin_memcpy (dst, src,8 ); +break ; +default : +memcpy (dst, src, n); + } +} + +static void qemu_ram_move_aligned (void *dst,const void *src,size_t n) +{ +switch (n) { +case 1 : +__builtin_memmove (dst, src,1 ); +break ; +case 2 : +__builtin_memmove (dst, src,2 ); +break ;system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3201
Quoted 10 lines
system/physmem.c +10
-0+case 4 : +__builtin_memmove (dst, src,4 ); +break ; +case 8 : +__builtin_memmove (dst, src,8 ); +break ; +default : +memmove (dst, src, n); + } +}A weird name for this. You call it for both aligned and unaligned.
Quoted 2 lines
maybe this is
_unsafeand the "unaligned" below is_safe?
Yeah, xxx_{aligned, unaligned}() should be renamed to xxx_{unsafe, safe}(). The question is why we're away from __builtin_{memcpy, memmove}() and memcpy/memmove since they're not safe as the new function name indicates? :-)
If we're away from memcpy/memmove() and switch to xxx_safe() for x86, we shouldn't have any more concerns except the performance lost. At least, everything should be working from function POV because we're not affected by glibc and the optimization enforced by the compiler any more.
Quoted 88 lines
Quoted 88 lines
system/physmem.c +85
-0+ +static void qemu_ram_copy_unaligned (void *dst,const void *src, +size_t n,size_t max_step) +{ +uintptr_t test, step; + +/* Aligned maximal step */ + max_step =pow2floor (max_step); + +while (n) { + test = (uintptr_t )src | (uintptr_t )dst | n | max_step; + step = test & -test; + +switch (step) { +case 1 : +qatomic_set ((uint8_t *)dst,qatomic_read ((uint8_t *)src)); + src +=1 ; + dst +=1 ; + n -=1 ; +break ; +case 2 : +qatomic_set ((uint16_t *)dst,qatomic_read ((uint16_t *)src)); + src +=2 ; + dst +=2 ; + n -=2 ; +break ; +case 4 : +qatomic_set ((uint32_t *)dst,qatomic_read ((uint32_t *)src)); + src +=4 ; + dst +=4 ; + n -=4 ; +break ; +case 8 : +qatomic_set ((uint64_t *)dst,qatomic_read ((uint64_t *)src)); + src +=8 ; + dst +=8 ; + n -=8 ; +break ; +default : +memcpy (dst, src, step); + src += step; + dst += step; + n -= step; + } + } +} + +static void qemu_ram_backwards_copy_unaligned (void *dst,const void *src, +size_t n,size_t max_step) +{ +uintptr_t test, step; + +/* Aligned maximal step */ + max_step =pow2floor (max_step); + +/* End of the blocks */ + src += n; + dst += n; + +while (n) { + test = (uintptr_t )src | (uintptr_t )dst | n | max_step; + step = test & -test; + +switch (step) { +case 1 : + src -=1 ; + dst -=1 ; + n -=1 ; +qatomic_set ((uint8_t *)dst,qatomic_read ((uint8_t *)src)); +break ; +case 2 : + src -=2 ; + dst -=2 ; + n -=2 ; +qatomic_set ((uint16_t *)dst,qatomic_read ((uint16_t *)src)); +break ; +case 4 : + src -=4 ; + dst -=4 ; + n -=4 ; +qatomic_set ((uint32_t *)dst,qatomic_read ((uint32_t *)src)); +break ; +case 8 : + src -=8 ; + dst -=8 ;
Quoted 8 lines
system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3294
Quoted 8 lines
system/physmem.c +8
-0+ n -=8 ; +qatomic_set ((uint64_t *)dst,qatomic_read ((uint64_t *)src)); +break ; +default : + src -= step; + dst -= step; + n -= step; +memmove (dst, src, step);
This should be memcpy() actually since src/dst don't overlap. However, I think Richard may suggest to avoid using memcpy/memmove() here.
Quoted 11 lines
system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3301
Quoted 7 lines
system/physmem.c +7
-0+ } + } +} + +/* x86 should work with __builtin_{memcpy, memmove}() for IO access */ +#if defined(__i386__) || defined(__x86_64__) +#define HOST_UNALIGNED_MMIO_OK 1maybe
HOST_UNALIGNED_VECTOR_MMIO_OK? We also care that vector stores are safe.
Yes, it should be renamed to HOST_UNALIGNED_VECTOR_MMIO_OK.
Quoted 51 lines
Quoted 51 lines
system/physmem.c +34
-2+#else +#define HOST_UNALIGNED_MMIO_OK 0 +#endif + +void qemu_ram_copy (void *dst,const void *src,size_t n) +{ +if (dst == src || n ==0 ) { +return ; + } + +if (HOST_UNALIGNED_MMIO_OK ) { +qemu_ram_copy_aligned (dst, src, n); + }else { +qemu_ram_copy_unaligned (dst, src, n,8 ); + } +} + +void qemu_ram_move (void *dst,const void *src,size_t n) +{ +if (src == dst || n ==0 ) { +return ; + } + +if (HOST_UNALIGNED_MMIO_OK ) { +qemu_ram_move_aligned (dst, src, n); + }else if (dst < src) { +qemu_ram_copy_unaligned (dst, src, n, src - dst); + }else { +qemu_ram_backwards_copy_unaligned (dst, src, n, dst - src); + } +} +int memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr) {unsigned access_size_max = mr->ops->valid.max_access_size;@@ -3272,7 +3443,7 @@ static MemTxResult flatview_write_continue_step(MemTxAttrs attrs,uint8_t *ram_ptr =qemu_ram_ptr_length (mr->ram_block, mr_addr, l, false, true); -memmove (ram_ptr, buf, *l); +qemu_ram_move (ram_ptr, buf, *l);invalidate_and_set_dirty (mr, mr_addr, *l);return MEMTX_OK ;@@ -3365,7 +3536,7 @@ static MemTxResult flatview_read_continue_step(MemTxAttrs attrs, uint8_t *buf,uint8_t *ram_ptr =qemu_ram_ptr_length (mr->ram_block, mr_addr, l, false, false); -memcpy (buf, ram_ptr, *l); +qemu_ram_copy (buf, ram_ptr, *l);
Quoted 15 lines
Quoted 14 lines
system/physmem.c +1
-2return MEMTX_OK ; }@@ -3503,8 +3674,7 @@ MemTxResult address_space_write_rom(AddressSpace *as, hwaddr addr, l =memory_access_size (mr, l, addr1); }else {/* ROM/RAM case */ -void *ram_ptr =qemu_map_ram_ptr (mr->ram_block, addr1); -memcpy (ram_ptr, buf, l); +qemu_ram_copy (qemu_map_ram_ptr (mr->ram_block, addr1), buf, l);invalidate_and_set_dirty (mr, addr1, l); } len -= l;Signature · 1 line
2.54.0
Thanks,
Gavin
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 197 lines · Gavin Shan
Quoted 195 lines · Michael S. Tsirkin
Quoted 181 lines · Gavin Shan
All ram device regions were turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.host$ lspci | grep GH100 0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1) host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64 \ -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T \ -accel kvm -cpu host -smp cpus=48 -m size=8G \ -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0 \ -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4 \ -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0 : guest$ cd cuda-samples/build guest$ make -j 20 clean guest$ make -j 20 : [ 54%] Linking CUDA executable graphMemoryNodes [ 54%] Built target graphMemoryNodes <no more output afterwards, guest becomes frozen here> guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources [ 555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 of the GH100 GPU card can be presented to the guest through memory hot-add. The page cache can then be allocated from the hot added memory blocks when cuda-samples is being built. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is an indirectly accessible ram device region in qemu. However, the max bounce bufer size is only 4096 bytes by default and that is exhausted quickly, leading to a reset on the virtio-blk device and frozen guest eventually.
QEMU ==== virtio_blk_handle_output virtio_blk_handle_vq virtio_blk_get_request virtqueue_pop virtqueue_split_pop virtqueue_map_desc address_space_map memory_access_is_direct # Return false memory_region_supports_direct_access (qemu) info mtree memory-region: pci_bridge_pci 0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci 0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4 0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4 0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]This adds qemu_ram_{copy, move}() and replaces {memcpy, memmove}() with them in the ram device memory region accessors, similar to what's done in commit
4a2e242bbbso that the issue (MMIO access instructions were optimized to SSE instructions) covered by that commit is fixed. This makes 'ram_device_mem_ops' redundant, paving the way to revert that commit to make ram device region directly accessible again in the next patch.Reported-by Julia Graham <[email protected]>Suggested-by Michael S. Tsirkin <[email protected]>Suggested-by Peter Xu <[email protected]>Suggested-by Richard Henderson <[email protected]>Suggested-by Peter Maydell <[email protected]>Signed-off-by Gavin Shan <[email protected]>---
v3: Documentation for qemu_ram_{copy, move} (Peter/Michael)
Supportqemu_ram_move()for overlapped src/dest (Richard)
Use {memcpy, memmove} if step is 16-bytes or more (Michael)
Code improvements (Richard/Michael)--- hw/remote/vfio-user-obj.c | 4 +- include/system/memory.h | 32 ++++++- system/physmem.c | 178 +++++++++++++++++++++++++++++++++++++- 3 files changed, 207 insertions(+), 7 deletions(-)hw/remote/vfio-user-obj.c +2
-2@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset, ram_ptr =memory_region_get_ram_ptr (mr);if (is_write) { -memcpy ((ram_ptr + offset), buf, size); +qemu_ram_copy (ram_ptr + offset, buf, size); }else { -memcpy (buf, (ram_ptr + offset), size); +qemu_ram_copy (buf, ram_ptr + offset, size); }return 0 ;include/system/memory.h +31
-1@@ -2897,6 +2897,36 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);void address_space_unregister_map_client (AddressSpace *as,QEMUBH *bh);/* Internal functions, part of the implementation of address_space_read. */ + +/** +* qemu_ram_copy: copy data to ramblock +* +* @dst: destination where the data is copied to +* @src: source where the data is copied from +* @n: length of data to be copied +* +* +* Copy @n bytes from @src to @dst with the assumption that @src and @dst +* do not overlap. Handles special cases such as uncacheable ramblocks +* correctly. Use this for accessing ramblock in response to DMA/VCPU IO, +* in preference to memcpy(). +*/ +void qemu_ram_copy (void *dest,const void *src,size_t n); + +/** +* qemu_ram_move: move data to ramblock +* +* @dst: destination where the data is moved to +* @src: source where the data is moved from +* @n: length of data to be moved +* +* Move @n bytes from @src to @dst with the assumption that @src and @dst +* can overlap. Handles special cases such as uncacheable ramblocks +* correctly. Use this for accessing ramblock in response to DMA/VCPU IO, +* in preference to memmove(). +*/ +void qemu_ram_move (void *dest,const void *src,size_t n); +MemTxResult address_space_read_full (AddressSpace *as,hwaddr addr,MemTxAttrs attrs,void *buf,hwaddr len); MemTxResult flatview_read_continue(FlatView *fv, hwaddr addr,@@ -2970,7 +3000,7 @@ MemTxResult address_space_read(AddressSpace *as, hwaddr addr, mr =flatview_translate (fv, addr, &addr1, &l, false, attrs);if (len == l &&memory_access_is_direct (mr, false, attrs)) { ptr =qemu_map_ram_ptr (mr->ram_block, addr1); -memcpy (buf, ptr, len); +qemu_ram_copy (buf, ptr, len); }else { result =flatview_read_continue (fv, addr, attrs, buf, len, addr1, l, mr);system/physmem.c +29
-0@@ -3160,6 +3160,177 @@ void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size)invalidate_and_set_dirty (mr, addr, size); } +static void qemu_ram_copy_aligned (void *dst,const void *src,size_t n) +{ +switch (n) { +case 1 : +__builtin_memcpy (dst, src,1 ); +break ; +case 2 : +__builtin_memcpy (dst, src,2 ); +break ; +case 4 : +__builtin_memcpy (dst, src,4 ); +break ; +case 8 : +__builtin_memcpy (dst, src,8 ); +break ; +default : +memcpy (dst, src, n); + } +} + +static void qemu_ram_move_aligned (void *dst,const void *src,size_t n) +{ +switch (n) { +case 1 : +__builtin_memmove (dst, src,1 ); +break ; +case 2 : +__builtin_memmove (dst, src,2 ); +break ;system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3200
Quoted 10 lines
system/physmem.c +10
-0+case 4 : +__builtin_memmove (dst, src,4 ); +break ; +case 8 : +__builtin_memmove (dst, src,8 ); +break ; +default : +memmove (dst, src, n); + } +}A weird name for this. You call it for both aligned and unaligned.
Quoted 6 lines
Quoted 2 lines
maybe this is
_unsafeand the "unaligned" below is_safe?Yeah, xxx_{aligned, unaligned}() should be renamed to xxx_{unsafe, safe}(). The question is why we're away from __builtin_{memcpy, memmove}() and memcpy/memmove since they're not safe as the new function name indicates? :-)
What is the question, exactly? That's why I made the list of 11 issues. Is that unclear, somehow? Another reason to include that, maybe.
Quoted 4 lines
If we're away from memcpy/memmove() and switch to
xxx_safe()for x86, we shouldn't have any more concerns except the performance lost. At least, everything should be working from function POV because we're not affected by glibc and the optimization enforced by the compiler any more.
There's no way to write C and not depend on compiler's whims for weird things such as unaligned memory accesses.
Quoted 179 lines
Quoted 88 lines
Quoted 88 lines
system/physmem.c +85
-0+ +static void qemu_ram_copy_unaligned (void *dst,const void *src, +size_t n,size_t max_step) +{ +uintptr_t test, step; + +/* Aligned maximal step */ + max_step =pow2floor (max_step); + +while (n) { + test = (uintptr_t )src | (uintptr_t )dst | n | max_step; + step = test & -test; + +switch (step) { +case 1 : +qatomic_set ((uint8_t *)dst,qatomic_read ((uint8_t *)src)); + src +=1 ; + dst +=1 ; + n -=1 ; +break ; +case 2 : +qatomic_set ((uint16_t *)dst,qatomic_read ((uint16_t *)src)); + src +=2 ; + dst +=2 ; + n -=2 ; +break ; +case 4 : +qatomic_set ((uint32_t *)dst,qatomic_read ((uint32_t *)src)); + src +=4 ; + dst +=4 ; + n -=4 ; +break ; +case 8 : +qatomic_set ((uint64_t *)dst,qatomic_read ((uint64_t *)src)); + src +=8 ; + dst +=8 ; + n -=8 ; +break ; +default : +memcpy (dst, src, step); + src += step; + dst += step; + n -= step; + } + } +} + +static void qemu_ram_backwards_copy_unaligned (void *dst,const void *src, +size_t n,size_t max_step) +{ +uintptr_t test, step; + +/* Aligned maximal step */ + max_step =pow2floor (max_step); + +/* End of the blocks */ + src += n; + dst += n; + +while (n) { + test = (uintptr_t )src | (uintptr_t )dst | n | max_step; + step = test & -test; + +switch (step) { +case 1 : + src -=1 ; + dst -=1 ; + n -=1 ; +qatomic_set ((uint8_t *)dst,qatomic_read ((uint8_t *)src)); +break ; +case 2 : + src -=2 ; + dst -=2 ; + n -=2 ; +qatomic_set ((uint16_t *)dst,qatomic_read ((uint16_t *)src)); +break ; +case 4 : + src -=4 ; + dst -=4 ; + n -=4 ; +qatomic_set ((uint32_t *)dst,qatomic_read ((uint32_t *)src)); +break ; +case 8 : + src -=8 ; + dst -=8 ;Quoted 8 lines
system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3294
Quoted 8 lines
system/physmem.c +8
-0+ n -=8 ; +qatomic_set ((uint64_t *)dst,qatomic_read ((uint64_t *)src)); +break ; +default : + src -= step; + dst -= step; + n -= step; +memmove (dst, src, step);This should be
memcpy()actually since src/dst don't overlap. However, I think Richard may suggest to avoid using memcpy/memmove() here.Quoted 11 lines
system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3301
Quoted 7 lines
system/physmem.c +7
-0+ } + } +} + +/* x86 should work with __builtin_{memcpy, memmove}() for IO access */ +#if defined(__i386__) || defined(__x86_64__) +#define HOST_UNALIGNED_MMIO_OK 1maybe
HOST_UNALIGNED_VECTOR_MMIO_OK? We also care that vector stores are safe.Yes, it should be renamed to
HOST_UNALIGNED_VECTOR_MMIO_OK.Quoted 53 lines
Quoted 53 lines
system/physmem.c +34
-2+#else +#define HOST_UNALIGNED_MMIO_OK 0 +#endif + +void qemu_ram_copy (void *dst,const void *src,size_t n) +{ +if (dst == src || n ==0 ) { +return ; + } + +if (HOST_UNALIGNED_MMIO_OK ) { +qemu_ram_copy_aligned (dst, src, n); + }else { +qemu_ram_copy_unaligned (dst, src, n,8 ); + } +} + +void qemu_ram_move (void *dst,const void *src,size_t n) +{ +if (src == dst || n ==0 ) { +return ; + } + +if (HOST_UNALIGNED_MMIO_OK ) { +qemu_ram_move_aligned (dst, src, n); + }else if (dst < src) { +qemu_ram_copy_unaligned (dst, src, n, src - dst); + }else { +qemu_ram_backwards_copy_unaligned (dst, src, n, dst - src); + } +} +int memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr) {unsigned access_size_max = mr->ops->valid.max_access_size;@@ -3272,7 +3443,7 @@ static MemTxResult flatview_write_continue_step(MemTxAttrs attrs,uint8_t *ram_ptr =qemu_ram_ptr_length (mr->ram_block, mr_addr, l, false, true); -memmove (ram_ptr, buf, *l); +qemu_ram_move (ram_ptr, buf, *l);invalidate_and_set_dirty (mr, mr_addr, *l);return MEMTX_OK ;@@ -3365,7 +3536,7 @@ static MemTxResult flatview_read_continue_step(MemTxAttrs attrs, uint8_t *buf,uint8_t *ram_ptr =qemu_ram_ptr_length (mr->ram_block, mr_addr, l, false, false); -memcpy (buf, ram_ptr, *l); +qemu_ram_copy (buf, ram_ptr, *l);return MEMTX_OK ; }@@ -3503,8 +3674,7 @@ MemTxResult address_space_write_rom(AddressSpace *as, hwaddr addr, l =memory_access_size (mr, l, addr1); }else {/* ROM/RAM case */ Quoted 9 lines
Quoted 8 lines
system/physmem.c +1
-2-void *ram_ptr =qemu_map_ram_ptr (mr->ram_block, addr1); -memcpy (ram_ptr, buf, l); +qemu_ram_copy (qemu_map_ram_ptr (mr->ram_block, addr1), buf, l);invalidate_and_set_dirty (mr, addr1, l); } len -= l;Signature · 1 line
2.54.0Thanks,
Gavin
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 201 lines · Michael S. Tsirkin
Quoted 200 lines · Gavin Shan
Quoted 197 lines · Michael S. Tsirkin
Quoted 181 lines · Gavin Shan
All ram device regions were turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.host$ lspci | grep GH100 0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1) host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64 \ -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T \ -accel kvm -cpu host -smp cpus=48 -m size=8G \ -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0 \ -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4 \ -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0 : guest$ cd cuda-samples/build guest$ make -j 20 clean guest$ make -j 20 : [ 54%] Linking CUDA executable graphMemoryNodes [ 54%] Built target graphMemoryNodes <no more output afterwards, guest becomes frozen here> guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources [ 555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 of the GH100 GPU card can be presented to the guest through memory hot-add. The page cache can then be allocated from the hot added memory blocks when cuda-samples is being built. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is an indirectly accessible ram device region in qemu. However, the max bounce bufer size is only 4096 bytes by default and that is exhausted quickly, leading to a reset on the virtio-blk device and frozen guest eventually.
QEMU ==== virtio_blk_handle_output virtio_blk_handle_vq virtio_blk_get_request virtqueue_pop virtqueue_split_pop virtqueue_map_desc address_space_map memory_access_is_direct # Return false memory_region_supports_direct_access (qemu) info mtree memory-region: pci_bridge_pci 0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci 0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4 0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4 0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]This adds qemu_ram_{copy, move}() and replaces {memcpy, memmove}() with them in the ram device memory region accessors, similar to what's done in commit
4a2e242bbbso that the issue (MMIO access instructions were optimized to SSE instructions) covered by that commit is fixed. This makes 'ram_device_mem_ops' redundant, paving the way to revert that commit to make ram device region directly accessible again in the next patch.Reported-by Julia Graham <[email protected]>Suggested-by Michael S. Tsirkin <[email protected]>Suggested-by Peter Xu <[email protected]>Suggested-by Richard Henderson <[email protected]>Suggested-by Peter Maydell <[email protected]>Signed-off-by Gavin Shan <[email protected]>---
v3: Documentation for qemu_ram_{copy, move} (Peter/Michael)
Supportqemu_ram_move()for overlapped src/dest (Richard)
Use {memcpy, memmove} if step is 16-bytes or more (Michael)
Code improvements (Richard/Michael)--- hw/remote/vfio-user-obj.c | 4 +- include/system/memory.h | 32 ++++++- system/physmem.c | 178 +++++++++++++++++++++++++++++++++++++- 3 files changed, 207 insertions(+), 7 deletions(-)hw/remote/vfio-user-obj.c +2
-2@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset, ram_ptr =memory_region_get_ram_ptr (mr);if (is_write) { -memcpy ((ram_ptr + offset), buf, size); +qemu_ram_copy (ram_ptr + offset, buf, size); }else { -memcpy (buf, (ram_ptr + offset), size); +qemu_ram_copy (buf, ram_ptr + offset, size); }return 0 ;include/system/memory.h +31
-1@@ -2897,6 +2897,36 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);void address_space_unregister_map_client (AddressSpace *as,QEMUBH *bh);/* Internal functions, part of the implementation of address_space_read. */ + +/** +* qemu_ram_copy: copy data to ramblock +* +* @dst: destination where the data is copied to +* @src: source where the data is copied from +* @n: length of data to be copied +* +* +* Copy @n bytes from @src to @dst with the assumption that @src and @dst +* do not overlap. Handles special cases such as uncacheable ramblocks +* correctly. Use this for accessing ramblock in response to DMA/VCPU IO, +* in preference to memcpy(). +*/ +void qemu_ram_copy (void *dest,const void *src,size_t n); + +/** +* qemu_ram_move: move data to ramblock +* +* @dst: destination where the data is moved to +* @src: source where the data is moved from +* @n: length of data to be moved +* +* Move @n bytes from @src to @dst with the assumption that @src and @dst +* can overlap. Handles special cases such as uncacheable ramblocks +* correctly. Use this for accessing ramblock in response to DMA/VCPU IO, +* in preference to memmove(). +*/ +void qemu_ram_move (void *dest,const void *src,size_t n); +MemTxResult address_space_read_full (AddressSpace *as,hwaddr addr,MemTxAttrs attrs,void *buf,hwaddr len); MemTxResult flatview_read_continue(FlatView *fv, hwaddr addr,@@ -2970,7 +3000,7 @@ MemTxResult address_space_read(AddressSpace *as, hwaddr addr, mr =flatview_translate (fv, addr, &addr1, &l, false, attrs);if (len == l &&memory_access_is_direct (mr, false, attrs)) { ptr =qemu_map_ram_ptr (mr->ram_block, addr1); -memcpy (buf, ptr, len); +qemu_ram_copy (buf, ptr, len); }else { result =flatview_read_continue (fv, addr, attrs, buf, len, addr1, l, mr);system/physmem.c +29
-0@@ -3160,6 +3160,177 @@ void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size)invalidate_and_set_dirty (mr, addr, size); } +static void qemu_ram_copy_aligned (void *dst,const void *src,size_t n) +{ +switch (n) { +case 1 : +__builtin_memcpy (dst, src,1 ); +break ; +case 2 : +__builtin_memcpy (dst, src,2 ); +break ; +case 4 : +__builtin_memcpy (dst, src,4 ); +break ; +case 8 : +__builtin_memcpy (dst, src,8 ); +break ; +default : +memcpy (dst, src, n); + } +} + +static void qemu_ram_move_aligned (void *dst,const void *src,size_t n) +{ +switch (n) { +case 1 : +__builtin_memmove (dst, src,1 ); +break ; +case 2 : +__builtin_memmove (dst, src,2 ); +break ;system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3200
Quoted 10 lines
system/physmem.c +10
-0+case 4 : +__builtin_memmove (dst, src,4 ); +break ; +case 8 : +__builtin_memmove (dst, src,8 ); +break ; +default : +memmove (dst, src, n); + } +}A weird name for this. You call it for both aligned and unaligned.
maybe this is
_unsafeand the "unaligned" below is_safe?
Quoted 7 lines
Quoted 3 lines
Yeah, xxx_{aligned, unaligned}() should be renamed to xxx_{unsafe, safe}(). The question is why we're away from __builtin_{memcpy, memmove}() and memcpy/memmove since they're not safe as the new function name indicates? :-)
What is the question, exactly? That's why I made the list of 11 issues. Is that unclear, somehow? Another reason to include that, maybe.
We can take the e1000 issue on aarch64 for example:
The software allocate ring buffer for hardware to receive packet.
In normal case, the hardware fill the RX ring descriptor, and with status DD bit=1 (Descriptor Done), then the ownership of the ring descriptor is converted to software, after the software consume the descriptor, it will write status DD bit=0 and give it back to hardware to refill data.
pci_dma_write calls memcpy at the underlying level, and on aarch64 with glibc 2.24+, the memcpy/memmove implementation uses a branchless sequence that copies the same byte three times when count == 1.
So the issue case:
qemu/e1000 | guest/dpdk
-----------------------------------------------------memcpy(dst, &desc.status, 1)
1st write dst DD=1
poll and see DD==1 consume the descriptor write status DD=0 back
2nd write dst DD=1
3rd write dst DD=1
That left the dirty state, and RX will hang.
Quoted 190 lines
Quoted 4 lines
If we're away from memcpy/memmove() and switch to
xxx_safe()for x86, we shouldn't have any more concerns except the performance lost. At least, everything should be working from function POV because we're not affected by glibc and the optimization enforced by the compiler any more.There's no way to write C and not depend on compiler's whims for weird things such as unaligned memory accesses.
Quoted 179 lines
Quoted 88 lines
Quoted 88 lines
system/physmem.c +85
-0+ +static void qemu_ram_copy_unaligned (void *dst,const void *src, +size_t n,size_t max_step) +{ +uintptr_t test, step; + +/* Aligned maximal step */ + max_step =pow2floor (max_step); + +while (n) { + test = (uintptr_t )src | (uintptr_t )dst | n | max_step; + step = test & -test; + +switch (step) { +case 1 : +qatomic_set ((uint8_t *)dst,qatomic_read ((uint8_t *)src)); + src +=1 ; + dst +=1 ; + n -=1 ; +break ; +case 2 : +qatomic_set ((uint16_t *)dst,qatomic_read ((uint16_t *)src)); + src +=2 ; + dst +=2 ; + n -=2 ; +break ; +case 4 : +qatomic_set ((uint32_t *)dst,qatomic_read ((uint32_t *)src)); + src +=4 ; + dst +=4 ; + n -=4 ; +break ; +case 8 : +qatomic_set ((uint64_t *)dst,qatomic_read ((uint64_t *)src)); + src +=8 ; + dst +=8 ; + n -=8 ; +break ; +default : +memcpy (dst, src, step); + src += step; + dst += step; + n -= step; + } + } +} + +static void qemu_ram_backwards_copy_unaligned (void *dst,const void *src, +size_t n,size_t max_step) +{ +uintptr_t test, step; + +/* Aligned maximal step */ + max_step =pow2floor (max_step); + +/* End of the blocks */ + src += n; + dst += n; + +while (n) { + test = (uintptr_t )src | (uintptr_t )dst | n | max_step; + step = test & -test; + +switch (step) { +case 1 : + src -=1 ; + dst -=1 ; + n -=1 ; +qatomic_set ((uint8_t *)dst,qatomic_read ((uint8_t *)src)); +break ; +case 2 : + src -=2 ; + dst -=2 ; + n -=2 ; +qatomic_set ((uint16_t *)dst,qatomic_read ((uint16_t *)src)); +break ; +case 4 : + src -=4 ; + dst -=4 ; + n -=4 ; +qatomic_set ((uint32_t *)dst,qatomic_read ((uint32_t *)src)); +break ; +case 8 : + src -=8 ; + dst -=8 ;Quoted 8 lines
system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3294
Quoted 8 lines
system/physmem.c +8
-0+ n -=8 ; +qatomic_set ((uint64_t *)dst,qatomic_read ((uint64_t *)src)); +break ; +default : + src -= step; + dst -= step; + n -= step; +memmove (dst, src, step);This should be
memcpy()actually since src/dst don't overlap. However, I think Richard may suggest to avoid using memcpy/memmove() here.Quoted 11 lines
system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3301
Quoted 7 lines
system/physmem.c +7
-0+ } + } +} + +/* x86 should work with __builtin_{memcpy, memmove}() for IO access */ +#if defined(__i386__) || defined(__x86_64__) +#define HOST_UNALIGNED_MMIO_OK 1maybe
HOST_UNALIGNED_VECTOR_MMIO_OK? We also care that vector stores are safe.Yes, it should be renamed to
HOST_UNALIGNED_VECTOR_MMIO_OK.Quoted 53 lines
Quoted 53 lines
system/physmem.c +34
-2+#else +#define HOST_UNALIGNED_MMIO_OK 0 +#endif + +void qemu_ram_copy (void *dst,const void *src,size_t n) +{ +if (dst == src || n ==0 ) { +return ; + } + +if (HOST_UNALIGNED_MMIO_OK ) { +qemu_ram_copy_aligned (dst, src, n); + }else { +qemu_ram_copy_unaligned (dst, src, n,8 ); + } +} + +void qemu_ram_move (void *dst,const void *src,size_t n) +{ +if (src == dst || n ==0 ) { +return ; + } + +if (HOST_UNALIGNED_MMIO_OK ) { +qemu_ram_move_aligned (dst, src, n); + }else if (dst < src) { +qemu_ram_copy_unaligned (dst, src, n, src - dst); + }else { +qemu_ram_backwards_copy_unaligned (dst, src, n, dst - src); + } +} +int memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr) {unsigned access_size_max = mr->ops->valid.max_access_size;@@ -3272,7 +3443,7 @@ static MemTxResult flatview_write_continue_step(MemTxAttrs attrs,uint8_t *ram_ptr =qemu_ram_ptr_length (mr->ram_block, mr_addr, l, false, true); -memmove (ram_ptr, buf, *l); +qemu_ram_move (ram_ptr, buf, *l);invalidate_and_set_dirty (mr, mr_addr, *l);return MEMTX_OK ;@@ -3365,7 +3536,7 @@ static MemTxResult flatview_read_continue_step(MemTxAttrs attrs, uint8_t *buf,uint8_t *ram_ptr =qemu_ram_ptr_length (mr->ram_block, mr_addr, l, false, false); -memcpy (buf, ram_ptr, *l); +qemu_ram_copy (buf, ram_ptr, *l);return MEMTX_OK ; }@@ -3503,8 +3674,7 @@ MemTxResult address_space_write_rom(AddressSpace *as, hwaddr addr, l =memory_access_size (mr, l, addr1); }else {/* ROM/RAM case */ Quoted 9 lines
Quoted 8 lines
system/physmem.c +1
-2-void *ram_ptr =qemu_map_ram_ptr (mr->ram_block, addr1); -memcpy (ram_ptr, buf, l); +qemu_ram_copy (qemu_map_ram_ptr (mr->ram_block, addr1), buf, l);invalidate_and_set_dirty (mr, addr1, l); } len -= l;Signature · 1 line
2.54.0Thanks,
Gavin
Signature · 2 lines
Thanks, - Ding Hui
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 8 lines · Ding Hui
Quoted 3 lines
What is the question, exactly? That's why I made the list of 11 issues. Is that unclear, somehow? Another reason to include that, maybe.
We can take the e1000 issue on aarch64 for example:
Quoted 8 lines
The software allocate ring buffer for hardware to receive packet.
In normal case, the hardware fill the RX ring descriptor, and with status DD bit=1 (Descriptor Done), then the ownership of the ring descriptor is converted to software, after the software consume the descriptor, it will write status DD bit=0 and give it back to hardware to refill data.
pci_dma_writecalls memcpy at the underlying level, and on aarch64 with glibc 2.24+, the memcpy/memmove implementation uses a branchless sequence that copies the same byte three times when count == 1.
Yes this is issue 11:
10. on non-x86 memcpy will do multiple overlapping stores even for single byte writes. E.g. it does it to avoid extra branches. This is causing issues in practice.
Signature · 1 line
MST
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 18 lines · Michael S. Tsirkin
Quoted 8 lines · Ding Hui
Quoted 3 lines
What is the question, exactly? That's why I made the list of 11 issues. Is that unclear, somehow? Another reason to include that, maybe.
We can take the e1000 issue on aarch64 for example:
Quoted 8 lines
The software allocate ring buffer for hardware to receive packet.
In normal case, the hardware fill the RX ring descriptor, and with status DD bit=1 (Descriptor Done), then the ownership of the ring descriptor is converted to software, after the software consume the descriptor, it will write status DD bit=0 and give it back to hardware to refill data.
pci_dma_writecalls memcpy at the underlying level, and on aarch64 with glibc 2.24+, the memcpy/memmove implementation uses a branchless sequence that copies the same byte three times when count == 1.
Quoted 6 lines
Yes this is issue 11:
10. on non-x86 memcpy will do multiple overlapping stores even for single byte writes. E.g. it does it to avoid extra branches. This is causing issues in practice.
This issue should be fixed by this series. With this series applied, the memcpy/memmove are replaced with 'qatomic_set((uint8_t *)dst, qatomic_read((uint8_t *)src))' in qemu_ram_{copy, backwards_copy}_unaligned(). I hope Ding can give this series a try to confirm.
After this series is applied:
pci_dma_write
pci_dma_rw
dma_memory_rw
dma_memory_rw_relaxed
address_space_rw
address_space_write
flatview_write
flatview_write_continue
flatview_write_continue_step
memory_access_is_direct // return true
qemu_ram_move // replaced original memmove()
qemu_ram_{copy, backwards_copy}_unaligned
qatomic_set((uint8_t *)dst, qatomic_read((uint8_t *)src));For this specific case, what's done in this series is similar to the proposed in that thread [1]. In the proposal, pci_dma_write() is replaced with address_space_stb(), and eventually bail into "*(uint8_t *)ptr = v;". However, address_space_stb() seems not friendly to DMA write from from syntax level because the limited size is supported by it.
[1] https://lore.kernel.org/qemu-devel/[email protected]/
address_space_stb
address_space_stm_internal
stm_p
stb_p
*(uint8_t *)ptr = v;Thanks,
Gavin
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
In reply to Gavin ShanQuoted 111 lines · Gavin Shan
All ram device regions were turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.host$ lspci | grep GH100 0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1) host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64 \ -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T \ -accel kvm -cpu host -smp cpus=48 -m size=8G \ -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0 \ -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4 \ -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0 : guest$ cd cuda-samples/build guest$ make -j 20 clean guest$ make -j 20 : [ 54%] Linking CUDA executable graphMemoryNodes [ 54%] Built target graphMemoryNodes <no more output afterwards, guest becomes frozen here>guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources
[ 555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 of the GH100 GPU card can be presented to the guest through memory hot-add. The page cache can then be allocated from the hot added memory blocks when cuda-samples is being built. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is an indirectly accessible ram device region in qemu. However, the max bounce bufer size is only 4096 bytes by default and that is exhausted quickly, leading to a reset on the virtio-blk device and frozen guest eventually.
QEMU
====
virtio_blk_handle_output
virtio_blk_handle_vq
virtio_blk_get_request
virtqueue_pop
virtqueue_split_pop
virtqueue_map_desc
address_space_map
memory_access_is_direct# Return false
memory_region_supports_direct_access(qemu) info mtree
memory-region:pci_bridge_pci0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci 0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4 0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4 0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]This adds qemu_ram_{copy, move}() and replaces {memcpy, memmove}() with them in the ram device memory region accessors, similar to what's done in commit
4a2e242bbbso that the issue (MMIO access instructions were optimized to SSE instructions) covered by that commit is fixed. This makes 'ram_device_mem_ops' redundant, paving the way to revert that commit to make ram device region directly accessible again in the next patch.Reported-by Julia Graham <[email protected]>Suggested-by Michael S. Tsirkin <[email protected]>Suggested-by Peter Xu <[email protected]>Suggested-by Richard Henderson <[email protected]>Suggested-by Peter Maydell <[email protected]>Signed-off-by Gavin Shan <[email protected]>---
v3: Documentation for qemu_ram_{copy, move} (Peter/Michael)
Supportqemu_ram_move()for overlapped src/dest (Richard)
Use {memcpy, memmove} if step is 16-bytes or more (Michael)
Code improvements (Richard/Michael)--- hw/remote/vfio-user-obj.c | 4 +- include/system/memory.h | 32 ++++++- system/physmem.c | 178 +++++++++++++++++++++++++++++++++++++- 3 files changed, 207 insertions(+), 7 deletions(-)hw/remote/vfio-user-obj.c +2
-2@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset, ram_ptr =memory_region_get_ram_ptr (mr);if (is_write) { -memcpy ((ram_ptr + offset), buf, size); +qemu_ram_copy (ram_ptr + offset, buf, size); }else { -memcpy (buf, (ram_ptr + offset), size); +qemu_ram_copy (buf, ram_ptr + offset, size); }return 0 ;include/system/memory.h +4
-0@@ -2897,6 +2897,36 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);void address_space_unregister_map_client (AddressSpace *as,QEMUBH *bh);/* Internal functions, part of the implementation of address_space_read. */ + +/** +* qemu_ram_copy: copy data to ramblock +*
include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2913
Quoted 10 lines
include/system/memory.h +10
-0+* @dst: destination where the data is copied to +* @src: source where the data is copied from +* @n: length of data to be copied +* +* +* Copy @n bytes from @src to @dst with the assumption that @src and @dst +* do not overlap. Handles special cases such as uncacheable ramblocks +* correctly. Use this for accessing ramblock in response to DMA/VCPU IO, +* in preference to memcpy(). +*/
The documentation for these functions needs to say what semantics the function is providing (e.g. can I rely on it to do a 4 byte aligned load as a single 4 byte read?). This does not.
include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2919
Quoted 6 lines
include/system/memory.h +6
-0+/* x86 should work with __builtin_{memcpy, memmove}() for IO access */ +#if defined(__i386__) || defined(__x86_64__) +#define HOST_UNALIGNED_MMIO_OK 1 +#else +#define HOST_UNALIGNED_MMIO_OK 0 +#endif
This is still wrong. We should not have "x86 magically works and all other hosts do something different" ifdefs. Define what semantics you need and then we can figure out how to implement them.
My current thought is that we need to handle accesses of 1, 2, 4 and 8 bytes that are naturally aligned by ensuring that we do exactly one host load/store of that type, and that anything else is "the guest isn't relying on specific semantics here, we can just assume it's plain old RAM and do whatever". That would not require any architecture specific ifdefs.
thanks
-- PMM
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 146 lines · Peter Maydell
Quoted 111 lines · Gavin Shan
All ram device regions were turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.host$ lspci | grep GH100 0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1) host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64 \ -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T \ -accel kvm -cpu host -smp cpus=48 -m size=8G \ -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0 \ -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4 \ -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0 : guest$ cd cuda-samples/build guest$ make -j 20 clean guest$ make -j 20 : [ 54%] Linking CUDA executable graphMemoryNodes [ 54%] Built target graphMemoryNodes <no more output afterwards, guest becomes frozen here>guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources
[ 555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 of the GH100 GPU card can be presented to the guest through memory hot-add. The page cache can then be allocated from the hot added memory blocks when cuda-samples is being built. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is an indirectly accessible ram device region in qemu. However, the max bounce bufer size is only 4096 bytes by default and that is exhausted quickly, leading to a reset on the virtio-blk device and frozen guest eventually.
QEMU
====
virtio_blk_handle_output
virtio_blk_handle_vq
virtio_blk_get_request
virtqueue_pop
virtqueue_split_pop
virtqueue_map_desc
address_space_map
memory_access_is_direct# Return false
memory_region_supports_direct_access(qemu) info mtree
memory-region:pci_bridge_pci0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci 0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4 0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4 0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]This adds qemu_ram_{copy, move}() and replaces {memcpy, memmove}() with them in the ram device memory region accessors, similar to what's done in commit
4a2e242bbbso that the issue (MMIO access instructions were optimized to SSE instructions) covered by that commit is fixed. This makes 'ram_device_mem_ops' redundant, paving the way to revert that commit to make ram device region directly accessible again in the next patch.Reported-by Julia Graham <[email protected]>Suggested-by Michael S. Tsirkin <[email protected]>Suggested-by Peter Xu <[email protected]>Suggested-by Richard Henderson <[email protected]>Suggested-by Peter Maydell <[email protected]>Signed-off-by Gavin Shan <[email protected]>---
v3: Documentation for qemu_ram_{copy, move} (Peter/Michael)
Supportqemu_ram_move()for overlapped src/dest (Richard)
Use {memcpy, memmove} if step is 16-bytes or more (Michael)
Code improvements (Richard/Michael)--- hw/remote/vfio-user-obj.c | 4 +- include/system/memory.h | 32 ++++++- system/physmem.c | 178 +++++++++++++++++++++++++++++++++++++- 3 files changed, 207 insertions(+), 7 deletions(-)hw/remote/vfio-user-obj.c +2
-2@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset, ram_ptr =memory_region_get_ram_ptr (mr);if (is_write) { -memcpy ((ram_ptr + offset), buf, size); +qemu_ram_copy (ram_ptr + offset, buf, size); }else { -memcpy (buf, (ram_ptr + offset), size); +qemu_ram_copy (buf, ram_ptr + offset, size); }return 0 ;include/system/memory.h +4
-0@@ -2897,6 +2897,36 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);void address_space_unregister_map_client (AddressSpace *as,QEMUBH *bh);/* Internal functions, part of the implementation of address_space_read. */ + +/** +* qemu_ram_copy: copy data to ramblock +* include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2913
Quoted 10 lines
include/system/memory.h +10
-0+* @dst: destination where the data is copied to +* @src: source where the data is copied from +* @n: length of data to be copied +* +* +* Copy @n bytes from @src to @dst with the assumption that @src and @dst +* do not overlap. Handles special cases such as uncacheable ramblocks +* correctly. Use this for accessing ramblock in response to DMA/VCPU IO, +* in preference to memcpy(). +*/ The documentation for these functions needs to say what semantics the function is providing (e.g. can I rely on it to do a 4 byte aligned load as a single 4 byte read?). This does not.
include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2919
Quoted 6 lines
include/system/memory.h +6
-0+/* x86 should work with __builtin_{memcpy, memmove}() for IO access */ +#if defined(__i386__) || defined(__x86_64__) +#define HOST_UNALIGNED_MMIO_OK 1 +#else +#define HOST_UNALIGNED_MMIO_OK 0 +#endif This is still wrong. We should not have "x86 magically works and all other hosts do something different" ifdefs. Define what semantics you need and then we can figure out how to implement them.
My current thought is that we need to handle accesses of 1, 2, 4 and 8 bytes that are naturally aligned by ensuring that we do exactly one host load/store of that type, and that anything else is "the guest isn't relying on specific semantics here, we can just assume it's plain old RAM and do whatever". That would not require any architecture specific ifdefs.
Quoted 2 lines
thanks
-- PMM
Well. X86 is special, as usual. It allows unaligned mmio so we really have no way to know an x86 guest does not intend just that. That can only be emulated perfectly on x86 which is sad but I see no reason to actively break it.
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 14 lines · Michael S. Tsirkin
Quoted 11 lines · Peter Maydell
This is still wrong. We should not have "x86 magically works and all other hosts do something different" ifdefs. Define what semantics you need and then we can figure out how to implement them.
My current thought is that we need to handle accesses of 1, 2, 4 and 8 bytes that are naturally aligned by ensuring that we do exactly one host load/store of that type, and that anything else is "the guest isn't relying on specific semantics here, we can just assume it's plain old RAM and do whatever". That would not require any architecture specific ifdefs.
Quoted 4 lines
Well. X86 is special, as usual. It allows unaligned mmio so we really have no way to know an x86 guest does not intend just that. That can only be emulated perfectly on x86 which is sad but I see no reason to actively break it.
What actual hardware will rely on unaligned accesses appearing to it as single unaligned accesses?
And if we do need this, we need to specify:
- what are the semantics we need to provide?
- what is the use case that relies on these? Then we can see how closely we can match that.
There's a lot of extra complexity added in here for something it I don't think is ever going to matter, and having the most common host architecture take a totally different codepath from everything else is a recipe for that other codepath breaking.
-- PMM
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 15 lines · Peter Maydell
Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · Peter Maydell
This is still wrong. We should not have "x86 magically works and all other hosts do something different" ifdefs. Define what semantics you need and then we can figure out how to implement them.
My current thought is that we need to handle accesses of 1, 2, 4 and 8 bytes that are naturally aligned by ensuring that we do exactly one host load/store of that type, and that anything else is "the guest isn't relying on specific semantics here, we can just assume it's plain old RAM and do whatever". That would not require any architecture specific ifdefs.
Quoted 7 lines
Quoted 4 lines
Well. X86 is special, as usual. It allows unaligned mmio so we really have no way to know an x86 guest does not intend just that. That can only be emulated perfectly on x86 which is sad but I see no reason to actively break it.
What actual hardware will rely on unaligned accesses appearing to it as single unaligned accesses?
/me shrugs
any hardware that was only tested under windows?
Quoted 2 lines
And if we do need this, we need to specify:
- what are the semantics we need to provide?
match what happens on bare metal
Quoted 2 lines
- what is the use case that relies on these? Then we can see how closely we can match that.
passhtrough of weird devices, first of all.
Quoted 8 lines
There's a lot of extra complexity added in here for something it I don't think is ever going to matter, and having the most common host architecture take a totally different codepath from everything else is a recipe for that other codepath breaking.
-- PMM
Not sure we have to be pedantic. We have arch specific code and the world did not end. It's not a lot of code, it's just for unaligned accesses on MMIO, which might or might not be broken anyway. And it worked already, so it's more about going back to what we did earlier.
Signature · 1 line
MST
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 22 lines · Michael S. Tsirkin
Quoted 20 lines · Peter Maydell
Quoted 18 lines · Michael S. Tsirkin
Quoted 11 lines · Peter Maydell
This is still wrong. We should not have "x86 magically works and all other hosts do something different" ifdefs. Define what semantics you need and then we can figure out how to implement them.
My current thought is that we need to handle accesses of 1, 2, 4 and 8 bytes that are naturally aligned by ensuring that we do exactly one host load/store of that type, and that anything else is "the guest isn't relying on specific semantics here, we can just assume it's plain old RAM and do whatever". That would not require any architecture specific ifdefs.
Well. X86 is special, as usual. It allows unaligned mmio so we really have no way to know an x86 guest does not intend just that. That can only be emulated perfectly on x86 which is sad but I see no reason to actively break it.
Quoted 5 lines
Quoted 2 lines
What actual hardware will rely on unaligned accesses appearing to it as single unaligned accesses?
/me shrugs
any hardware that was only tested under windows?
I seriously doubt any hardware designer implements registers that are not aligned, especially on PCI devices. "Only tested under Windows" doesn't imply "hardware designer did something nuts".
Quoted 4 lines
Quoted 2 lines
And if we do need this, we need to specify:
- what are the semantics we need to provide?match what happens on bare metal
This is too vague. We need to write down what we mean, in enough specifics that we can then say "OK, we can do this this way", or "we can do this this way on some hosts, and these corner cases are not covered on other hosts".
Quoted 4 lines
Quoted 2 lines
- what is the use case that relies on these? Then we can see how closely we can match that.
passhtrough of weird devices, first of all.
This isn't a use case, it's a vague hypothetical.
Quoted 11 lines
Quoted 5 lines
There's a lot of extra complexity added in here for something it I don't think is ever going to matter, and having the most common host architecture take a totally different codepath from everything else is a recipe for that other codepath breaking.
Not sure we have to be pedantic. We have arch specific code and the world did not end. It's not a lot of code, it's just for unaligned accesses on MMIO, which might or might not be broken anyway. And it worked already, so it's more about going back to what we did earlier.
But we should not have arch specific code unless we have a reason to do it. And if we have a reason to do it then we should say what that reason is and what the cases the host has to cover are, so that when we can implement it correctly we do that.
-- PMM
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 26 lines · Peter Maydell
Quoted 25 lines · Michael S. Tsirkin
Quoted 22 lines · Peter Maydell
Quoted 14 lines · Michael S. Tsirkin
Quoted 11 lines · Peter Maydell
This is still wrong. We should not have "x86 magically works and all other hosts do something different" ifdefs. Define what semantics you need and then we can figure out how to implement them.
My current thought is that we need to handle accesses of 1, 2, 4 and 8 bytes that are naturally aligned by ensuring that we do exactly one host load/store of that type, and that anything else is "the guest isn't relying on specific semantics here, we can just assume it's plain old RAM and do whatever". That would not require any architecture specific ifdefs.
Quoted 4 lines
Well. X86 is special, as usual. It allows unaligned mmio so we really have no way to know an x86 guest does not intend just that. That can only be emulated perfectly on x86 which is sad but I see no reason to actively break it.
What actual hardware will rely on unaligned accesses appearing to it as single unaligned accesses?
Quoted 7 lines
Quoted 2 lines
/me shrugs
any hardware that was only tested under windows?I seriously doubt any hardware designer implements registers that are not aligned, especially on PCI devices. "Only tested under Windows" doesn't imply "hardware designer did something nuts".
Well I worry.
Quoted 10 lines
Quoted 4 lines
Quoted 2 lines
And if we do need this, we need to specify:
- what are the semantics we need to provide?match what happens on bare metal
This is too vague. We need to write down what we mean, in enough specifics that we can then say "OK, we can do this this way", or "we can do this this way on some hosts, and these corner cases are not covered on other hosts".
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset 0x1 within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.
I tried reading LEDCTL on e1000e:
byte @ 0xe00: 0x64
byte @ 0xe01: 0x2a
byte @ 0xe02: 0x00
byte @ 0xe03: 0x00
word @ 0xe00: 0x2a64
word @ 0xe01: 0x002aWorks fine.
How important is it to have a script like this to work correctly? I don't know.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
- what is the use case that relies on these? Then we can see how closely we can match that.
passhtrough of weird devices, first of all.
This isn't a use case, it's a vague hypothetical.
True.
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
There's a lot of extra complexity added in here for something it I don't think is ever going to matter, and having the most common host architecture take a totally different codepath from everything else is a recipe for that other codepath breaking.
Not sure we have to be pedantic. We have arch specific code and the world did not end. It's not a lot of code, it's just for unaligned accesses on MMIO, which might or might not be broken anyway. And it worked already, so it's more about going back to what we did earlier.
Quoted 7 lines
But we should not have arch specific code unless we have a reason to do it. And if we have a reason to do it then we should say what that reason is and what the cases the host has to cover are, so that when we can implement it correctly we do that.
-- PMM
I just like it when we implement whatever host does, qemu being a hardware emulator, after all.
If you feel it's nonsense, I'm not gonnu fight.
Signature · 1 line
MST
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 7 lines · Michael S. Tsirkin
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset
0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.I tried reading LEDCTL on e1000e:
Quoted 8 lines
byte @ 0xe00: 0x64 byte @ 0xe01: 0x2a byte @ 0xe02: 0x00 byte @ 0xe03: 0x00 word @ 0xe00: 0x2a64 word @ 0xe01: 0x002aWorks fine.
The e1000e datasheet actually documents what it does in this case (slightly surprising, since hardware engineers love to leave this kind of corner case undocumented):
# For registers that should be accessed as 32-bit double words,
# partial writes (less than a 32-bit double word) does not take # effect (such as, the write is ignored). Partial reads # return all 32 bits of data regardless of the byte enables. # # Note: Partial reads to clear-by-read registers (such as, ICR) # can have unexpected results since all 32 bits are actually read # regardless of the byte enables. Partial reads should not be done.
So for this specific device that access is out-of-spec.
I guess what I'm wondering is: can we just have code that does an aligned exact-width access in the 1/2/4/8 byte aligned case, and the host's best approximation to an unaligned exact-width access for the 2/4/8 byte unaligned case? (so on sparc you get multiple smaller accesses, and on most archs including x86 and arm you get an unaligned load). That would mean the guest could potentially provoke a fault on the load/store on an access to a passthrough device, but if you give the guest passthrough access it can very likely provoke a fault anyway, depending on exactly what the device is.
I think the most likely reason for an unaligned access in this codepath is "it's actually RAM, either really host RAM or else something memory-like in a BAR", and either way if the guest does a 4-byte unaligned access then doing a 4-byte unaligned access seems better than second-guessing it, even on non-x86.
-- PMM
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 31 lines · Peter Maydell
Quoted 7 lines · Michael S. Tsirkin
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset
0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.I tried reading LEDCTL on e1000e:
Quoted 8 lines
byte @ 0xe00: 0x64 byte @ 0xe01: 0x2a byte @ 0xe02: 0x00 byte @ 0xe03: 0x00 word @ 0xe00: 0x2a64 word @ 0xe01: 0x002aWorks fine.
The e1000e datasheet actually documents what it does in this case (slightly surprising, since hardware engineers love to leave this kind of corner case undocumented):
# For registers that should be accessed as 32-bit double words,
# partial writes (less than a 32-bit double word) does not take # effect (such as, the write is ignored). # Partial reads # return all 32 bits of data regardless of the byte enables. # # Note: Partial reads to clear-by-read registers (such as, ICR) # can have unexpected results since all 32 bits are actually read # regardless of the byte enables. Partial reads should not be done.
Quoted 1 line
So for this specific device that access is out-of-spec.
You mean that access to clear by read should not be done, right?
Quoted 5 lines
I guess what I'm wondering is: can we just have code that does an aligned exact-width access in the 1/2/4/8 byte aligned case, and the host's best approximation to an unaligned exact-width access for the 2/4/8 byte unaligned case?
That's my idea, too.
Quoted 3 lines
(so on sparc you get multiple smaller accesses, and on most archs including x86 and arm you get an unaligned load).
I thought unaligned load from uncacheable on arm is also a fault?
Quoted 6 lines
That would mean the guest could
potentially provoke a fault on the load/store on an access to a passthrough device, but if you give the guest passthrough access it can very likely provoke a fault anyway, depending on exactly what the device is.
Quoted 8 lines
I think the most likely reason for an unaligned access in this codepath is "it's actually RAM, either really host RAM or else something memory-like in a BAR", and either way if the guest does a 4-byte unaligned access then doing a 4-byte unaligned access seems better than second-guessing it, even on non-x86.
-- PMM
Right though remember: whether it's RAM doesn't matter. What matters is how we map it. qemu might fault because it maps NC but guest maps cacheable and it's ok.
But, all this in theory. At a high level, I personally think going with what you propose as a 1st approximation is entirely reasonable, except for one thing: we really should not crash qemu, since access can be from guest userspace.
qemu should know how things are mapped, and predict it will fault. Approximating by splitting in that case sounds reasonable.
Signature · 1 line
MST
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 33 lines · Michael S. Tsirkin
Quoted 31 lines · Peter Maydell
Quoted 7 lines · Michael S. Tsirkin
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset
0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.I tried reading LEDCTL on e1000e:
Quoted 8 lines
byte @ 0xe00: 0x64 byte @ 0xe01: 0x2a byte @ 0xe02: 0x00 byte @ 0xe03: 0x00 word @ 0xe00: 0x2a64 word @ 0xe01: 0x002aWorks fine.
The e1000e datasheet actually documents what it does in this case (slightly surprising, since hardware engineers love to leave this kind of corner case undocumented):
# For registers that should be accessed as 32-bit double words,
# partial writes (less than a 32-bit double word) does not take # effect (such as, the write is ignored). # Partial reads # return all 32 bits of data regardless of the byte enables. # # Note: Partial reads to clear-by-read registers (such as, ICR) # can have unexpected results since all 32 bits are actually read # regardless of the byte enables. Partial reads should not be done.
Quoted 3 lines
Quoted 1 line
So for this specific device that access is out-of-spec.
You mean that access to clear by read should not be done, right?
The datasheet is ambiguous about whether "Partial reads should not be done" is meant to apply generally or only to clear-by-read registers.
Quoted 6 lines
Quoted 5 lines
I guess what I'm wondering is: can we just have code that does an aligned exact-width access in the 1/2/4/8 byte aligned case, and the host's best approximation to an unaligned exact-width access for the 2/4/8 byte unaligned case?
Quoted 8 lines
That's my idea, too.
Quoted 3 lines
(so on sparc you get multiple smaller accesses, and on most archs including x86 and arm you get an unaligned load).
I thought unaligned load from uncacheable on arm is also a fault?
For Arm the distinction is not cacheable/uncacheable but Normal vs Device. (Device is essentially for things which are not RAM; Normal is for RAM and RAM-like things, and includes all of Normal Non-cacheable, Normal WT-Cacheable and Normal WB-Cacheable.) Things mapped as Normal memory don't generate unaligned faults (unless the guest turned them on deliberately). For Device memory, it is IMPLEMENTATION DEFINED whether you get an alignment fault or not if you map something as Device that could have handled unaligned accesses if you had mapped it as Normal.
Quoted 13 lines
Quoted 6 lines
That would mean the guest could
potentially provoke a fault on the load/store on an access to a passthrough device, but if you give the guest passthrough access it can very likely provoke a fault anyway, depending on exactly what the device is.Quoted 6 lines
I think the most likely reason for an unaligned access in this codepath is "it's actually RAM, either really host RAM or else something memory-like in a BAR", and either way if the guest does a 4-byte unaligned access then doing a 4-byte unaligned access seems better than second-guessing it, even on non-x86.
Quoted 3 lines
Right though remember: whether it's RAM doesn't matter. What matters is how we map it. qemu might fault because it maps NC but guest maps cacheable and it's ok.
If QEMU and the guest disagree about the memory attributes on Arm then we have already lost, because the architecture says that memory attribute mismatches result in a variety of undesirable effects including things like loss of cache coherency (i.e. read and writes via QEMU's NC mapping disagree with ones via the guest's cacheable mapping because the latter are hitting in the cache and the former are bypassing it).
Quoted 4 lines
But, all this in theory. At a high level, I personally think going with what you propose as a 1st approximation is entirely reasonable, except for one thing: we really should not crash qemu, since access can be from guest userspace.
You can't prevent faults entirely, though -- if the device being mapped has e.g. behaviour that says "unaligned accesses will fault" and then the x86 guest does an unaligned access, then the device will trigger a fault, and the fault is what you want because it's what the guest would see on real h/w. Unfortunately we don't have a convenient way to feed the fault back to the guest. At some level if you pass through host hardware you're relying on the guest to not do totally stupid things.
thanks
-- PMM
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 34 lines · Peter Maydell
Quoted 33 lines · Michael S. Tsirkin
Quoted 31 lines · Peter Maydell
Quoted 7 lines · Michael S. Tsirkin
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset
0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.I tried reading LEDCTL on e1000e:
Quoted 8 lines
byte @ 0xe00: 0x64 byte @ 0xe01: 0x2a byte @ 0xe02: 0x00 byte @ 0xe03: 0x00 word @ 0xe00: 0x2a64 word @ 0xe01: 0x002aWorks fine.
The e1000e datasheet actually documents what it does in this case (slightly surprising, since hardware engineers love to leave this kind of corner case undocumented):
# For registers that should be accessed as 32-bit double words,
# partial writes (less than a 32-bit double word) does not take # effect (such as, the write is ignored). # Partial reads # return all 32 bits of data regardless of the byte enables. # # Note: Partial reads to clear-by-read registers (such as, ICR) # can have unexpected results since all 32 bits are actually read # regardless of the byte enables. Partial reads should not be done.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
So for this specific device that access is out-of-spec.
You mean that access to clear by read should not be done, right?
The datasheet is ambiguous about whether "Partial reads should not be done" is meant to apply generally or only to clear-by-read registers.
I think the document intends to say: the partial reads shouldn't be issued to the clear-by-read registers. Those partial reads on no-read-side-affect registers would be fine.
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
I guess what I'm wondering is: can we just have code that does an aligned exact-width access in the 1/2/4/8 byte aligned case, and the host's best approximation to an unaligned exact-width access for the 2/4/8 byte unaligned case?
That's my idea, too.
I assume this the conclusion of our discussions? If so, we just need to have unified function where __builtin_{memcpy, memmove}() are used for 1/2/4/8 bytes access no matter if the address is aligned, and fall back to {memcpy, memmove}() for other cases?
Quoted 58 lines
Quoted 6 lines
Quoted 3 lines
(so on sparc you get multiple smaller accesses, and on most archs including x86 and arm you get an unaligned load).
I thought unaligned load from uncacheable on arm is also a fault?
For Arm the distinction is not cacheable/uncacheable but Normal vs Device. (Device is essentially for things which are not RAM; Normal is for RAM and RAM-like things, and includes all of Normal Non-cacheable, Normal WT-Cacheable and Normal WB-Cacheable.) Things mapped as Normal memory don't generate unaligned faults (unless the guest turned them on deliberately). For Device memory, it is IMPLEMENTATION DEFINED whether you get an alignment fault or not if you map something as Device that could have handled unaligned accesses if you had mapped it as Normal.
Quoted 13 lines
Quoted 6 lines
That would mean the guest could
potentially provoke a fault on the load/store on an access to a passthrough device, but if you give the guest passthrough access it can very likely provoke a fault anyway, depending on exactly what the device is.Quoted 6 lines
I think the most likely reason for an unaligned access in this codepath is "it's actually RAM, either really host RAM or else something memory-like in a BAR", and either way if the guest does a 4-byte unaligned access then doing a 4-byte unaligned access seems better than second-guessing it, even on non-x86.
Quoted 3 lines
Right though remember: whether it's RAM doesn't matter. What matters is how we map it. qemu might fault because it maps NC but guest maps cacheable and it's ok.
If QEMU and the guest disagree about the memory attributes on Arm then we have already lost, because the architecture says that memory attribute mismatches result in a variety of undesirable effects including things like loss of cache coherency (i.e. read and writes via QEMU's NC mapping disagree with ones via the guest's cacheable mapping because the latter are hitting in the cache and the former are bypassing it).
Quoted 4 lines
But, all this in theory. At a high level, I personally think going with what you propose as a 1st approximation is entirely reasonable, except for one thing: we really should not crash qemu, since access can be from guest userspace.
You can't prevent faults entirely, though -- if the device being mapped has e.g. behaviour that says "unaligned accesses will fault" and then the x86 guest does an unaligned access, then the device will trigger a fault, and the fault is what you want because it's what the guest would see on real h/w. Unfortunately we don't have a convenient way to feed the fault back to the guest. At some level if you pass through host hardware you're relying on the guest to not do totally stupid things.
Quoted 3 lines
thanks
-- PMM
Thanks,
Gavin
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Hi Michael and Peter,
Quoted 54 lines · Gavin Shan
Quoted 34 lines · Peter Maydell
Quoted 33 lines · Michael S. Tsirkin
Quoted 31 lines · Peter Maydell
Quoted 7 lines · Michael S. Tsirkin
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset
0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.I tried reading LEDCTL on e1000e:
Quoted 8 lines
byte @
0xe00:0x64
byte @0xe01:0x2a
byte @0xe02:0x00
byte @0xe03:0x00
word @0xe00:0x2a64
word @0xe01:0x002aWorks fine.
The e1000e datasheet actually documents what it does in this case (slightly surprising, since hardware engineers love to leave this kind of corner case undocumented):
# For registers that should be accessed as 32-bit double words,
# partial writes (less than a 32-bit double word) does not take # effect (such as, the write is ignored). # Partial reads # return all 32 bits of data regardless of the byte enables. # # Note: Partial reads to clear-by-read registers (such as, ICR) # can have unexpected results since all 32 bits are actually read # regardless of the byte enables. Partial reads should not be done.Quoted 8 lines
Quoted 3 lines
Quoted 1 line
So for this specific device that access is out-of-spec.
You mean that access to clear by read should not be done, right?
The datasheet is ambiguous about whether "Partial reads should not be done" is meant to apply generally or only to clear-by-read registers.
I think the document intends to say: the partial reads shouldn't be issued to the clear-by-read registers. Those partial reads on no-read-side-affect registers would be fine.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
I guess what I'm wondering is: can we just have code that does an aligned exact-width access in the 1/2/4/8 byte aligned case, and the host's best approximation to an unaligned exact-width access for the 2/4/8 byte unaligned case?
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
That's my idea, too.
I assume this the conclusion of our discussions? If so, we just need to have unified function where __builtin_{memcpy, memmove}() are used for 1/2/4/8 bytes access no matter if the address is aligned, and fall back to {memcpy, memmove}() for other cases?
Could you help to confirm if the conclusion is to have unified qemu_ram_{copy, move}() for all architectures? In qemu_ram_{copy, move}(), the fixed sized accesses like 1/2/4/8 bytes are handled by __builtin_{memcpy, memmove}() and fall back to memcpy/memmove() for other sized accesses. It's basically what was proposed by Michael :-)
Something like below.
-----> include/system/memory.h
+/**
+ * qemu_ram_copy: copy data to ramblock
+ *
+ * @dst: destination where the data is copied to
+ * @src: source where the data is copied from
+ * @n: length of data to be copied
+ *
+ *
+ * Copy @n bytes from @src to @dst with the assumption that @src and @dst
+ * do not overlap. The fixed sized accesses like 1/2/4/8 bytes are handled
+ * by __builtin_memcpy() to avoid the optimizations applied to memcpy(),
+ * which can be unsafe to DMA/VCPU IO. Fall back to memcpy() for other
+ * sized accesses and the safety is ensured by users.
+ */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+
+/**
+ * qemu_ram_move: move data to ramblock
+ *
+ * @dst: destination where the data is moved to
+ * @src: source where the data is moved from
+ * @n: length of data to be moved
+ *
+ * Move @n bytes from @src to @dst with the assumption that @src and @dst
+ * can overlap. The fixed sized accesses like 1/2/4/8 bytes are handled
+ * by __builtin_memmove() to avoid the optimizations applied to memmove(),
+ * which can be unsafe to DMA/VCPU IO. Fall back to memmove() for other
+ * sized accesses and the safety is ensured by users.
+ */
+void qemu_ram_move(void *dest, const void *src, size_t n);
+
-----> system/physmem.c
+void qemu_ram_copy(void *dst, const void *src, size_t n) +{
+ switch (n) {
+ case 1:
+ __builtin_memcpy(dst, src, 1);
+ break;
+ case 2:
+ __builtin_memcpy(dst, src, 2);
+ break;
+ case 4:
+ __builtin_memcpy(dst, src, 4);
+ break;
+ case 8:
+ __builtin_memcpy(dst, src, 8);
+ break;
+ default:
+ memcpy(dst, src, n);
+ }
+}
+
+void qemu_ram_move(void *dst, const void *src, size_t n) +{
+ switch (n) {
+ case 1:
+ __builtin_memmove(dst, src, 1);
+ break;
+ case 2:
+ __builtin_memmove(dst, src, 2);
+ break;
+ case 4:
+ __builtin_memmove(dst, src, 4);
+ break;
+ case 8:
+ __builtin_memmove(dst, src, 8);
+ break;
+ default:
+ memmove(dst, src, n);
+ }
+}
+
Quoted 58 lines
Quoted 58 lines
Quoted 6 lines
Quoted 3 lines
(so on sparc you get multiple smaller accesses, and on most archs including x86 and arm you get an unaligned load).
I thought unaligned load from uncacheable on arm is also a fault?
For Arm the distinction is not cacheable/uncacheable but Normal vs Device. (Device is essentially for things which are not RAM; Normal is for RAM and RAM-like things, and includes all of Normal Non-cacheable, Normal WT-Cacheable and Normal WB-Cacheable.) Things mapped as Normal memory don't generate unaligned faults (unless the guest turned them on deliberately). For Device memory, it is IMPLEMENTATION DEFINED whether you get an alignment fault or not if you map something as Device that could have handled unaligned accesses if you had mapped it as Normal.
Quoted 13 lines
Quoted 6 lines
That would mean the guest could
potentially provoke a fault on the load/store on an access to a passthrough device, but if you give the guest passthrough access it can very likely provoke a fault anyway, depending on exactly what the device is.Quoted 6 lines
I think the most likely reason for an unaligned access in this codepath is "it's actually RAM, either really host RAM or else something memory-like in a BAR", and either way if the guest does a 4-byte unaligned access then doing a 4-byte unaligned access seems better than second-guessing it, even on non-x86.
Quoted 3 lines
Right though remember: whether it's RAM doesn't matter. What matters is how we map it. qemu might fault because it maps NC but guest maps cacheable and it's ok.
If QEMU and the guest disagree about the memory attributes on Arm then we have already lost, because the architecture says that memory attribute mismatches result in a variety of undesirable effects including things like loss of cache coherency (i.e. read and writes via QEMU's NC mapping disagree with ones via the guest's cacheable mapping because the latter are hitting in the cache and the former are bypassing it).
Quoted 4 lines
But, all this in theory. At a high level, I personally think going with what you propose as a 1st approximation is entirely reasonable, except for one thing: we really should not crash qemu, since access can be from guest userspace.
You can't prevent faults entirely, though -- if the device being mapped has e.g. behaviour that says "unaligned accesses will fault" and then the x86 guest does an unaligned access, then the device will trigger a fault, and the fault is what you want because it's what the guest would see on real h/w. Unfortunately we don't have a convenient way to feed the fault back to the guest. At some level if you pass through host hardware you're relying on the guest to not do totally stupid things.
Quoted 3 lines
Quoted 3 lines
thanks
-- PMM
Thanks,
Gavin
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 66 lines · Gavin Shan
Hi Michael and Peter,
Quoted 54 lines · Gavin Shan
Quoted 34 lines · Peter Maydell
Quoted 33 lines · Michael S. Tsirkin
Quoted 31 lines · Peter Maydell
Quoted 7 lines · Michael S. Tsirkin
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset
0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.I tried reading LEDCTL on e1000e:
Quoted 8 lines
byte @
0xe00:0x64
byte @0xe01:0x2a
byte @0xe02:0x00
byte @0xe03:0x00
word @0xe00:0x2a64
word @0xe01:0x002aWorks fine.
The e1000e datasheet actually documents what it does in this case (slightly surprising, since hardware engineers love to leave this kind of corner case undocumented):
# For registers that should be accessed as 32-bit double words,
# partial writes (less than a 32-bit double word) does not take # effect (such as, the write is ignored). # Partial reads # return all 32 bits of data regardless of the byte enables. # # Note: Partial reads to clear-by-read registers (such as, ICR) # can have unexpected results since all 32 bits are actually read # regardless of the byte enables. Partial reads should not be done.Quoted 8 lines
Quoted 3 lines
Quoted 1 line
So for this specific device that access is out-of-spec.
You mean that access to clear by read should not be done, right?
The datasheet is ambiguous about whether "Partial reads should not be done" is meant to apply generally or only to clear-by-read registers.
I think the document intends to say: the partial reads shouldn't be issued to the clear-by-read registers. Those partial reads on no-read-side-affect registers would be fine.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
I guess what I'm wondering is: can we just have code that does an aligned exact-width access in the 1/2/4/8 byte aligned case, and the host's best approximation to an unaligned exact-width access for the 2/4/8 byte unaligned case?
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
That's my idea, too.
I assume this the conclusion of our discussions? If so, we just need to have unified function where __builtin_{memcpy, memmove}() are used for 1/2/4/8 bytes access no matter if the address is aligned, and fall back to {memcpy, memmove}() for other cases?
Quoted 5 lines
Could you help to confirm if the conclusion is to have unified qemu_ram_{copy, move}() for all architectures? In qemu_ram_{copy, move}(), the fixed sized accesses like 1/2/4/8 bytes are handled by __builtin_{memcpy, memmove}() and fall back to memcpy/memmove() for other sized accesses. It's basically what was proposed by Michael :-)
No. __builtin_ and memcpy/memmove are same on modern linuxes.
I'd say as a 1st step, on x86 hosts we can do a fixed size memmove for 1/2/4/8, on other hosts
*aligned* 1/2/4/8 bytes can do
__atomic_store ... __ATOMIC_RELAXED
and the rest can hobble along on memmove
and hope for the best.
And memcpy is likely not the right thing, ever.
Quoted 144 lines
Something like below.
----->
include/system/memory.h(quoted diff) +71
-1+/** + * qemu_ram_copy: copy data to ramblock + * + * @dst: destination where the data is copied to + * @src: source where the data is copied from + * @n: length of data to be copied + * + * + * Copy @n bytes from @src to @dst with the assumption that @src and @dst + * do not overlap. The fixed sized accesses like 1/2/4/8 bytes are handled + * by __builtin_memcpy() to avoid the optimizations applied to memcpy(), + * which can be unsafe to DMA/VCPU IO. Fall back to memcpy() for other + * sized accesses and the safety is ensured by users. + */ +void qemu_ram_copy(void *dest, const void *src, size_t n); + +/** + * qemu_ram_move: move data to ramblock + * + * @dst: destination where the data is moved to + * @src: source where the data is moved from + * @n: length of data to be moved + * + * Move @n bytes from @src to @dst with the assumption that @src and @dst + * can overlap. The fixed sized accesses like 1/2/4/8 bytes are handled + * by __builtin_memmove() to avoid the optimizations applied to memmove(), + * which can be unsafe to DMA/VCPU IO. Fall back to memmove() for other + * sized accesses and the safety is ensured by users. + */ +void qemu_ram_move(void *dest, const void *src, size_t n); + -----> system/physmem.c +void qemu_ram_copy(void *dst, const void *src, size_t n) +{ + switch (n) { + case 1: + __builtin_memcpy(dst, src, 1); + break; + case 2: + __builtin_memcpy(dst, src, 2); + break; + case 4: + __builtin_memcpy(dst, src, 4); + break; + case 8: + __builtin_memcpy(dst, src, 8); + break; + default: + memcpy(dst, src, n); + } +} + +void qemu_ram_move(void *dst, const void *src, size_t n) +{ + switch (n) { + case 1: + __builtin_memmove(dst, src, 1); + break; + case 2: + __builtin_memmove(dst, src, 2); + break; + case 4: + __builtin_memmove(dst, src, 4); + break; + case 8: + __builtin_memmove(dst, src, 8); + break; + default: + memmove(dst, src, n); + } +} +Quoted 60 lines
Quoted 60 lines
Quoted 8 lines
Quoted 5 lines
(so on sparc you get multiple smaller accesses, and on most archs including x86 and arm you get an unaligned load).
I thought unaligned load from uncacheable on arm is also a fault?
For Arm the distinction is not cacheable/uncacheable but Normal vs Device. (Device is essentially for things which are not RAM; Normal is for RAM and RAM-like things, and includes all of Normal Non-cacheable, Normal WT-Cacheable and Normal WB-Cacheable.) Things mapped as Normal memory don't generate unaligned faults (unless the guest turned them on deliberately). For Device memory, it is IMPLEMENTATION DEFINED whether you get an alignment fault or not if you map something as Device that could have handled unaligned accesses if you had mapped it as Normal.
Quoted 13 lines
Quoted 6 lines
That would mean the guest could
potentially provoke a fault on the load/store on an access to a passthrough device, but if you give the guest passthrough access it can very likely provoke a fault anyway, depending on exactly what the device is.Quoted 6 lines
I think the most likely reason for an unaligned access in this codepath is "it's actually RAM, either really host RAM or else something memory-like in a BAR", and either way if the guest does a 4-byte unaligned access then doing a 4-byte unaligned access seems better than second-guessing it, even on non-x86.
Quoted 3 lines
Right though remember: whether it's RAM doesn't matter. What matters is how we map it. qemu might fault because it maps NC but guest maps cacheable and it's ok.
If QEMU and the guest disagree about the memory attributes on Arm then we have already lost, because the architecture says that memory attribute mismatches result in a variety of undesirable effects including things like loss of cache coherency (i.e. read and writes via QEMU's NC mapping disagree with ones via the guest's cacheable mapping because the latter are hitting in the cache and the former are bypassing it).
Quoted 4 lines
But, all this in theory. At a high level, I personally think going with what you propose as a 1st approximation is entirely reasonable, except for one thing: we really should not crash qemu, since access can be from guest userspace.
You can't prevent faults entirely, though -- if the device being mapped has e.g. behaviour that says "unaligned accesses will fault" and then the x86 guest does an unaligned access, then the device will trigger a fault, and the fault is what you want because it's what the guest would see on real h/w. Unfortunately we don't have a convenient way to feed the fault back to the guest. At some level if you pass through host hardware you're relying on the guest to not do totally stupid things.
Quoted 3 lines
Quoted 3 lines
thanks
-- PMMThanks,
Gavin
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 75 lines · Michael S. Tsirkin
Quoted 66 lines · Gavin Shan
Hi Michael and Peter,
Quoted 54 lines · Gavin Shan
Quoted 34 lines · Peter Maydell
Quoted 33 lines · Michael S. Tsirkin
Quoted 31 lines · Peter Maydell
Quoted 7 lines · Michael S. Tsirkin
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset
0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.I tried reading LEDCTL on e1000e:
Quoted 8 lines
byte @
0xe00:0x64
byte @0xe01:0x2a
byte @0xe02:0x00
byte @0xe03:0x00
word @0xe00:0x2a64
word @0xe01:0x002aWorks fine.
The e1000e datasheet actually documents what it does in this case (slightly surprising, since hardware engineers love to leave this kind of corner case undocumented):
# For registers that should be accessed as 32-bit double words,
# partial writes (less than a 32-bit double word) does not take # effect (such as, the write is ignored). # Partial reads # return all 32 bits of data regardless of the byte enables. # # Note: Partial reads to clear-by-read registers (such as, ICR) # can have unexpected results since all 32 bits are actually read # regardless of the byte enables. Partial reads should not be done.Quoted 8 lines
Quoted 3 lines
Quoted 1 line
So for this specific device that access is out-of-spec.
You mean that access to clear by read should not be done, right?
The datasheet is ambiguous about whether "Partial reads should not be done" is meant to apply generally or only to clear-by-read registers.
I think the document intends to say: the partial reads shouldn't be issued to the clear-by-read registers. Those partial reads on no-read-side-affect registers would be fine.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
I guess what I'm wondering is: can we just have code that does an aligned exact-width access in the 1/2/4/8 byte aligned case, and the host's best approximation to an unaligned exact-width access for the 2/4/8 byte unaligned case?
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
That's my idea, too.
I assume this the conclusion of our discussions? If so, we just need to have unified function where __builtin_{memcpy, memmove}() are used for 1/2/4/8 bytes access no matter if the address is aligned, and fall back to {memcpy, memmove}() for other cases?
Quoted 5 lines
Could you help to confirm if the conclusion is to have unified qemu_ram_{copy, move}() for all architectures? In qemu_ram_{copy, move}(), the fixed sized accesses like 1/2/4/8 bytes are handled by __builtin_{memcpy, memmove}() and fall back to memcpy/memmove() for other sized accesses. It's basically what was proposed by Michael :-)
No.
__builtin_and memcpy/memmove are same on modern linuxes.
Quoted 8 lines
I'd say as a 1st step, on x86 hosts we can do a fixed size memmove for 1/2/4/8, on other hosts
*aligned* 1/2/4/8 bytes can do
__atomic_store...__ATOMIC_RELAXED
and the rest can hobble along on memmove
and hope for the best.And memcpy is likely not the right thing, ever.
Or maybe better hobble along on a loop over __atomic_store. Safer.
And as a 2nd step, extend vfio to tell us how is the device mapped, and then we can maybe fallback to a loop when we know it is not safe to access with memcpy.
Quoted 145 lines
Quoted 145 lines
Something like below.
----->
include/system/memory.h(quoted diff) +71
-1+/** + * qemu_ram_copy: copy data to ramblock + * + * @dst: destination where the data is copied to + * @src: source where the data is copied from + * @n: length of data to be copied + * + * + * Copy @n bytes from @src to @dst with the assumption that @src and @dst + * do not overlap. The fixed sized accesses like 1/2/4/8 bytes are handled + * by __builtin_memcpy() to avoid the optimizations applied to memcpy(), + * which can be unsafe to DMA/VCPU IO. Fall back to memcpy() for other + * sized accesses and the safety is ensured by users. + */ +void qemu_ram_copy(void *dest, const void *src, size_t n); + +/** + * qemu_ram_move: move data to ramblock + * + * @dst: destination where the data is moved to + * @src: source where the data is moved from + * @n: length of data to be moved + * + * Move @n bytes from @src to @dst with the assumption that @src and @dst + * can overlap. The fixed sized accesses like 1/2/4/8 bytes are handled + * by __builtin_memmove() to avoid the optimizations applied to memmove(), + * which can be unsafe to DMA/VCPU IO. Fall back to memmove() for other + * sized accesses and the safety is ensured by users. + */ +void qemu_ram_move(void *dest, const void *src, size_t n); + -----> system/physmem.c +void qemu_ram_copy(void *dst, const void *src, size_t n) +{ + switch (n) { + case 1: + __builtin_memcpy(dst, src, 1); + break; + case 2: + __builtin_memcpy(dst, src, 2); + break; + case 4: + __builtin_memcpy(dst, src, 4); + break; + case 8: + __builtin_memcpy(dst, src, 8); + break; + default: + memcpy(dst, src, n); + } +} + +void qemu_ram_move(void *dst, const void *src, size_t n) +{ + switch (n) { + case 1: + __builtin_memmove(dst, src, 1); + break; + case 2: + __builtin_memmove(dst, src, 2); + break; + case 4: + __builtin_memmove(dst, src, 4); + break; + case 8: + __builtin_memmove(dst, src, 8); + break; + default: + memmove(dst, src, n); + } +} +Quoted 60 lines
Quoted 60 lines
Quoted 8 lines
Quoted 5 lines
(so on sparc you get multiple smaller accesses, and on most archs including x86 and arm you get an unaligned load).
I thought unaligned load from uncacheable on arm is also a fault?
For Arm the distinction is not cacheable/uncacheable but Normal vs Device. (Device is essentially for things which are not RAM; Normal is for RAM and RAM-like things, and includes all of Normal Non-cacheable, Normal WT-Cacheable and Normal WB-Cacheable.) Things mapped as Normal memory don't generate unaligned faults (unless the guest turned them on deliberately). For Device memory, it is IMPLEMENTATION DEFINED whether you get an alignment fault or not if you map something as Device that could have handled unaligned accesses if you had mapped it as Normal.
Quoted 13 lines
Quoted 6 lines
That would mean the guest could
potentially provoke a fault on the load/store on an access to a passthrough device, but if you give the guest passthrough access it can very likely provoke a fault anyway, depending on exactly what the device is.Quoted 6 lines
I think the most likely reason for an unaligned access in this codepath is "it's actually RAM, either really host RAM or else something memory-like in a BAR", and either way if the guest does a 4-byte unaligned access then doing a 4-byte unaligned access seems better than second-guessing it, even on non-x86.
Quoted 3 lines
Right though remember: whether it's RAM doesn't matter. What matters is how we map it. qemu might fault because it maps NC but guest maps cacheable and it's ok.
If QEMU and the guest disagree about the memory attributes on Arm then we have already lost, because the architecture says that memory attribute mismatches result in a variety of undesirable effects including things like loss of cache coherency (i.e. read and writes via QEMU's NC mapping disagree with ones via the guest's cacheable mapping because the latter are hitting in the cache and the former are bypassing it).
Quoted 4 lines
But, all this in theory. At a high level, I personally think going with what you propose as a 1st approximation is entirely reasonable, except for one thing: we really should not crash qemu, since access can be from guest userspace.
You can't prevent faults entirely, though -- if the device being mapped has e.g. behaviour that says "unaligned accesses will fault" and then the x86 guest does an unaligned access, then the device will trigger a fault, and the fault is what you want because it's what the guest would see on real h/w. Unfortunately we don't have a convenient way to feed the fault back to the guest. At some level if you pass through host hardware you're relying on the guest to not do totally stupid things.
Quoted 3 lines
Quoted 3 lines
thanks
-- PMMThanks,
Gavin
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
In reply to Michael S. TsirkinQuoted 11 lines · Michael S. Tsirkin
Quoted 8 lines · Gavin Shan
Hi Michael and Peter,
Could you help to confirm if the conclusion is to have unified qemu_ram_{copy, move}() for all architectures? In qemu_ram_{copy, move}(), the fixed sized accesses like 1/2/4/8 bytes are handled by __builtin_{memcpy, memmove}() and fall back to memcpy/memmove() for other sized accesses. It's basically what was proposed by Michael :-)
Quoted 8 lines
No.
__builtin_and memcpy/memmove are same on modern linuxes.I'd say as a 1st step, on x86 hosts we can do a fixed size memmove for 1/2/4/8, on other hosts
*aligned* 1/2/4/8 bytes can do
__atomic_store...__ATOMIC_RELAXED
and the rest can hobble along on memmove
and hope for the best.
I don't think we should be doing something that's x86-host specific, as I've mentioned previously.
-- PMM
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 21 lines · Peter Maydell
Quoted 11 lines · Michael S. Tsirkin
Quoted 8 lines · Gavin Shan
Hi Michael and Peter,
Could you help to confirm if the conclusion is to have unified qemu_ram_{copy, move}() for all architectures? In qemu_ram_{copy, move}(), the fixed sized accesses like 1/2/4/8 bytes are handled by __builtin_{memcpy, memmove}() and fall back to memcpy/memmove() for other sized accesses. It's basically what was proposed by Michael :-)
Quoted 8 lines
No.
__builtin_and memcpy/memmove are same on modern linuxes.I'd say as a 1st step, on x86 hosts we can do a fixed size memmove for 1/2/4/8, on other hosts
*aligned* 1/2/4/8 bytes can do
__atomic_store...__ATOMIC_RELAXED
and the rest can hobble along on memmove
and hope for the best.
Quoted 4 lines
I don't think we should be doing something that's x86-host specific, as I've mentioned previously.
-- PMM
Shrug. Of course we already have arch specific code in asm. But we can make the unaligned 1/2/4/8 for x86 host a separate optimization on top. They are unlikely to be common.
Signature · 1 line
MST
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 22 lines · Michael S. Tsirkin
Quoted 21 lines · Peter Maydell
Quoted 19 lines · Michael S. Tsirkin
Quoted 8 lines · Gavin Shan
Hi Michael and Peter,
Could you help to confirm if the conclusion is to have unified qemu_ram_{copy, move}() for all architectures? In qemu_ram_{copy, move}(), the fixed sized accesses like 1/2/4/8 bytes are handled by __builtin_{memcpy, memmove}() and fall back to memcpy/memmove() for other sized accesses. It's basically what was proposed by Michael :-)
No.
__builtin_and memcpy/memmove are same on modern linuxes.I'd say as a 1st step, on x86 hosts we can do a fixed size memmove for 1/2/4/8, on other hosts
*aligned* 1/2/4/8 bytes can do
__atomic_store...__ATOMIC_RELAXED
and the rest can hobble along on memmove
and hope for the best.
Quoted 8 lines
Quoted 4 lines
I don't think we should be doing something that's x86-host specific, as I've mentioned previously.
-- PMM
Shrug. Of course we already have arch specific code in asm. But we can make the unaligned 1/2/4/8 for x86 host a separate optimization on top. They are unlikely to be common.
Note we have include/qemu/host-pci-mmio.h with s390x specific optimizations in util/s390x_pci_mmio.c ...
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 22 lines · Philippe Mathieu-Daudé
Quoted 21 lines · Michael S. Tsirkin
Quoted 20 lines · Peter Maydell
Quoted 7 lines · Michael S. Tsirkin
Quoted 5 lines · Gavin Shan
Could you help to confirm if the conclusion is to have unified qemu_ram_{copy, move}() for all architectures? In qemu_ram_{copy, move}(), the fixed sized accesses like 1/2/4/8 bytes are handled by __builtin_{memcpy, memmove}() and fall back to memcpy/memmove() for other sized accesses. It's basically what was proposed by Michael :-)
Quoted 8 lines
No.
__builtin_and memcpy/memmove are same on modern linuxes.I'd say as a 1st step, on x86 hosts we can do a fixed size memmove for 1/2/4/8, on other hosts
*aligned* 1/2/4/8 bytes can do
__atomic_store...__ATOMIC_RELAXED
and the rest can hobble along on memmove
and hope for the best.I don't think we should be doing something that's x86-host specific, as I've mentioned previously.
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
-- PMM
Shrug. Of course we already have arch specific code in asm. But we can make the unaligned 1/2/4/8 for x86 host a separate optimization on top. They are unlikely to be common.
Note we have
include/qemu/host-pci-mmio.hwith s390x specific optimizations inutil/s390x_pci_mmio.c...
Philippe, Thanks for pointing it out that s390x already had private PCI MMIO accessors.
If Peter is fine with two variants for x86 and non-x86 architectures. I can post (v4) for further review. That will be something like below and let me know if there are any other improvements are needed.
----- include/system/memory.h
+/**
+ * qemu_ram_copy: copy data to ramblock
+ *
+ * @dst: destination where the data is copied to
+ * @src: source where the data is copied from
+ * @n: length of data to be copied
+ *
+ * Copy @n bytes from @src to @dst with the assumption that @src and @dst
+ * do not overlap. For x86, the fixed sized accesses like 1/2/4/8 bytes are
+ * handled by __builtin_memcpy() no matter whether @src or @dst is aligned
+ * to the access size, and fall back to memcpy() for other access sizes. For
+ * other architectures, data is copied from @src to @dst in units of 1/2/4/8
+ * bytes, ensuring proper alignment in each unit and order among the units.
+ */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+
+/**
+ * qemu_ram_move: move data to ramblock
+ *
+ * @dst: destination where the data is moved to
+ * @src: source where the data is moved from
+ * @n: length of data to be moved
+ *
+ * Move @n bytes from @src to @dst with the assumption that @src and @dst
+ * can overlap. For x86, the fixed sized accesses like 1/2/4/8 bytes are
+ * handled by __builtin_memmove() no matter whether @src or @dst is aligned
+ * to the access size, and fall back to memmove() for other access sizes.
+ * For other architectures, data is moved from @src to @dst in units of
+ * 1/2/4/8 bytes, ensuring proper alignment in each unit and order among
+ * the units.
+ */
+void qemu_ram_move(void *dest, const void *src, size_t n);
+
----- system/physmem.c
+static void qemu_ram_unsafe_copy(void *dst, const void *src, size_t n) +{
+ switch (n) {
+ case 1:
+ __builtin_memcpy(dst, src, 1);
+ break;
+ case 2:
+ __builtin_memcpy(dst, src, 2);
+ break;
+ case 4:
+ __builtin_memcpy(dst, src, 4);
+ break;
+ case 8:
+ __builtin_memcpy(dst, src, 8);
+ break;
+ default:
+ memcpy(dst, src, n);
+ }
+}
+
+static void qemu_ram_unsafe_move(void *dst, const void *src, size_t n) +{
+ switch (n) {
+ case 1:
+ __builtin_memmove(dst, src, 1);
+ break;
+ case 2:
+ __builtin_memmove(dst, src, 2);
+ break;
+ case 4:
+ __builtin_memmove(dst, src, 4);
+ break;
+ case 8:
+ __builtin_memmove(dst, src, 8);
+ break;
+ default:
+ memmove(dst, src, n);
+ }
+}
+
+static void qemu_ram_safe_copy(void *dst, const void *src,
+ size_t n, size_t max_step)
+{
+ uintptr_t test, step;
+
+ /* Aligned maximal step */
+ max_step = pow2floor(max_step);
+
+ while (n) {
+ test = (uintptr_t)src | (uintptr_t)dst | n | max_step;
+ step = test & -test;
+
+ switch (step) {
+ case 1:
+ qatomic_set((uint8_t *)dst, qatomic_read((uint8_t *)src)); + src += 1; + dst += 1; + n -= 1; + break; + case 2: + qatomic_set((uint16_t *)dst, qatomic_read((uint16_t *)src)); + src += 2; + dst += 2; + n -= 2; + break; + case 4: + qatomic_set((uint32_t *)dst, qatomic_read((uint32_t *)src)); + src += 4; + dst += 4; + n -= 4; + break; + case 8: + default: + qatomic_set((uint64_t *)dst, qatomic_read((uint64_t *)src)); + src += 8; + dst += 8; + n -= 8;
+ }
+ }
+}
+
+static void qemu_ram_safe_copy_backwards(void *dst, const void *src,
+ size_t n, size_t max_step)
+{
+ uintptr_t test, step;
+
+ /* Aligned maximal step */
+ max_step = pow2floor(max_step);
+
+ /* End of the blocks */ + src += n; + dst += n;
+
+ while (n) {
+ test = (uintptr_t)src | (uintptr_t)dst | n | max_step;
+ step = test & -test;
+
+ switch (step) {
+ case 1:
+ src -= 1; + dst -= 1; + n -= 1; + qatomic_set((uint8_t *)dst, qatomic_read((uint8_t *)src)); + break; + case 2: + src -= 2; + dst -= 2; + n -= 2; + qatomic_set((uint16_t *)dst, qatomic_read((uint16_t *)src)); + break; + case 4: + src -= 4; + dst -= 4; + n -= 4; + qatomic_set((uint32_t *)dst, qatomic_read((uint32_t *)src)); + break; + case 8: + default: + src -= 8; + dst -= 8; + n -= 8; + qatomic_set((uint64_t *)dst, qatomic_read((uint64_t *)src));
+ }
+ }
+}
+
+/* x86 should work with __builtin_{memcpy, memmove}() for IO access */ +#if defined(__i386__) || defined(__x86_64__)
+#define HOST_UNALIGNED_VECTOR_MMIO_OK 1
+#else
+#define HOST_UNALIGNED_VECTOR_MMIO_OK 0
+#endif
+
+void qemu_ram_copy(void *dst, const void *src, size_t n)
+{
+ if (src == dst || n == 0) {
+ return;
+ }
+
+ if (HOST_UNALIGNED_VECTOR_MMIO_OK) {
+ qemu_ram_unsafe_copy(dst, src, n);
+ } else {
+ qemu_ram_safe_copy(dst, src, n, 8);
+ }
+}
+void qemu_ram_move(void *dst, const void *src, size_t n)
+{
+ if (src == dst || n == 0) {
+ return;
+ }
+
+ if (HOST_UNALIGNED_VECTOR_MMIO_OK) {
+ qemu_ram_unsafe_move(dst, src, n);
+ } else if (dst < src) {
+ qemu_ram_safe_copy(dst, src, n, src - dst);
+ } else {
+ qemu_ram_safe_copy_backwards(dst, src, n, dst - src);
+ }
+}
Thanks,
Gavin
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 3 lines · Gavin Shan
If Peter is fine with two variants for x86 and non-x86 architectures. I can post (v4) for further review. That will be something like below and let me know if there are any other improvements are needed.
I have a generic question on the "unaligned access for x86": I think the question is about the one Michael raised here on unaligned access may break x86 here:
https://lore.kernel.org/qemu-devel/[email protected]/
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
Is that a real problem we need to consider, or can we start with unified approach and leave it for later?
PS: I apologize if I missed important piece of info along the way; I didn't follow closely on the discussion on this topic in the past few weeks.
One thing to mention is, what we change should only need to affect ram_device, AFAIU.. so most memcpy()/memmove() shouldn't be changed for any arch when it's pure RAM.
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 9 lines · Peter Xu
Quoted 3 lines · Gavin Shan
If Peter is fine with two variants for x86 and non-x86 architectures. I can post (v4) for further review. That will be something like below and let me know if there are any other improvements are needed.
I have a generic question on the "unaligned access for x86": I think the question is about the one Michael raised here on unaligned access may break x86 here:
Quoted 9 lines
https://lore.kernel.org/qemu-devel/[email protected]/
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
Is that a real problem we need to consider, or can we start with unified approach and leave it for later?
I'm leaving this question to Michael.
Quoted 7 lines
PS: I apologize if I missed important piece of info along the way; I didn't follow closely on the discussion on this topic in the past few weeks.
One thing to mention is, what we change should only need to affect
ram_device, AFAIU.. so most memcpy()/memmove() shouldn't be changed for any arch when it's pure RAM.
It depends. This patch intends to fix issue [1] in the lower layer by using the newly added accessors (qemu_ram_{copy, move}) on all directly accessible regions including the regular (pure) RAM region. Otherwise, the newly added accessors should be limited to ram_device regions only as you said.
[1] https://lore.kernel.org/qemu-devel/[email protected]/
Thanks,
Gavin
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 16 lines · Gavin Shan
Quoted 15 lines · Peter Xu
Quoted 3 lines · Gavin Shan
If Peter is fine with two variants for x86 and non-x86 architectures. I can post (v4) for further review. That will be something like below and let me know if there are any other improvements are needed.
I have a generic question on the "unaligned access for x86": I think the question is about the one Michael raised here on unaligned access may break x86 here:
https://lore.kernel.org/qemu-devel/[email protected]/
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
Quoted 5 lines
Quoted 3 lines
Is that a real problem we need to consider, or can we start with unified approach and leave it for later?
I'm leaving this question to Michael.
Knowing what I know about hardware designers, it's something someone somewhere does)
It can be made a separate patch, just to show - it should be all of ~10LOC.
Quoted 13 lines
Quoted 7 lines
PS: I apologize if I missed important piece of info along the way; I didn't follow closely on the discussion on this topic in the past few weeks.
One thing to mention is, what we change should only need to affect
ram_device, AFAIU.. so most memcpy()/memmove() shouldn't be changed for any arch when it's pure RAM.It depends. This patch intends to fix issue [1] in the lower layer by using the newly added accessors (qemu_ram_{copy, move}) on all directly accessible regions including the regular (pure) RAM region. Otherwise, the newly added accessors should be limited to
ram_deviceregions only as you said.
Quoted 4 lines
[1] https://lore.kernel.org/qemu-devel/[email protected]/
Thanks,
Gavin
using memcpy()/memmove() to emulate guest's atomics is generally kinda broken.
but yes there are architectures where doing it to device ram is more broken than doing it to regular ram.
Signature · 1 line
MST
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 21 lines · Michael S. Tsirkin
Quoted 20 lines · Gavin Shan
Quoted 18 lines · Peter Xu
Quoted 3 lines · Gavin Shan
If Peter is fine with two variants for x86 and non-x86 architectures. I can post (v4) for further review. That will be something like below and let me know if there are any other improvements are needed.
I have a generic question on the "unaligned access for x86": I think the question is about the one Michael raised here on unaligned access may break x86 here:
https://lore.kernel.org/qemu-devel/[email protected]/
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
Is that a real problem we need to consider, or can we start with unified approach and leave it for later?
Quoted 6 lines
Quoted 1 line
I'm leaving this question to Michael.
Knowing what I know about hardware designers, it's something someone somewhere does)
It can be made a separate patch, just to show - it should be all of ~10LOC.
It's only about removal of anything that might be controversial for now, thanks. I also wonder if anything would break, then it's more solid proof that per-arch change is required.
Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
PS: I apologize if I missed important piece of info along the way; I didn't follow closely on the discussion on this topic in the past few weeks.
One thing to mention is, what we change should only need to affect
ram_device, AFAIU.. so most memcpy()/memmove() shouldn't be changed for any arch when it's pure RAM.It depends. This patch intends to fix issue [1] in the lower layer by using the newly added accessors (qemu_ram_{copy, move}) on all directly accessible regions including the regular (pure) RAM region. Otherwise, the newly added accessors should be limited to
ram_deviceregions only as you said.
Quoted 7 lines
Quoted 2 lines
Thanks,
Gavinusing memcpy()/memmove() to emulate guest's atomics is generally kinda broken.
but yes there are architectures where doing it to device ram is more broken than doing it to regular ram.
If we keep memcpy()/memmove() for len>8 (aligned or not), I am thinking no perf issue will happen, then looks like we can indeed change this even for pure RAM operations, which we can't identify in case of e1000e driver use case.
Then does it mean we should not use __builtin_memcpy()/memmove()? Even if we know constants 1/2/4/8 would work there, why not we go ahead and use qatomics, which is even more future proof? I also stumbled on top of commit 77b1757090 ("include/qemu/bswap.h: Use __builtin_memcpy() in accessor functions"), which seems to say the same thing "for the long term".
For "should we still do unaligned access if the guest did it, so as to keep the original behavior of a bare metal" question, are we on the same page that we should just break it into aligned accesses for all archs? I think it means, we will not be able to emulate guest faults correctly as what will happen on bare metal, but we're missing the fault injection logics anyway, so whenever it's implemented we _could_ switch it back to unaligned accesses. Before that, breaking unaligned seems like a better way to go to
(1) satisfy all legit users, and (2) don't make QEMU crash by guest operations.
Last, one silly question: why do we need any helper named with *memcpy*, if memmove is always superior (to consider range overlap)? Can we stick with memmove all over the places?
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 24 lines · Peter Xu
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Gavin Shan
Quoted 9 lines · Peter Xu
Quoted 3 lines · Gavin Shan
If Peter is fine with two variants for x86 and non-x86 architectures. I can post (v4) for further review. That will be something like below and let me know if there are any other improvements are needed.
I have a generic question on the "unaligned access for x86": I think the question is about the one Michael raised here on unaligned access may break x86 here:
Quoted 9 lines
https://lore.kernel.org/qemu-devel/[email protected]/
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
Is that a real problem we need to consider, or can we start with unified approach and leave it for later?
I'm leaving this question to Michael.
Quoted 8 lines
Quoted 4 lines
Knowing what I know about hardware designers, it's something someone somewhere does)
It can be made a separate patch, just to show - it should be all of ~10LOC.It's only about removal of anything that might be controversial for now, thanks. I also wonder if anything would break, then it's more solid proof that per-arch change is required.
Repeating:
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset 0x1 within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.
It's easy to find more examples of such hardware if one looks. For example, LEDCTL on e1000e:
A claim that no software uses this hardware capability is the strong claim that needs proof, not the other way around.
What to do on non-x86? reading a dword might work, or reading byte by byte might work. Both can cause issues, just different ones, but given we did byte by byte previously i guess let's keep doing that.
Quoted 54 lines
Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
PS: I apologize if I missed important piece of info along the way; I didn't follow closely on the discussion on this topic in the past few weeks.
One thing to mention is, what we change should only need to affect
ram_device, AFAIU.. so most memcpy()/memmove() shouldn't be changed for any arch when it's pure RAM.It depends. This patch intends to fix issue [1] in the lower layer by using the newly added accessors (qemu_ram_{copy, move}) on all directly accessible regions including the regular (pure) RAM region. Otherwise, the newly added accessors should be limited to
ram_deviceregions only as you said.Quoted 7 lines
Quoted 2 lines
Thanks,
Gavinusing memcpy()/memmove() to emulate guest's atomics is generally kinda broken.
but yes there are architectures where doing it to device ram is more broken than doing it to regular ram.If we keep memcpy()/memmove() for len>8 (aligned or not), I am thinking no perf issue will happen, then looks like we can indeed change this even for pure RAM operations, which we can't identify in case of e1000e driver use case.
Then does it mean we should not use __builtin_memcpy()/memmove()? Even if we know constants 1/2/4/8 would work there, why not we go ahead and use qatomics, which is even more future proof? I also stumbled on top of commit
77b1757090("include/qemu/bswap.h: Use__builtin_memcpy()in accessor functions"), which seems to say the same thing "for the long term".For "should we still do unaligned access if the guest did it, so as to keep the original behavior of a bare metal" question, are we on the same page that we should just break it into aligned accesses for all archs? I think it means, we will not be able to emulate guest faults correctly as what will happen on bare metal, but we're missing the fault injection logics anyway, so whenever it's implemented we _could_ switch it back to unaligned accesses. Before that, breaking unaligned seems like a better way to go to
(1) satisfy all legit users, and (2) don't make QEMU crash by guest operations.Last, one silly question: why do we need any helper named with *memcpy*, if memmove is always superior (to consider range overlap)? Can we stick with memmove all over the places?
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 34 lines · Michael S. Tsirkin
Quoted 24 lines · Peter Xu
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Gavin Shan
Quoted 9 lines · Peter Xu
Quoted 3 lines · Gavin Shan
If Peter is fine with two variants for x86 and non-x86 architectures. I can post (v4) for further review. That will be something like below and let me know if there are any other improvements are needed.
I have a generic question on the "unaligned access for x86": I think the question is about the one Michael raised here on unaligned access may break x86 here:
Quoted 9 lines
https://lore.kernel.org/qemu-devel/[email protected]/
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
Is that a real problem we need to consider, or can we start with unified approach and leave it for later?
I'm leaving this question to Michael.
Quoted 8 lines
Quoted 4 lines
Knowing what I know about hardware designers, it's something someone somewhere does)
It can be made a separate patch, just to show - it should be all of ~10LOC.It's only about removal of anything that might be controversial for now, thanks. I also wonder if anything would break, then it's more solid proof that per-arch change is required.
Quoted 5 lines
Repeating:
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.
Isn't this out of the CPU arch, dealt with at the bus level?
It looks we try to be clever with modern PCI code by optimizing this access -- not saying we can change that, I know it is too late after 20+ years -- relying on hw behavior that was done that way to support legacy hw, in particular broken I/O accesses.
For Alpha / HPPA / MIPS there were ASIC in the PCI I/O path to handle these odd unaligned accesses inherited from x86 world.
Quoted 72 lines
It's easy to find more examples of such hardware if one looks. For example, LEDCTL on e1000e:
A claim that no software uses this hardware capability is the strong claim that needs proof, not the other way around.
What to do on non-x86? reading a dword might work, or reading byte by byte might work. Both can cause issues, just different ones, but given we did byte by byte previously i guess let's keep doing that.
Quoted 54 lines
Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
PS: I apologize if I missed important piece of info along the way; I didn't follow closely on the discussion on this topic in the past few weeks.
One thing to mention is, what we change should only need to affect
ram_device, AFAIU.. so most memcpy()/memmove() shouldn't be changed for any arch when it's pure RAM.It depends. This patch intends to fix issue [1] in the lower layer by using the newly added accessors (qemu_ram_{copy, move}) on all directly accessible regions including the regular (pure) RAM region. Otherwise, the newly added accessors should be limited to
ram_deviceregions only as you said.Quoted 7 lines
Quoted 2 lines
Thanks,
Gavinusing memcpy()/memmove() to emulate guest's atomics is generally kinda broken.
but yes there are architectures where doing it to device ram is more broken than doing it to regular ram.If we keep memcpy()/memmove() for len>8 (aligned or not), I am thinking no perf issue will happen, then looks like we can indeed change this even for pure RAM operations, which we can't identify in case of e1000e driver use case.
Then does it mean we should not use __builtin_memcpy()/memmove()? Even if we know constants 1/2/4/8 would work there, why not we go ahead and use qatomics, which is even more future proof? I also stumbled on top of commit
77b1757090("include/qemu/bswap.h: Use__builtin_memcpy()in accessor functions"), which seems to say the same thing "for the long term".For "should we still do unaligned access if the guest did it, so as to keep the original behavior of a bare metal" question, are we on the same page that we should just break it into aligned accesses for all archs? I think it means, we will not be able to emulate guest faults correctly as what will happen on bare metal, but we're missing the fault injection logics anyway, so whenever it's implemented we _could_ switch it back to unaligned accesses. Before that, breaking unaligned seems like a better way to go to
(1) satisfy all legit users, and (2) don't make QEMU crash by guest operations.Last, one silly question: why do we need any helper named with *memcpy*, if memmove is always superior (to consider range overlap)? Can we stick with memmove all over the places?
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 41 lines · Philippe Mathieu-Daudé
Quoted 34 lines · Michael S. Tsirkin
Quoted 24 lines · Peter Xu
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Gavin Shan
Quoted 9 lines · Peter Xu
Quoted 3 lines · Gavin Shan
If Peter is fine with two variants for x86 and non-x86 architectures. I can post (v4) for further review. That will be something like below and let me know if there are any other improvements are needed.
I have a generic question on the "unaligned access for x86": I think the question is about the one Michael raised here on unaligned access may break x86 here:
Quoted 9 lines
https://lore.kernel.org/qemu-devel/[email protected]/
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
Is that a real problem we need to consider, or can we start with unified approach and leave it for later?
I'm leaving this question to Michael.
Quoted 8 lines
Quoted 4 lines
Knowing what I know about hardware designers, it's something someone somewhere does)
It can be made a separate patch, just to show - it should be all of ~10LOC.It's only about removal of anything that might be controversial for now, thanks. I also wonder if anything would break, then it's more solid proof that per-arch change is required.
Quoted 5 lines
Repeating:
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.
Quoted 6 lines
Isn't this out of the CPU arch, dealt with at the bus level?
It looks we try to be clever with modern PCI code by optimizing this access -- not saying we can change that, I know it is too late after 20+ years -- relying on hw behavior that was done that way to support legacy hw, in particular broken I/O accesses.
Not sure what the question is. We were dicussing how to emulate unaligned accesses from x86 guests if they happen.
On an x86 host we can do that easily, and it's just a couple of LOC. Though Peter Maydell dislikes host arch specific code. But I hope if it's a separate patch on top and it is visible how small it is, he will reconsider)
On other hosts we can't emulate them 100%, we either need to split to byte accesses or over-access and mask. Byte accesses feel safer. What qemu currently does with memmove is clearly not safe in the general case.
As for optimizing - there is space for optimization e.g. vfio could report the properties of a BAR to userspace for optimization purposes. On x86 you can then get good speed with just memmove. Other arches you will likely need to write arch specific code. But this has to do with e.g. DMA into device BAR not CPU accesses. As an aside, it is a pity qemu uses same thing for DMA and CPU access, we know how virtio accesses work and this might allow optimization.
Quoted 3 lines
For Alpha / HPPA / MIPS there were ASIC in the PCI I/O path to handle these odd unaligned accesses inherited from x86 world.
Right.
Quoted 72 lines
Quoted 72 lines
It's easy to find more examples of such hardware if one looks. For example, LEDCTL on e1000e:
A claim that no software uses this hardware capability is the strong claim that needs proof, not the other way around.
What to do on non-x86? reading a dword might work, or reading byte by byte might work. Both can cause issues, just different ones, but given we did byte by byte previously i guess let's keep doing that.
Quoted 54 lines
Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
PS: I apologize if I missed important piece of info along the way; I didn't follow closely on the discussion on this topic in the past few weeks.
One thing to mention is, what we change should only need to affect
ram_device, AFAIU.. so most memcpy()/memmove() shouldn't be changed for any arch when it's pure RAM.It depends. This patch intends to fix issue [1] in the lower layer by using the newly added accessors (qemu_ram_{copy, move}) on all directly accessible regions including the regular (pure) RAM region. Otherwise, the newly added accessors should be limited to
ram_deviceregions only as you said.Quoted 7 lines
Quoted 2 lines
Thanks,
Gavinusing memcpy()/memmove() to emulate guest's atomics is generally kinda broken.
but yes there are architectures where doing it to device ram is more broken than doing it to regular ram.If we keep memcpy()/memmove() for len>8 (aligned or not), I am thinking no perf issue will happen, then looks like we can indeed change this even for pure RAM operations, which we can't identify in case of e1000e driver use case.
Then does it mean we should not use __builtin_memcpy()/memmove()? Even if we know constants 1/2/4/8 would work there, why not we go ahead and use qatomics, which is even more future proof? I also stumbled on top of commit
77b1757090("include/qemu/bswap.h: Use__builtin_memcpy()in accessor functions"), which seems to say the same thing "for the long term".For "should we still do unaligned access if the guest did it, so as to keep the original behavior of a bare metal" question, are we on the same page that we should just break it into aligned accesses for all archs? I think it means, we will not be able to emulate guest faults correctly as what will happen on bare metal, but we're missing the fault injection logics anyway, so whenever it's implemented we _could_ switch it back to unaligned accesses. Before that, breaking unaligned seems like a better way to go to
(1) satisfy all legit users, and (2) don't make QEMU crash by guest operations.Last, one silly question: why do we need any helper named with *memcpy*, if memmove is always superior (to consider range overlap)? Can we stick with memmove all over the places?
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 49 lines · Michael S. Tsirkin
Quoted 41 lines · Philippe Mathieu-Daudé
Quoted 34 lines · Michael S. Tsirkin
Quoted 24 lines · Peter Xu
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Gavin Shan
Quoted 9 lines · Peter Xu
Quoted 3 lines · Gavin Shan
If Peter is fine with two variants for x86 and non-x86 architectures. I can post (v4) for further review. That will be something like below and let me know if there are any other improvements are needed.
I have a generic question on the "unaligned access for x86": I think the question is about the one Michael raised here on unaligned access may break x86 here:
Quoted 9 lines
https://lore.kernel.org/qemu-devel/[email protected]/
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
Is that a real problem we need to consider, or can we start with unified approach and leave it for later?
I'm leaving this question to Michael.
Quoted 8 lines
Quoted 4 lines
Knowing what I know about hardware designers, it's something someone somewhere does)
It can be made a separate patch, just to show - it should be all of ~10LOC.It's only about removal of anything that might be controversial for now, thanks. I also wonder if anything would break, then it's more solid proof that per-arch change is required.
Quoted 5 lines
Repeating:
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.Quoted 6 lines
Isn't this out of the CPU arch, dealt with at the bus level?
It looks we try to be clever with modern PCI code by optimizing this access -- not saying we can change that, I know it is too late after 20+ years -- relying on hw behavior that was done that way to support legacy hw, in particular broken I/O accesses.
Quoted 6 lines
Not sure what the question is. We were dicussing how to emulate unaligned accesses from x86 guests if they happen.
On an x86 host we can do that easily, and it's just a couple of LOC. Though Peter Maydell dislikes host arch specific code. But I hope if it's a separate patch on top and it is visible how small it is, he will reconsider)
Yes, if we still want x86 specific change, it would be better to be put separately.
Said so, I don't think the e1000e LEDCTL test illustrated what might break.. Isn't that only an exmaple showing unaligned access is "supported", however nothing breaks even if we use 1B*2?
My question was more about a real breakage, hence whenever it happened "it's more solid proof that per-arch change is required".
Quoted 4 lines
On other hosts we can't emulate them 100%, we either need to split to byte accesses or over-access and mask. Byte accesses feel safer. What qemu currently does with memmove is clearly not safe in the general case.
We should have another option that is not arch-dependent but keep the unaligned behavior.
For current master, AFAIU we do unaligned access for both ram_device and rest. Say, even with ram_device_mem_ops, it has both .unaligned=true for both .valid & .impl. I think it means indeed we have unaligned behavior even for ram_device. It also means what matters in regards to the Realtek bug was only about aligned access (with subpage presence).
I think it means we can always keep unaligned to stick with memcpy()/memmove(), but only use atomic ops for the aligned cases of 1/2/4/8. With that, I think we can also remove ram_device_mem_ops and fix the bounce buffer issue.
I think it means we'll stick with memcpy()/memmove() for all archs for unaligned, which is again not safe... but that can be an existing but separate problem to solve too.
One more thing to mention below...
Quoted 12 lines
As for optimizing - there is space for optimization e.g. vfio could report the properties of a BAR to userspace for optimization purposes. On x86 you can then get good speed with just memmove. Other arches you will likely need to write arch specific code. But this has to do with e.g. DMA into device BAR not CPU accesses. As an aside, it is a pity qemu uses same thing for DMA and CPU access, we know how virtio accesses work and this might allow optimization.
Quoted 1 line
Quoted 44 lines
Quoted 2 lines
For Alpha / HPPA / MIPS there were ASIC in the PCI I/O path to handle these odd unaligned accesses inherited from x86 world.
Right.
Quoted 39 lines
Quoted 39 lines
It's easy to find more examples of such hardware if one looks. For example, LEDCTL on e1000e:
A claim that no software uses this hardware capability is the strong claim that needs proof, not the other way around.
What to do on non-x86? reading a dword might work, or reading byte by byte might work. Both can cause issues, just different ones, but given we did byte by byte previously i guess let's keep doing that.
Quoted 23 lines
Quoted 23 lines
Quoted 14 lines
Quoted 8 lines
PS: I apologize if I missed important piece of info along the way; I didn't follow closely on the discussion on this topic in the past few weeks.
One thing to mention is, what we change should only need to affect
ram_device, AFAIU.. so most memcpy()/memmove() shouldn't be changed for any arch when it's pure RAM.It depends. This patch intends to fix issue [1] in the lower layer by using the newly added accessors (qemu_ram_{copy, move}) on all directly accessible regions including the regular (pure) RAM region. Otherwise, the newly added accessors should be limited to
ram_deviceregions only as you said.Quoted 4 lines
[1] https://lore.kernel.org/qemu-devel/[email protected]/
Thanks,
Gavinusing memcpy()/memmove() to emulate guest's atomics is generally kinda broken.
but yes there are architectures where doing it to device ram is more broken than doing it to regular ram.
.. please someone (at least.. Gavin?) still have a look at below. It could matter how the next version looks, especially I want to make sure if we should get rid of __builtin_*() in the new helpers.
Thanks,
Quoted 34 lines
Quoted 33 lines
Quoted 33 lines
Quoted 31 lines
If we keep memcpy()/memmove() for len>8 (aligned or not), I am thinking no perf issue will happen, then looks like we can indeed change this even for pure RAM operations, which we can't identify in case of e1000e driver use case.
Then does it mean we should not use __builtin_memcpy()/memmove()? Even if we know constants 1/2/4/8 would work there, why not we go ahead and use qatomics, which is even more future proof? I also stumbled on top of commit
77b1757090("include/qemu/bswap.h: Use__builtin_memcpy()in accessor functions"), which seems to say the same thing "for the long term".For "should we still do unaligned access if the guest did it, so as to keep the original behavior of a bare metal" question, are we on the same page that we should just break it into aligned accesses for all archs? I think it means, we will not be able to emulate guest faults correctly as what will happen on bare metal, but we're missing the fault injection logics anyway, so whenever it's implemented we _could_ switch it back to unaligned accesses. Before that, breaking unaligned seems like a better way to go to
(1) satisfy all legit users, and (2) don't make QEMU crash by guest operations.Last, one silly question: why do we need any helper named with *memcpy*, if memmove is always superior (to consider range overlap)? Can we stick with memmove all over the places?
Thanks,
Signature · 1 line
Peter Xu
Signature · 1 line
Peter Xu
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 81 lines · Peter Xu
Quoted 49 lines · Michael S. Tsirkin
Quoted 41 lines · Philippe Mathieu-Daudé
Quoted 34 lines · Michael S. Tsirkin
Quoted 24 lines · Peter Xu
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Gavin Shan
Quoted 9 lines · Peter Xu
Quoted 3 lines · Gavin Shan
If Peter is fine with two variants for x86 and non-x86 architectures. I can post (v4) for further review. That will be something like below and let me know if there are any other improvements are needed.
I have a generic question on the "unaligned access for x86": I think the question is about the one Michael raised here on unaligned access may break x86 here:
Quoted 9 lines
https://lore.kernel.org/qemu-devel/[email protected]/
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
Is that a real problem we need to consider, or can we start with unified approach and leave it for later?
I'm leaving this question to Michael.
Quoted 8 lines
Quoted 4 lines
Knowing what I know about hardware designers, it's something someone somewhere does)
It can be made a separate patch, just to show - it should be all of ~10LOC.It's only about removal of anything that might be controversial for now, thanks. I also wonder if anything would break, then it's more solid proof that per-arch change is required.
Quoted 5 lines
Repeating:
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.Quoted 6 lines
Isn't this out of the CPU arch, dealt with at the bus level?
It looks we try to be clever with modern PCI code by optimizing this access -- not saying we can change that, I know it is too late after 20+ years -- relying on hw behavior that was done that way to support legacy hw, in particular broken I/O accesses.
Quoted 6 lines
Not sure what the question is. We were dicussing how to emulate unaligned accesses from x86 guests if they happen.
On an x86 host we can do that easily, and it's just a couple of LOC. Though Peter Maydell dislikes host arch specific code. But I hope if it's a separate patch on top and it is visible how small it is, he will reconsider)Yes, if we still want x86 specific change, it would be better to be put separately.
Said so, I don't think the e1000e LEDCTL test illustrated what might break.. Isn't that only an exmaple showing unaligned access is "supported", however nothing breaks even if we use 1B*2?
My question was more about a real breakage, hence whenever it happened "it's more solid proof that per-arch change is required".
Quoted 4 lines
On other hosts we can't emulate them 100%, we either need to split to byte accesses or over-access and mask. Byte accesses feel safer. What qemu currently does with memmove is clearly not safe in the general case.
We should have another option that is not arch-dependent but keep the unaligned behavior.
For current master, AFAIU we do unaligned access for both
ram_deviceand rest. Say, even withram_device_mem_ops, it has both .unaligned=true for both .valid & .impl. I think it means indeed we have unaligned behavior even forram_device. It also means what matters in regards to the Realtek bug was only about aligned access (with subpage presence).
Quoted 9 lines
I think it means we can always keep unaligned to stick with memcpy()/memmove(), but only use atomic ops for the aligned cases of 1/2/4/8. With that, I think we can also remove
ram_device_mem_opsand fix the bounce buffer issue.I think it means we'll stick with memcpy()/memmove() for all archs for unaligned, which is again not safe... but that can be an existing but separate problem to solve too.
Lets see if Peter Maydell and Michael are happy with this option. At least, we will have unified qemu_ram_move() for all architectures with this option. Note that qemu_ram_copy() won't be needed.
I'm putting note on what's to be done in (v4) if this option is to be picked up. Let me know if there are missed points. It's basically combing what's done by ram_device_mem_ops to upper layer (e.g. in qemu_ram_move()).
address_space_write
address_space_to_flatview
flatview_write
flatview_translate
flatview_write_continue
flatview_write_continue_step
memmove // (A) to replace it with qemu_ram_move()
/**
* qemu_ram_move: move data to ramblock
*
* @dst: destination where the data is moved to
* @src: source where the data is moved from
* @n: length of data to be moved
*
* Move @n bytes from @src to @dst with the assumption that @src and @dst
* can overlap. The access is atomic if the source and destination buffer
* aren't overlapped for a well aligned and small-sized access. Otherwise,
* fall back to the standard memmove().
*/
static void qemu_ram_move(void *dst, const void *src, size_t n)
{
uintptr_t test, len;
if (src == dst || n == 0) {
return;
}
/* Overlapped buffers */
if (src < (dst + n) && dst < (src + n)) {
memmove(dst, src, n);
return;
}
test = (uintptr_t)src | (uintptr_t)dst | n;
len = test & -test;
/* Unaligned or oversized access */
if (n > 8 || len != n) {
memmove(dst, src, n);
return;
}
switch (len) {
case 1:
qatomic_set((uint8_t *)dst, qatomic_read((uint8_t *)src));
break;
case 2:
qatomic_set((uint16_t *)dst, qatomic_read((uint16_t *)src));
break;
case 4:
qatomic_set((uint32_t *)dst, qatomic_read((uint32_t *)src));
break;
case 8:
qatomic_set((uint64_t *)dst, qatomic_read((uint64_t *)src));
break;
default:
g_assert_not_reached();
}
}Quoted 2 lines
One more thing to mention below...
Please see below responses.
Quoted 57 lines
Quoted 12 lines
As for optimizing - there is space for optimization e.g. vfio could report the properties of a BAR to userspace for optimization purposes. On x86 you can then get good speed with just memmove. Other arches you will likely need to write arch specific code. But this has to do with e.g. DMA into device BAR not CPU accesses. As an aside, it is a pity qemu uses same thing for DMA and CPU access, we know how virtio accesses work and this might allow optimization.
Quoted 1 line
Quoted 44 lines
Quoted 2 lines
For Alpha / HPPA / MIPS there were ASIC in the PCI I/O path to handle these odd unaligned accesses inherited from x86 world.
Right.
Quoted 39 lines
Quoted 39 lines
It's easy to find more examples of such hardware if one looks. For example, LEDCTL on e1000e:
A claim that no software uses this hardware capability is the strong claim that needs proof, not the other way around.
What to do on non-x86? reading a dword might work, or reading byte by byte might work. Both can cause issues, just different ones, but given we did byte by byte previously i guess let's keep doing that.
Quoted 23 lines
Quoted 23 lines
Quoted 14 lines
Quoted 8 lines
PS: I apologize if I missed important piece of info along the way; I didn't follow closely on the discussion on this topic in the past few weeks.
One thing to mention is, what we change should only need to affect
ram_device, AFAIU.. so most memcpy()/memmove() shouldn't be changed for any arch when it's pure RAM.It depends. This patch intends to fix issue [1] in the lower layer by using the newly added accessors (qemu_ram_{copy, move}) on all directly accessible regions including the regular (pure) RAM region. Otherwise, the newly added accessors should be limited to
ram_deviceregions only as you said.Quoted 4 lines
[1] https://lore.kernel.org/qemu-devel/[email protected]/
Thanks,
Gavinusing memcpy()/memmove() to emulate guest's atomics is generally kinda broken.
but yes there are architectures where doing it to device ram is more broken than doing it to regular ram.
Quoted 19 lines
.. please someone (at least.. Gavin?) still have a look at below. It could matter how the next version looks, especially I want to make sure if we should get rid of __builtin_*() in the new helpers.
Thanks,
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
If we keep memcpy()/memmove() for len>8 (aligned or not), I am thinking no perf issue will happen, then looks like we can indeed change this even for pure RAM operations, which we can't identify in case of e1000e driver use case.
Then does it mean we should not use __builtin_memcpy()/memmove()? Even if we know constants 1/2/4/8 would work there, why not we go ahead and use qatomics, which is even more future proof? I also stumbled on top of commit
77b1757090("include/qemu/bswap.h: Use__builtin_memcpy()in accessor functions"), which seems to say the same thing "for the long term".
Yes, I think qatomics are more future proof. For the aligned accesses in above qemu_ram_copy(), I think qatomics instead of __built_in_memmove() should be used.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
For "should we still do unaligned access if the guest did it, so as to keep the original behavior of a bare metal" question, are we on the same page that we should just break it into aligned accesses for all archs? I think it means, we will not be able to emulate guest faults correctly as what will happen on bare metal, but we're missing the fault injection logics anyway, so whenever it's implemented we _could_ switch it back to unaligned accesses. Before that, breaking unaligned seems like a better way to go to
(1) satisfy all legit users, and (2) don't make QEMU crash by guest operations.
Not really. For our specific case where the ram device region is turned to directly accessible, it's fine not breaking down the unaligned access to aligned accesses, which is consistent with the expected behavior from the existing ram_device_mem_ops. An unaligned access is allowed by ram_device_mem_ops. So we can also allow an unaligned access in qemu_ram_move().
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Last, one silly question: why do we need any helper named with *memcpy*, if memmove is always superior (to consider range overlap)? Can we stick with memmove all over the places?
Yes, I think one preparatory patch can added in (v4) to replace memcpy() with memmove() in the following paths, extending commit 4a73aee8814 ("softmmu: Use memmove in flatview_write_continue"). With this replacement, qemu_ram_copy() won't be needed in (v4).
hw/remote/vfio-user-obj.c::vfu_object_mr_rw
include/system/memory.h::address_space_read
system/physmem.c::flatview_read_continue_step
system/physmem.c::address_space_write_rom
Thanks,
Gavin
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 121 lines · Gavin Shan
Quoted 81 lines · Peter Xu
Quoted 49 lines · Michael S. Tsirkin
Quoted 41 lines · Philippe Mathieu-Daudé
Quoted 34 lines · Michael S. Tsirkin
Quoted 24 lines · Peter Xu
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Gavin Shan
Quoted 9 lines · Peter Xu
Quoted 3 lines · Gavin Shan
If Peter is fine with two variants for x86 and non-x86 architectures. I can post (v4) for further review. That will be something like below and let me know if there are any other improvements are needed.
I have a generic question on the "unaligned access for x86": I think the question is about the one Michael raised here on unaligned access may break x86 here:
Quoted 9 lines
https://lore.kernel.org/qemu-devel/[email protected]/
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
Is that a real problem we need to consider, or can we start with unified approach and leave it for later?
I'm leaving this question to Michael.
Quoted 8 lines
Quoted 4 lines
Knowing what I know about hardware designers, it's something someone somewhere does)
It can be made a separate patch, just to show - it should be all of ~10LOC.It's only about removal of anything that might be controversial for now, thanks. I also wonder if anything would break, then it's more solid proof that per-arch change is required.
Quoted 5 lines
Repeating:
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.Quoted 6 lines
Isn't this out of the CPU arch, dealt with at the bus level?
It looks we try to be clever with modern PCI code by optimizing this access -- not saying we can change that, I know it is too late after 20+ years -- relying on hw behavior that was done that way to support legacy hw, in particular broken I/O accesses.
Quoted 6 lines
Not sure what the question is. We were dicussing how to emulate unaligned accesses from x86 guests if they happen.
On an x86 host we can do that easily, and it's just a couple of LOC. Though Peter Maydell dislikes host arch specific code. But I hope if it's a separate patch on top and it is visible how small it is, he will reconsider)Yes, if we still want x86 specific change, it would be better to be put separately.
Said so, I don't think the e1000e LEDCTL test illustrated what might break.. Isn't that only an exmaple showing unaligned access is "supported", however nothing breaks even if we use 1B*2?
My question was more about a real breakage, hence whenever it happened "it's more solid proof that per-arch change is required".
Quoted 4 lines
On other hosts we can't emulate them 100%, we either need to split to byte accesses or over-access and mask. Byte accesses feel safer. What qemu currently does with memmove is clearly not safe in the general case.
We should have another option that is not arch-dependent but keep the unaligned behavior.
For current master, AFAIU we do unaligned access for both
ram_deviceand rest. Say, even withram_device_mem_ops, it has both .unaligned=true for both .valid & .impl. I think it means indeed we have unaligned behavior even forram_device. It also means what matters in regards to the Realtek bug was only about aligned access (with subpage presence).Quoted 9 lines
I think it means we can always keep unaligned to stick with memcpy()/memmove(), but only use atomic ops for the aligned cases of 1/2/4/8. With that, I think we can also remove
ram_device_mem_opsand fix the bounce buffer issue.I think it means we'll stick with memcpy()/memmove() for all archs for unaligned, which is again not safe... but that can be an existing but separate problem to solve too.
Lets see if Peter Maydell and Michael are happy with this option. At least, we will have unified
qemu_ram_move()for all architectures with this option. Note thatqemu_ram_copy()won't be needed.I'm putting note on what's to be done in (v4) if this option is to be picked up. Let me know if there are missed points. It's basically combing what's done by
ram_device_mem_opsto upper layer (e.g. inqemu_ram_move()).
address_space_write
address_space_to_flatview
flatview_write
flatview_translate
flatview_write_continue
flatview_write_continue_step
memmove // (A) to replace it withqemu_ram_move()/**
*qemu_ram_move: move data to ramblock
*
* @dst: destination where the data is moved to
* @src: source where the data is moved from
* @n: length of data to be moved
*
* Move @n bytes from @src to @dst with the assumption that @src and @dst
* can overlap. The access is atomic if the source and destination buffer
* aren't overlapped for a well aligned and small-sized access. Otherwise,
* fall back to the standardmemmove().
*/
static voidqemu_ram_move(void *dst, const void *src, size_t n)
Quoted 9 lines
{
uintptr_t test, len;
if (src == dst || n == 0) {
return;
}
/* Overlapped buffers */
if (src < (dst + n) && dst < (src + n)) {s/&&/||/?
It's a bit weird to request memmove() for overlapped, e.g. I don't know if P2P can overlap too when some fuzzer fills in some MMIO address shifted for src/dst.. but I think I get what you want to simplify and it looks fine.
Otherwise it looks good.
Quoted 30 lines
memmove(dst, src, n);
return;
}
test = (uintptr_t)src | (uintptr_t)dst | n;
len = test & -test;
/* Unaligned or oversized access */
if (n > 8 || len != n) {
memmove(dst, src, n);
return;
}
switch (len) {
case 1:
qatomic_set((uint8_t *)dst, qatomic_read((uint8_t *)src));
break;
case 2:
qatomic_set((uint16_t *)dst, qatomic_read((uint16_t *)src));
break;
case 4:
qatomic_set((uint32_t *)dst, qatomic_read((uint32_t *)src));
break;
case 8:
qatomic_set((uint64_t *)dst, qatomic_read((uint64_t *)src));
break;
default:
g_assert_not_reached();
}
}[...]
Quoted 9 lines
Yes, I think one preparatory patch can added in (v4) to replace
memcpy()withmemmove()in the following paths, extending commit4a73aee8814("softmmu: Use memmove inflatview_write_continue"). With this replacement,qemu_ram_copy()won't be needed in (v4).hw/remote/vfio-user-obj.c::vfu_object_mr_rw
include/system/memory.h::address_space_read
system/physmem.c::flatview_read_continue_step
system/physmem.c::address_space_write_rom
Now after a second look, I think it's safe to drop address_space_write_rom() in the change list because it always directly manipulates the real RAM (that plays the ROM role). I recall it was used to be used in debugging path, but at least now when I look at master branch it's not. So we can drop.
I see you already ruled out address_space_read(), I'm not sure if it's about having that __builtin_constant_p() early check: I think it's still better to switch that to the new API too, then we get rid of implicit assumptions of memcpy() over builtin constants.
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 125 lines · Peter Xu
Quoted 124 lines · Gavin Shan
Quoted 81 lines · Peter Xu
Quoted 49 lines · Michael S. Tsirkin
Quoted 41 lines · Philippe Mathieu-Daudé
Quoted 34 lines · Michael S. Tsirkin
Quoted 24 lines · Peter Xu
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Gavin Shan
Quoted 9 lines · Peter Xu
Quoted 3 lines · Gavin Shan
If Peter is fine with two variants for x86 and non-x86 architectures. I can post (v4) for further review. That will be something like below and let me know if there are any other improvements are needed.
I have a generic question on the "unaligned access for x86": I think the question is about the one Michael raised here on unaligned access may break x86 here:
Quoted 9 lines
https://lore.kernel.org/qemu-devel/[email protected]/
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
Is that a real problem we need to consider, or can we start with unified approach and leave it for later?
I'm leaving this question to Michael.
Quoted 8 lines
Quoted 4 lines
Knowing what I know about hardware designers, it's something someone somewhere does)
It can be made a separate patch, just to show - it should be all of ~10LOC.It's only about removal of anything that might be controversial for now, thanks. I also wonder if anything would break, then it's more solid proof that per-arch change is required.
Quoted 5 lines
Repeating:
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.Quoted 6 lines
Isn't this out of the CPU arch, dealt with at the bus level?
It looks we try to be clever with modern PCI code by optimizing this access -- not saying we can change that, I know it is too late after 20+ years -- relying on hw behavior that was done that way to support legacy hw, in particular broken I/O accesses.
Quoted 6 lines
Not sure what the question is. We were dicussing how to emulate unaligned accesses from x86 guests if they happen.
On an x86 host we can do that easily, and it's just a couple of LOC. Though Peter Maydell dislikes host arch specific code. But I hope if it's a separate patch on top and it is visible how small it is, he will reconsider)Yes, if we still want x86 specific change, it would be better to be put separately.
Said so, I don't think the e1000e LEDCTL test illustrated what might break.. Isn't that only an exmaple showing unaligned access is "supported", however nothing breaks even if we use 1B*2?
My question was more about a real breakage, hence whenever it happened "it's more solid proof that per-arch change is required".
Quoted 4 lines
On other hosts we can't emulate them 100%, we either need to split to byte accesses or over-access and mask. Byte accesses feel safer. What qemu currently does with memmove is clearly not safe in the general case.
We should have another option that is not arch-dependent but keep the unaligned behavior.
For current master, AFAIU we do unaligned access for both
ram_deviceand rest. Say, even withram_device_mem_ops, it has both .unaligned=true for both .valid & .impl. I think it means indeed we have unaligned behavior even forram_device. It also means what matters in regards to the Realtek bug was only about aligned access (with subpage presence).Quoted 9 lines
I think it means we can always keep unaligned to stick with memcpy()/memmove(), but only use atomic ops for the aligned cases of 1/2/4/8. With that, I think we can also remove
ram_device_mem_opsand fix the bounce buffer issue.I think it means we'll stick with memcpy()/memmove() for all archs for unaligned, which is again not safe... but that can be an existing but separate problem to solve too.
Lets see if Peter Maydell and Michael are happy with this option. At least, we will have unified
qemu_ram_move()for all architectures with this option. Note thatqemu_ram_copy()won't be needed.I'm putting note on what's to be done in (v4) if this option is to be picked up. Let me know if there are missed points. It's basically combing what's done by
ram_device_mem_opsto upper layer (e.g. inqemu_ram_move()).
address_space_write
address_space_to_flatview
flatview_write
flatview_translate
flatview_write_continue
flatview_write_continue_step
memmove // (A) to replace it withqemu_ram_move()/**
*qemu_ram_move: move data to ramblock
*
* @dst: destination where the data is moved to
* @src: source where the data is moved from
* @n: length of data to be moved
*
* Move @n bytes from @src to @dst with the assumption that @src and @dst
* can overlap. The access is atomic if the source and destination buffer
* aren't overlapped for a well aligned and small-sized access. Otherwise,
* fall back to the standardmemmove().
*/
static voidqemu_ram_move(void *dst, const void *src, size_t n){ uintptr_t test, len;
Quoted 5 lines
Quoted 5 lines
if (src == dst || n == 0) { return; } /* Overlapped buffers */
overlapping
Quoted 9 lines
Quoted 1 line
if (src < (dst + n) && dst < (src + n)) {s/&&/||/?
It's a bit weird to request
memmove()for overlapped, e.g. I don't know if P2P can overlap too when some fuzzer fills in some MMIO address shifted for src/dst.. but I think I get what you want to simplify and it looks fine.Otherwise it looks good.
Why do we bother special casing overlapping buffers though? I do not get it, looks like rest of logic works exactly the same for overlapping and non.
Quoted 58 lines
Quoted 30 lines
memmove(dst, src, n); return; } test = (uintptr_t)src | (uintptr_t)dst | n; len = test & -test; /* Unaligned or oversized access */ if (n > 8 || len != n) { memmove(dst, src, n); return; } switch (len) { case 1: qatomic_set((uint8_t *)dst, qatomic_read((uint8_t *)src)); break; case 2: qatomic_set((uint16_t *)dst, qatomic_read((uint16_t *)src)); break; case 4: qatomic_set((uint32_t *)dst, qatomic_read((uint32_t *)src)); break; case 8: qatomic_set((uint64_t *)dst, qatomic_read((uint64_t *)src)); break; default: g_assert_not_reached(); } }[...]
Quoted 9 lines
Yes, I think one preparatory patch can added in (v4) to replace
memcpy()withmemmove()in the following paths, extending commit4a73aee8814("softmmu: Use memmove inflatview_write_continue"). With this replacement,qemu_ram_copy()won't be needed in (v4).hw/remote/vfio-user-obj.c::vfu_object_mr_rw
include/system/memory.h::address_space_read
system/physmem.c::flatview_read_continue_step
system/physmem.c::address_space_write_romNow after a second look, I think it's safe to drop
address_space_write_rom()in the change list because it always directly manipulates the real RAM (that plays the ROM role). I recall it was used to be used in debugging path, but at least now when I look at master branch it's not. So we can drop.I see you already ruled out
address_space_read(), I'm not sure if it's about having that__builtin_constant_p()early check: I think it's still better to switch that to the new API too, then we get rid of implicit assumptions ofmemcpy()over builtin constants.Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Xu
Quoted 2 lines · Gavin Shan
Quoted 1 line · Peter Xu
On Thu, Jul 23, 2026 at 05:05:08AM -0400, Michael S. Tsirkin wrote:
[...]
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 9 lines
We should have another option that is not arch-dependent but keep the unaligned behavior.
For current master, AFAIU we do unaligned access for both
ram_deviceand rest. Say, even withram_device_mem_ops, it has both .unaligned=true for both .valid & .impl. I think it means indeed we have unaligned behavior even forram_device. It also means what matters in regards to the Realtek bug was only about aligned access (with subpage presence).Quoted 9 lines
I think it means we can always keep unaligned to stick with memcpy()/memmove(), but only use atomic ops for the aligned cases of 1/2/4/8. With that, I think we can also remove
ram_device_mem_opsand fix the bounce buffer issue.I think it means we'll stick with memcpy()/memmove() for all archs for unaligned, which is again not safe... but that can be an existing but separate problem to solve too.
Lets see if Peter Maydell and Michael are happy with this option. At least, we will have unified
qemu_ram_move()for all architectures with this option. Note thatqemu_ram_copy()won't be needed.I'm putting note on what's to be done in (v4) if this option is to be picked up. Let me know if there are missed points. It's basically combing what's done by
ram_device_mem_opsto upper layer (e.g. inqemu_ram_move()).
address_space_write
address_space_to_flatview
flatview_write
flatview_translate
flatview_write_continue
flatview_write_continue_step
memmove // (A) to replace it withqemu_ram_move()/**
*qemu_ram_move: move data to ramblock
*
* @dst: destination where the data is moved to
* @src: source where the data is moved from
* @n: length of data to be moved
*
* Move @n bytes from @src to @dst with the assumption that @src and @dst
* can overlap. The access is atomic if the source and destination buffer
* aren't overlapped for a well aligned and small-sized access. Otherwise,
* fall back to the standardmemmove().
*/
static voidqemu_ram_move(void *dst, const void *src, size_t n){ uintptr_t test, len;
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
if (src == dst || n == 0) { return; } /* Overlapped buffers */overlapping
Corrected in v4.
https://lore.kernel.org/qemu-arm/[email protected]/
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
if (src < (dst + n) && dst < (src + n)) {s/&&/||/?
It's a bit weird to request
memmove()for overlapped, e.g. I don't know if P2P can overlap too when some fuzzer fills in some MMIO address shifted for src/dst.. but I think I get what you want to simplify and it looks fine.
Quoted 6 lines
Quoted 1 line
Otherwise it looks good.
Why do we bother special casing overlapping buffers though? I do not get it, looks like rest of logic works exactly the same for overlapping and non.
Indeed. The explicit check if src/dst are overlapping has been dropped in v4.
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
memmove(dst, src, n); return; } test = (uintptr_t)src | (uintptr_t)dst | n; len = test & -test; /* Unaligned or oversized access */ if (n > 8 || len != n) { memmove(dst, src, n); return; } switch (len) { case 1: qatomic_set((uint8_t *)dst, qatomic_read((uint8_t *)src)); break; case 2: qatomic_set((uint16_t *)dst, qatomic_read((uint16_t *)src)); break; case 4: qatomic_set((uint32_t *)dst, qatomic_read((uint32_t *)src)); break;
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
case 8: qatomic_set((uint64_t *)dst, qatomic_read((uint64_t *)src)); break; default: g_assert_not_reached(); } }
[...]
Thanks,
Gavin
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
In reply to Peter XuQuoted 132 lines · Peter Xu
Quoted 121 lines · Gavin Shan
Quoted 81 lines · Peter Xu
Quoted 49 lines · Michael S. Tsirkin
Quoted 41 lines · Philippe Mathieu-Daudé
Quoted 34 lines · Michael S. Tsirkin
Quoted 24 lines · Peter Xu
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Gavin Shan
Quoted 9 lines · Peter Xu
Quoted 3 lines · Gavin Shan
If Peter is fine with two variants for x86 and non-x86 architectures. I can post (v4) for further review. That will be something like below and let me know if there are any other improvements are needed.
I have a generic question on the "unaligned access for x86": I think the question is about the one Michael raised here on unaligned access may break x86 here:
Quoted 9 lines
https://lore.kernel.org/qemu-devel/[email protected]/
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
Is that a real problem we need to consider, or can we start with unified approach and leave it for later?
I'm leaving this question to Michael.
Quoted 8 lines
Quoted 4 lines
Knowing what I know about hardware designers, it's something someone somewhere does)
It can be made a separate patch, just to show - it should be all of ~10LOC.It's only about removal of anything that might be controversial for now, thanks. I also wonder if anything would break, then it's more solid proof that per-arch change is required.
Quoted 5 lines
Repeating:
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.Quoted 6 lines
Isn't this out of the CPU arch, dealt with at the bus level?
It looks we try to be clever with modern PCI code by optimizing this access -- not saying we can change that, I know it is too late after 20+ years -- relying on hw behavior that was done that way to support legacy hw, in particular broken I/O accesses.
Quoted 6 lines
Not sure what the question is. We were dicussing how to emulate unaligned accesses from x86 guests if they happen.
On an x86 host we can do that easily, and it's just a couple of LOC. Though Peter Maydell dislikes host arch specific code. But I hope if it's a separate patch on top and it is visible how small it is, he will reconsider)Yes, if we still want x86 specific change, it would be better to be put separately.
Said so, I don't think the e1000e LEDCTL test illustrated what might break.. Isn't that only an exmaple showing unaligned access is "supported", however nothing breaks even if we use 1B*2?
My question was more about a real breakage, hence whenever it happened "it's more solid proof that per-arch change is required".
Quoted 4 lines
On other hosts we can't emulate them 100%, we either need to split to byte accesses or over-access and mask. Byte accesses feel safer. What qemu currently does with memmove is clearly not safe in the general case.
We should have another option that is not arch-dependent but keep the unaligned behavior.
For current master, AFAIU we do unaligned access for both
ram_deviceand rest. Say, even withram_device_mem_ops, it has both .unaligned=true for both .valid & .impl. I think it means indeed we have unaligned behavior even forram_device. It also means what matters in regards to the Realtek bug was only about aligned access (with subpage presence).Quoted 9 lines
I think it means we can always keep unaligned to stick with memcpy()/memmove(), but only use atomic ops for the aligned cases of 1/2/4/8. With that, I think we can also remove
ram_device_mem_opsand fix the bounce buffer issue.I think it means we'll stick with memcpy()/memmove() for all archs for unaligned, which is again not safe... but that can be an existing but separate problem to solve too.
Lets see if Peter Maydell and Michael are happy with this option. At least, we will have unified
qemu_ram_move()for all architectures with this option. Note thatqemu_ram_copy()won't be needed.I'm putting note on what's to be done in (v4) if this option is to be picked up. Let me know if there are missed points. It's basically combing what's done by
ram_device_mem_opsto upper layer (e.g. inqemu_ram_move()).
address_space_write
address_space_to_flatview
flatview_write
flatview_translate
flatview_write_continue
flatview_write_continue_step
memmove // (A) to replace it withqemu_ram_move()/**
*qemu_ram_move: move data to ramblock
*
* @dst: destination where the data is moved to
* @src: source where the data is moved from
* @n: length of data to be moved
*
* Move @n bytes from @src to @dst with the assumption that @src and @dst
* can overlap. The access is atomic if the source and destination buffer
* aren't overlapped for a well aligned and small-sized access. Otherwise,
* fall back to the standardmemmove().
*/
static voidqemu_ram_move(void *dst, const void *src, size_t n)Quoted 9 lines
{ uintptr_t test, len; if (src == dst || n == 0) { return; } /* Overlapped buffers */ if (src < (dst + n) && dst < (src + n)) {
Quoted 8 lines
s/&&/||/?
It's a bit weird to request
memmove()for overlapped, e.g. I don't know if P2P can overlap too when some fuzzer fills in some MMIO address shifted for src/dst.. but I think I get what you want to simplify and it looks fine.Otherwise it looks good.
The condition is correct to check if src/dst are overlapped. As Michael pointed out the overlapping case can be well handled by "if (n > 8 || len != n)", so this check has been dropped in v4.
https://lore.kernel.org/qemu-arm/[email protected]/
I don't think src/dst can be overlapped since p2p happens on two separate devices. My idea to ignore the overlapping case is that would happen very rarely, maybe only valid in the fuzzer case.
Quoted 43 lines
Quoted 30 lines
memmove(dst, src, n); return; } test = (uintptr_t)src | (uintptr_t)dst | n; len = test & -test; /* Unaligned or oversized access */ if (n > 8 || len != n) { memmove(dst, src, n); return; } switch (len) { case 1: qatomic_set((uint8_t *)dst, qatomic_read((uint8_t *)src)); break; case 2: qatomic_set((uint16_t *)dst, qatomic_read((uint16_t *)src)); break; case 4: qatomic_set((uint32_t *)dst, qatomic_read((uint32_t *)src)); break; case 8: qatomic_set((uint64_t *)dst, qatomic_read((uint64_t *)src)); break; default: g_assert_not_reached(); } }[...]
Quoted 9 lines
Yes, I think one preparatory patch can added in (v4) to replace
memcpy()withmemmove()in the following paths, extending commit4a73aee8814("softmmu: Use memmove inflatview_write_continue"). With this replacement,qemu_ram_copy()won't be needed in (v4).hw/remote/vfio-user-obj.c::vfu_object_mr_rw include/system/memory.h::address_space_read system/physmem.c::flatview_read_continue_step system/physmem.c::address_space_write_rom
Quoted 6 lines
Now after a second look, I think it's safe to drop
address_space_write_rom()in the change list because it always directly manipulates the real RAM (that plays the ROM role). I recall it was used to be used in debugging path, but at least now when I look at master branch it's not. So we can drop.
Ack, address_space_write_rom() is dropped in v4.
Quoted 5 lines
I see you already ruled out
address_space_read(), I'm not sure if it's about having that__builtin_constant_p()early check: I think it's still better to switch that to the new API too, then we get rid of implicit assumptions ofmemcpy()over builtin constants.
Yes, qemu_ram_move() is also used for __builtin_constant_p() case in v4.
Thanks,
Gavin
Re: [PATCH v3 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors
In reply to Peter MaydellQuoted 34 lines · Peter Maydell
Quoted 33 lines · Michael S. Tsirkin
Quoted 31 lines · Peter Maydell
Quoted 7 lines · Michael S. Tsirkin
I think there is exactly 1 kinda reasonable case. A 2 byte read/write at offset
0x1within a dword. This maps nicely to even classical PCI byte enable mechanism and so yes it works if your CPU can initiate these things, and it's atomic.I tried reading LEDCTL on e1000e:
Quoted 8 lines
byte @ 0xe00: 0x64 byte @ 0xe01: 0x2a byte @ 0xe02: 0x00 byte @ 0xe03: 0x00 word @ 0xe00: 0x2a64 word @ 0xe01: 0x002aWorks fine.
The e1000e datasheet actually documents what it does in this case (slightly surprising, since hardware engineers love to leave this kind of corner case undocumented):
# For registers that should be accessed as 32-bit double words,
# partial writes (less than a 32-bit double word) does not take # effect (such as, the write is ignored). # Partial reads # return all 32 bits of data regardless of the byte enables. # # Note: Partial reads to clear-by-read registers (such as, ICR) # can have unexpected results since all 32 bits are actually read # regardless of the byte enables. Partial reads should not be done.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
So for this specific device that access is out-of-spec.
You mean that access to clear by read should not be done, right?
The datasheet is ambiguous about whether "Partial reads should not be done" is meant to apply generally or only to clear-by-read registers.
I mean it's in a note about clear-by-read registers. Seems unambigous.
Quoted 15 lines
Quoted 6 lines
Quoted 5 lines
I guess what I'm wondering is: can we just have code that does an aligned exact-width access in the 1/2/4/8 byte aligned case, and the host's best approximation to an unaligned exact-width access for the 2/4/8 byte unaligned case?
Quoted 8 lines
That's my idea, too.
Quoted 3 lines
(so on sparc you get multiple smaller accesses, and on most archs including x86 and arm you get an unaligned load).
I thought unaligned load from uncacheable on arm is also a fault?
Quoted 10 lines
For Arm the distinction is not cacheable/uncacheable but Normal vs Device. (Device is essentially for things which are not RAM; Normal is for RAM and RAM-like things, and includes all of Normal Non-cacheable, Normal WT-Cacheable and Normal WB-Cacheable.) Things mapped as Normal memory don't generate unaligned faults (unless the guest turned them on deliberately). For Device memory, it is IMPLEMENTATION DEFINED whether you get an alignment fault or not if you map something as Device that could have handled unaligned accesses if you had mapped it as Normal.
Right. Sorry I was unclear. I was referring to pgprot_noncached in Linux.
Quoted 17 lines
Quoted 13 lines
Quoted 6 lines
That would mean the guest could
potentially provoke a fault on the load/store on an access to a passthrough device, but if you give the guest passthrough access it can very likely provoke a fault anyway, depending on exactly what the device is.Quoted 6 lines
I think the most likely reason for an unaligned access in this codepath is "it's actually RAM, either really host RAM or else something memory-like in a BAR", and either way if the guest does a 4-byte unaligned access then doing a 4-byte unaligned access seems better than second-guessing it, even on non-x86.
Quoted 3 lines
Right though remember: whether it's RAM doesn't matter. What matters is how we map it. qemu might fault because it maps NC but guest maps cacheable and it's ok.
Quoted 7 lines
If QEMU and the guest disagree about the memory attributes on Arm then we have already lost, because the architecture says that memory attribute mismatches result in a variety of undesirable effects including things like loss of cache coherency (i.e. read and writes via QEMU's NC mapping disagree with ones via the guest's cacheable mapping because the latter are hitting in the cache and the former are bypassing it).
But we don't know how guest mapped it, at least without VFIO's help. I guess DMA into device BAR is just often broken on ARM except if guest happens to use same attributes as set up by vfio.
Quoted 9 lines
Quoted 4 lines
But, all this in theory. At a high level, I personally think going with what you propose as a 1st approximation is entirely reasonable, except for one thing: we really should not crash qemu, since access can be from guest userspace.
You can't prevent faults entirely, though -- if the device being mapped has e.g. behaviour that says "unaligned accesses will fault" and then the x86 guest does an unaligned access, then the device will trigger a fault,
I don't get this part. How can a device "trigger a fault" on the x86 CPU? But on non x86 in theory yes software could want the fault. Hopefully it does not.
Quoted 4 lines
and the fault is what
you want because it's what the guest would see on real h/w. Unfortunately we don't have a convenient way to feed the fault back to the guest.
I didn't check but I mean we could add it to kvm I am guessing, if we want. A lot of work so I would maybe wait for an actual issue before spending engineering time on this.
Quoted 3 lines
At some level if you pass through
host hardware you're relying on the guest to not do totally stupid things.
Indeed.
Quoted 2 lines
thanks
-- PMM
PATCH v3 2/2 system/memory: Make ram device region directly accessible
In reply to Gavin ShanThis basically reverts 4a2e242bbb30 ("memory: Don't use memcpy for ram_device regions") to make ram device region directly accessible again. With this, the bounce buffer is bypassed in address_space_map() when a ram device region is involved, potentially avoid to overrun the (small) bounce buffer.
--- include/system/memory.h | 11 ++--------- system/memory.c | 41 +---------------------------------------- system/trace-events | 2 -- 3 files changed, 3 insertions(+), 51 deletions(-)
include/system/memory.h +2 -9
@@ -2944,15 +2944,8 @@ static inline bool memory_region_supports_direct_access(const MemoryRegion *mr)if (memory_region_is_romd (mr)) {return true; } -if (!memory_region_is_ram (mr)) { -return false; - } -/* -* RAM DEVICE regions can be accessed directly using memcpy, but it might -* be MMIO and access using mempy can be wrong (e.g., using instructions not -* intended for MMIO access). So we treat this as IO. -*/ -return !memory_region_is_ram_device (mr); + +return memory_region_is_ram (mr); } static inline bool memory_access_is_direct(const MemoryRegion *mr,
system/memory.c +1 -40
54 lines not on this page · raw message
system/trace-events +0 -2
@@ -20,8 +20,6 @@ memory_region_ops_read(int cpu_index, void *mr, uint64_t addr, uint64_t value, u memory_region_ops_write(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size, const char *name) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u name '%s'" memory_region_subpage_read(int cpu_index, void *mr, uint64_t offset, uint64_t value, unsigned size) "cpu %d mr %p offset 0x%"PRIx64" value 0x%"PRIx64" size %u" memory_region_subpage_write(int cpu_index, void *mr, uint64_t offset, uint64_t value, unsigned size) "cpu %d mr %p offset 0x%"PRIx64" value 0x%"PRIx64" size %u" -memory_region_ram_device_read(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u" -memory_region_ram_device_write(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u" memory_region_sync_dirty(const char *mr, const char *listener, int global) "mr '%s' listener '%s' synced (global=%d)" flatview_new(void *view, void *root) "%p (root %p)" flatview_destroy(void *view, void *root) "%p (root %p)"
Signature · 1 line
2.54.0
Quoted 2 lines · Gavin Shan
All ram device regions was turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions").
What is missing here is the list of issues around
direct ram access and how we are solving them.
Thanks,
Signature · 1 line
MST
Quoted 8 lines · Michael S. Tsirkin
Quoted 2 lines · Gavin Shan
All ram device regions was turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions").What is missing here is the list of issues around
direct ram access and how we are solving them.
Sorry that I saw your request for the inclusion after v3 series was posted. They have been appended to the thread as a reply.
Thanks,
Gavin
Quoted 1 line
Thanks,
Quoted 18 lines · Gavin Shan
All ram device regions was turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to a hanged guest where a NVidia GH100 GPU is passed from host. The memory in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take DMA bounce buffer inaddress_space_map()to cover the DMA request. However, the bounce buffer size is 4096 bytes and we're overrunning it easily when the guest has significant disk activities on compiling 'cuda-samples'. The full log and problem description can be found from PATCH[1/2]'s commit log.Try to fix the issue handled in commit
4a2e242bbbby replacing memcopy()/memmove()with newly added helpers qemu_ram_{copy, move}() that works on top of __builtin_{memcpy, memmove} or unaligned access friendly memory movement in the accessors to the ram device regions. With this, we can basically revert that commit to make ram device region directly accessible again and bypass the bounce buffer inaddress_space_map()where the guest hang is caused.
Quoted 3 lines
PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors PATCH[2] makes ram device region directly accessible again
Michael asked to include below context in the cover letter in v3, but I didn't noticed that before I sent v3 series, appended with them.
----
The issues listed by Michael:
1. On x86, memcpy is different from __builtin_memcpy if one uses old 1.0 force-headers from 2019. Likely no longer relevant.
2. variable length memcpy can translate 2,4,8 byte guest access into multiple byte accesses. doing this for mmio is guaranteed to break devices.
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
4. also on x86, vector instructions for large (>16 byte) writes into pgprot_noncached memory are safe and faster than multiple 8 byte ones.
5. also on x86 it so happens that if you write a fixed-size memcpy this gets optimized to a single store/load and it works for aligned and unaligned addresses on that architecture. How to ensure this keeps being correct is left as an excerise for the reader. But qemu already relies on this and did for years.
6. on non-x86 both unaligned accesses and vector instructions for accessing UC memory are illegal.
7. standard vfio gives KVM VM_ALLOW_ANY_UNCACHED, so even on non x86 guest can map the memory as as pgprot_noncached/ioremap or pgprot_writecombine/ioremap_uc. If it does the second then it can use unaligned or vector for access. This is why normal passthrough tends to work - it never traps to qemu at all. But for qemu, vfio uses pgprot_noncached unconditionally so qemu can't use unaligned or vector instructions on non-x86.
8. But for nvgrace RAM, vfio has a driver that uses pgprot_writecombine/ioremap_uc. so qemu could safely use unaligned/vector instructioons even on non-x86.
9. Except sadly, vfio currently does not tell qemu how it maps the memory, so qemu can not know what is safe on non-x86.
Now, what is to be done?
A. on x86, we must avoid converting 2,4,8 byte accesses into byte accesses. At least for aligned, perferably for unaligned accesses too. Fixed width memcpy seems to work for this. Whether we should bother with __builtin to work around broken old fortify headers, I donnu.
I do not have any answer how to check that compiler does this correctly. If anyone is motivated enough, adding a GCC builtin could be possible. Given qemu did this for years, I think we can leave solving this for another day.
B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf.
C. on non-x86, we currently must not memcpy since we do not know if it is pgprot_noncached. yes, performance will be bad for DMA into device RAM.
D. It goes without saying that casting an unaligned address to unint32_t (be it for qatomic_set or whatever) is undefined behaviour in C and so a bad idea on any architecture.
E. also for non-x86, we really should teach vfio to tell qemu whether it maps device pgprot_noncached or pgprot_writecombine.
we will then be able to use memcpy for >8 accesses.
Anyone, correct me if I'm wrong? Maybe I should start a new thread with this summary?
Thanks,
Gavin
Quoted 59 lines · Gavin Shan
Quoted 18 lines · Gavin Shan
All ram device regions was turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to a hanged guest where a NVidia GH100 GPU is passed from host. The memory in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take DMA bounce buffer inaddress_space_map()to cover the DMA request. However, the bounce buffer size is 4096 bytes and we're overrunning it easily when the guest has significant disk activities on compiling 'cuda-samples'. The full log and problem description can be found from PATCH[1/2]'s commit log.Try to fix the issue handled in commit
4a2e242bbbby replacing memcopy()/memmove()with newly added helpers qemu_ram_{copy, move}() that works on top of __builtin_{memcpy, memmove} or unaligned access friendly memory movement in the accessors to the ram device regions. With this, we can basically revert that commit to make ram device region directly accessible again and bypass the bounce buffer inaddress_space_map()where the guest hang is caused.Quoted 3 lines
PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors PATCH[2] makes ram device region directly accessible again
Michael asked to include below context in the cover letter in v3, but I didn't noticed that before I sent v3 series, appended with them.
----
The issues listed by Michael:
1. On x86, memcpy is different from
__builtin_memcpyif one uses old 1.0 force-headers from 2019. Likely no longer relevant.2. variable length memcpy can translate 2,4,8 byte guest access into multiple byte accesses. doing this for mmio is guaranteed to break devices.
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
4. also on x86, vector instructions for large (>16 byte) writes into
pgprot_noncachedmemory are safe and faster than multiple 8 byte ones.5. also on x86 it so happens that if you write a fixed-size memcpy this gets optimized to a single store/load and it works for aligned and unaligned addresses on that architecture. How to ensure this keeps being correct is left as an excerise for the reader. But qemu already relies on this and did for years.
6. on non-x86 both unaligned accesses and vector instructions for accessing UC memory are illegal.
7. standard vfio gives KVM
VM_ALLOW_ANY_UNCACHED, so even on non x86 guest can map the memory as as pgprot_noncached/ioremap or pgprot_writecombine/ioremap_uc. If it does the second then it can use unaligned or vector for access. This is why normal passthrough tends to work - it never traps to qemu at all. But for qemu, vfio usespgprot_noncachedunconditionally so qemu can't use unaligned or vector instructions on non-x86.
Quoted 6 lines
8. But for nvgrace RAM, vfio has a driver that uses pgprot_writecombine/ioremap_uc. so qemu could safely use unaligned/vector instructioons even on non-x86.
9. Except sadly, vfio currently does not tell qemu how it maps the memory, so qemu can not know what is safe on non-x86.
And more:
10. on x86 memcpy will sometimes do multiple overlapping stores when size is not a power of 2. for example, a 15 byte write is done with 2 8-byte stores. This is theoretically an issue
if guest does something super clever with ordering,
but does not seem to be in practice.
10. on non-x86 memcpy will do multiple overlapping stores even for single byte writes. E.g. it does it to avoid extra branches. This is causing issues in practice.
Quoted 31 lines
Now, what is to be done?
A. on x86, we must avoid converting 2,4,8 byte accesses into byte accesses. At least for aligned, perferably for unaligned accesses too. Fixed width memcpy seems to work for this. Whether we should bother with
__builtinto work around broken old fortify headers, I donnu.
I do not have any answer how to check that compiler does this correctly. If anyone is motivated enough, adding a GCC builtin could be possible. Given qemu did this for years, I think we can leave solving this for another day.B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf.
C. on non-x86, we currently must not memcpy since we do not know if it is
pgprot_noncached. yes, performance will be bad for DMA into device RAM.D. It goes without saying that casting an unaligned address to
unint32_t(be it forqatomic_setor whatever) is undefined behaviour in C and so a bad idea on any architecture.E. also for non-x86, we really should teach vfio to tell qemu whether it maps device
pgprot_noncachedorpgprot_writecombine.
we will then be able to use memcpy for >8 accesses.Anyone, correct me if I'm wrong? Maybe I should start a new thread with this summary?
Thanks,
Gavin
Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Gavin Shan
Quoted 18 lines · Gavin Shan
All ram device regions was turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to a hanged guest where a NVidia GH100 GPU is passed from host. The memory in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take DMA bounce buffer inaddress_space_map()to cover the DMA request. However, the bounce buffer size is 4096 bytes and we're overrunning it easily when the guest has significant disk activities on compiling 'cuda-samples'. The full log and problem description can be found from PATCH[1/2]'s commit log.Try to fix the issue handled in commit
4a2e242bbbby replacing memcopy()/memmove()with newly added helpers qemu_ram_{copy, move}() that works on top of __builtin_{memcpy, memmove} or unaligned access friendly memory movement in the accessors to the ram device regions. With this, we can basically revert that commit to make ram device region directly accessible again and bypass the bounce buffer inaddress_space_map()where the guest hang is caused.
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors PATCH[2] makes ram device region directly accessible again
Michael asked to include below context in the cover letter in v3, but I didn't noticed that before I sent v3 series, appended with them.
Looking at the list of issues (questions) raised by Michael, I don't understand every one before I'm able to put more time to dig, but I feel this series has too ambitious goal to cover accesses to all the directly accessible regions with the newly introduced qemu_ram_{copy, move}. It causes too many behavior changes and concerns, making this series impossible to land.
I would suggest to break down the goal and step back to apply the newly introduced qemu_ram_{copy, move} to the ram device regions only? It's actually something proposed by Peter Xu in the earlier replies. Taking address_space_write() as an example, the indirectly accessible regions are covered by memory_region_dispatch_write() in (1), the ram device region is covered by qemu_ram_move() in (2), and all other directly accessible regions are covered by memmove() in (3).
address_space_write
flatview_write
flatview_write_continue
flatview_write_continue_step
memory_access_size // (1) indirectly accessible region
memory_region_dispatch_write
access_with_adjusted_size
memory_region_write_accessor
mr->ops->write
qemu_ram_move // (2) ram device region
memmove // (3) all other directly accessible regions
With the limitation, only the ram device regions in (2) are affected. We're basically moving the accesses to the ram device region from (1) to (2). No changes introduced to other types of regions. The goal is to make the ram device region accessible so that the bounce buffer can be bypassed in DMA path.
Quoted 49 lines
Quoted 34 lines
----
The issues listed by Michael:
1. On x86, memcpy is different from
__builtin_memcpyif one uses old 1.0 force-headers from 2019. Likely no longer relevant.2. variable length memcpy can translate 2,4,8 byte guest access into multiple byte accesses. doing this for mmio is guaranteed to break devices.
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
4. also on x86, vector instructions for large (>16 byte) writes into
pgprot_noncachedmemory are safe and faster than multiple 8 byte ones.5. also on x86 it so happens that if you write a fixed-size memcpy this gets optimized to a single store/load and it works for aligned and unaligned addresses on that architecture. How to ensure this keeps being correct is left as an excerise for the reader. But qemu already relies on this and did for years.
6. on non-x86 both unaligned accesses and vector instructions for accessing UC memory are illegal.
7. standard vfio gives KVM
VM_ALLOW_ANY_UNCACHED, so even on non x86 guest can map the memory as as pgprot_noncached/ioremap or pgprot_writecombine/ioremap_uc. If it does the second then it can use unaligned or vector for access. This is why normal passthrough tends to work - it never traps to qemu at all. But for qemu, vfio usespgprot_noncachedunconditionally so qemu can't use unaligned or vector instructions on non-x86.Quoted 6 lines
8. But for nvgrace RAM, vfio has a driver that uses pgprot_writecombine/ioremap_uc. so qemu could safely use unaligned/vector instructioons even on non-x86.
9. Except sadly, vfio currently does not tell qemu how it maps the memory, so qemu can not know what is safe on non-x86.
And more:
10. on x86 memcpy will sometimes do multiple overlapping stores when size is not a power of 2. for example, a 15 byte write is done with 2 8-byte stores. This is theoretically an issue
if guest does something super clever with ordering,
but does not seem to be in practice.
Quoted 37 lines
10. on non-x86 memcpy will do multiple overlapping stores even for single byte writes. E.g. it does it to avoid extra branches. This is causing issues in practice.
Quoted 29 lines
Now, what is to be done?
A. on x86, we must avoid converting 2,4,8 byte accesses into byte accesses. At least for aligned, perferably for unaligned accesses too. Fixed width memcpy seems to work for this. Whether we should bother with
__builtinto work around broken old fortify headers, I donnu.
I do not have any answer how to check that compiler does this correctly. If anyone is motivated enough, adding a GCC builtin could be possible. Given qemu did this for years, I think we can leave solving this for another day.B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf.
C. on non-x86, we currently must not memcpy since we do not know if it is
pgprot_noncached. yes, performance will be bad for DMA into device RAM.D. It goes without saying that casting an unaligned address to
unint32_t(be it forqatomic_setor whatever) is undefined behaviour in C and so a bad idea on any architecture.E. also for non-x86, we really should teach vfio to tell qemu whether it maps device
pgprot_noncachedorpgprot_writecombine.
we will then be able to use memcpy for >8 accesses.Anyone, correct me if I'm wrong? Maybe I should start a new thread with this summary?
Thanks,
Gavin
Quoted 24 lines · Gavin Shan
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Gavin Shan
Quoted 21 lines · Gavin Shan
All ram device regions was turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to a hanged guest where a NVidia GH100 GPU is passed from host. The memory in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take DMA bounce buffer inaddress_space_map()to cover the DMA request. However, the bounce buffer size is 4096 bytes and we're overrunning it easily when the guest has significant disk activities on compiling 'cuda-samples'. The full log and problem description can be found from PATCH[1/2]'s commit log.Try to fix the issue handled in commit
4a2e242bbbby replacing memcopy()/memmove()with newly added helpers qemu_ram_{copy, move}() that works on top of __builtin_{memcpy, memmove} or unaligned access friendly memory movement in the accessors to the ram device regions. With this, we can basically revert that commit to make ram device region directly accessible again and bypass the bounce buffer inaddress_space_map()where the guest hang is caused.PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors PATCH[2] makes ram device region directly accessible again
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Michael asked to include below context in the cover letter in v3, but I didn't noticed that before I sent v3 series, appended with them.
Looking at the list of issues (questions) raised by Michael, I don't understand every one
Gavin, I doubt one should make memory.c changes without understanding the issues it is trying to address.
What is unclear? Ask away.
Quoted 24 lines
before I'm able to put more time to dig, but I feel this series has too ambitious goal to cover accesses to all the directly accessible regions with the newly introduced qemu_ram_{copy, move}. It causes too many behavior changes and concerns, making this series impossible to land.
I would suggest to break down the goal and step back to apply the newly introduced qemu_ram_{copy, move} to the ram device regions only? It's actually something proposed by Peter Xu in the earlier replies. Taking
address_space_write()as an example, the indirectly accessible regions are covered bymemory_region_dispatch_write()in (1), the ram device region is covered byqemu_ram_move()in (2), and all other directly accessible regions are covered bymemmove()in (3).
address_space_write
flatview_write
flatview_write_continue
flatview_write_continue_step
memory_access_size// (1) indirectly accessible region
memory_region_dispatch_write
access_with_adjusted_size
memory_region_write_accessor
mr->ops->write
qemu_ram_move// (2) ram device region
memmove // (3) all other directly accessible regions
Quoted 4 lines
With the limitation, only the ram device regions in (2) are affected. We're basically moving the accesses to the ram device region from (1) to (2). No changes introduced to other types of regions. The goal is to make the ram device region accessible so that the bounce buffer can be bypassed in DMA path.
Esthetics aside - ram device regions have all the same issues.
Maybe you can limit the scope of the changes,
but I doubt you can get out understanding)
Quoted 89 lines
Quoted 49 lines
Quoted 34 lines
----
The issues listed by Michael:
1. On x86, memcpy is different from
__builtin_memcpyif one uses old 1.0 force-headers from 2019. Likely no longer relevant.2. variable length memcpy can translate 2,4,8 byte guest access into multiple byte accesses. doing this for mmio is guaranteed to break devices.
3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices.
4. also on x86, vector instructions for large (>16 byte) writes into
pgprot_noncachedmemory are safe and faster than multiple 8 byte ones.5. also on x86 it so happens that if you write a fixed-size memcpy this gets optimized to a single store/load and it works for aligned and unaligned addresses on that architecture. How to ensure this keeps being correct is left as an excerise for the reader. But qemu already relies on this and did for years.
6. on non-x86 both unaligned accesses and vector instructions for accessing UC memory are illegal.
7. standard vfio gives KVM
VM_ALLOW_ANY_UNCACHED, so even on non x86 guest can map the memory as as pgprot_noncached/ioremap or pgprot_writecombine/ioremap_uc. If it does the second then it can use unaligned or vector for access. This is why normal passthrough tends to work - it never traps to qemu at all. But for qemu, vfio usespgprot_noncachedunconditionally so qemu can't use unaligned or vector instructions on non-x86.Quoted 6 lines
8. But for nvgrace RAM, vfio has a driver that uses pgprot_writecombine/ioremap_uc. so qemu could safely use unaligned/vector instructioons even on non-x86.
9. Except sadly, vfio currently does not tell qemu how it maps the memory, so qemu can not know what is safe on non-x86.
And more:
10. on x86 memcpy will sometimes do multiple overlapping stores when size is not a power of 2. for example, a 15 byte write is done with 2 8-byte stores. This is theoretically an issue
if guest does something super clever with ordering,
but does not seem to be in practice.Quoted 37 lines
10. on non-x86 memcpy will do multiple overlapping stores even for single byte writes. E.g. it does it to avoid extra branches. This is causing issues in practice.
Quoted 29 lines
Now, what is to be done?
A. on x86, we must avoid converting 2,4,8 byte accesses into byte accesses. At least for aligned, perferably for unaligned accesses too. Fixed width memcpy seems to work for this. Whether we should bother with
__builtinto work around broken old fortify headers, I donnu.
I do not have any answer how to check that compiler does this correctly. If anyone is motivated enough, adding a GCC builtin could be possible. Given qemu did this for years, I think we can leave solving this for another day.B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf.
C. on non-x86, we currently must not memcpy since we do not know if it is
pgprot_noncached. yes, performance will be bad for DMA into device RAM.D. It goes without saying that casting an unaligned address to
unint32_t(be it forqatomic_setor whatever) is undefined behaviour in C and so a bad idea on any architecture.E. also for non-x86, we really should teach vfio to tell qemu whether it maps device
pgprot_noncachedorpgprot_writecombine.
we will then be able to use memcpy for >8 accesses.Anyone, correct me if I'm wrong? Maybe I should start a new thread with this summary?
Thanks,
Gavin
Hi Michael,
Quoted 29 lines · Michael S. Tsirkin
Quoted 28 lines · Gavin Shan
Quoted 26 lines · Michael S. Tsirkin
Quoted 25 lines · Gavin Shan
Quoted 18 lines · Gavin Shan
All ram device regions was turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to a hanged guest where a NVidia GH100 GPU is passed from host. The memory in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take DMA bounce buffer inaddress_space_map()to cover the DMA request. However, the bounce buffer size is 4096 bytes and we're overrunning it easily when the guest has significant disk activities on compiling 'cuda-samples'. The full log and problem description can be found from PATCH[1/2]'s commit log.Try to fix the issue handled in commit
4a2e242bbbby replacing memcopy()/memmove()with newly added helpers qemu_ram_{copy, move}() that works on top of __builtin_{memcpy, memmove} or unaligned access friendly memory movement in the accessors to the ram device regions. With this, we can basically revert that commit to make ram device region directly accessible again and bypass the bounce buffer inaddress_space_map()where the guest hang is caused.Quoted 3 lines
PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors PATCH[2] makes ram device region directly accessible again
Michael asked to include below context in the cover letter in v3, but I didn't noticed that before I sent v3 series, appended with them.
Quoted 8 lines
Quoted 2 lines
Looking at the list of issues (questions) raised by Michael, I don't understand every one
Gavin, I doubt one should make
memory.cchanges without understanding the issues it is trying to address.What is unclear? Ask away.
Yeah, absolutely. I need some time to understand all questions or suggestions by digging the code a bit, before I'm able to come back to you, but not very soon though :-)
Quoted 30 lines
Quoted 25 lines
before I'm able to put more time to dig, but I feel this series has too ambitious goal to cover accesses to all the directly accessible regions with the newly introduced qemu_ram_{copy, move}. It causes too many behavior changes and concerns, making this series impossible to land.
I would suggest to break down the goal and step back to apply the newly introduced qemu_ram_{copy, move} to the ram device regions only? It's actually something proposed by Peter Xu in the earlier replies. Taking
address_space_write()as an example, the indirectly accessible regions are covered bymemory_region_dispatch_write()in (1), the ram device region is covered byqemu_ram_move()in (2), and all other directly accessible regions are covered bymemmove()in (3).
address_space_write
flatview_write
flatview_write_continue
flatview_write_continue_step
memory_access_size// (1) indirectly accessible region
memory_region_dispatch_write
access_with_adjusted_size
memory_region_write_accessor
mr->ops->write
qemu_ram_move// (2) ram device region
memmove // (3) all other directly accessible regionsQuoted 4 lines
With the limitation, only the ram device regions in (2) are affected. We're basically moving the accesses to the ram device region from (1) to (2). No changes introduced to other types of regions. The goal is to make the ram device region accessible so that the bounce buffer can be bypassed in DMA path.
Quoted 5 lines
Esthetics aside - ram device regions have all the same issues.
Maybe you can limit the scope of the changes,
but I doubt you can get out understanding)
Yes. Limited to the scope of VFIO and PCI BARs, it depends on how the PCI BARs are mapped, pgprot_noncached or pgprot_writecombine. The listed problems and concerns are existing on the ram device region exposed with pgprot_noncached. However, everything should be just fine if the region is exposed with pgprot_writecombine. Am I understanding this correctly?
[...]
Thanks,
Gavin
Quoted 81 lines · Gavin Shan
Hi Michael,
Quoted 29 lines · Michael S. Tsirkin
Quoted 28 lines · Gavin Shan
Quoted 26 lines · Michael S. Tsirkin
Quoted 25 lines · Gavin Shan
Quoted 18 lines · Gavin Shan
All ram device regions was turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to a hanged guest where a NVidia GH100 GPU is passed from host. The memory in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take DMA bounce buffer inaddress_space_map()to cover the DMA request. However, the bounce buffer size is 4096 bytes and we're overrunning it easily when the guest has significant disk activities on compiling 'cuda-samples'. The full log and problem description can be found from PATCH[1/2]'s commit log.Try to fix the issue handled in commit
4a2e242bbbby replacing memcopy()/memmove()with newly added helpers qemu_ram_{copy, move}() that works on top of __builtin_{memcpy, memmove} or unaligned access friendly memory movement in the accessors to the ram device regions. With this, we can basically revert that commit to make ram device region directly accessible again and bypass the bounce buffer inaddress_space_map()where the guest hang is caused.Quoted 3 lines
PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors PATCH[2] makes ram device region directly accessible again
Michael asked to include below context in the cover letter in v3, but I didn't noticed that before I sent v3 series, appended with them.
Quoted 8 lines
Quoted 2 lines
Looking at the list of issues (questions) raised by Michael, I don't understand every one
Gavin, I doubt one should make
memory.cchanges without understanding the issues it is trying to address.What is unclear? Ask away.
Yeah, absolutely. I need some time to understand all questions or suggestions by digging the code a bit, before I'm able to come back to you, but not very soon though :-)
Quoted 30 lines
Quoted 25 lines
before I'm able to put more time to dig, but I feel this series has too ambitious goal to cover accesses to all the directly accessible regions with the newly introduced qemu_ram_{copy, move}. It causes too many behavior changes and concerns, making this series impossible to land.
I would suggest to break down the goal and step back to apply the newly introduced qemu_ram_{copy, move} to the ram device regions only? It's actually something proposed by Peter Xu in the earlier replies. Taking
address_space_write()as an example, the indirectly accessible regions are covered bymemory_region_dispatch_write()in (1), the ram device region is covered byqemu_ram_move()in (2), and all other directly accessible regions are covered bymemmove()in (3).
address_space_write
flatview_write
flatview_write_continue
flatview_write_continue_step
memory_access_size// (1) indirectly accessible region
memory_region_dispatch_write
access_with_adjusted_size
memory_region_write_accessor
mr->ops->write
qemu_ram_move// (2) ram device region
memmove // (3) all other directly accessible regionsQuoted 4 lines
With the limitation, only the ram device regions in (2) are affected. We're basically moving the accesses to the ram device region from (1) to (2). No changes introduced to other types of regions. The goal is to make the ram device region accessible so that the bounce buffer can be bypassed in DMA path.
Quoted 5 lines
Esthetics aside - ram device regions have all the same issues.
Maybe you can limit the scope of the changes,
but I doubt you can get out understanding)
Quoted 5 lines
Yes. Limited to the scope of VFIO and PCI BARs, it depends on how the PCI BARs are mapped,
pgprot_noncachedorpgprot_writecombine. The listed problems and concerns are existing on the ram device region exposed withpgprot_noncached. However, everything should be just fine if the region is exposed withpgprot_writecombine. Am I understanding this correctly?
Not everything)
We have issues around guest RAM access, too.
But yes you can then mostly access device RAM as guest RAM on arm and x86. I am not sure about power. for power pgprot_writecombine is cache inhibited and I am not sure e.g. vector instructions with misaligned addresses behave the same.
Quoted 4 lines
[...]
Thanks,
Gavin