system/memory: Make ram device region directly accessible
v5 cover system/memory: Make ram device region directly accessible
All ram device regions was turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to a frozen guest where a NVidia GH100 GPU is passed from host. The memory in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take DMA bounce buffer in address_space_map() to cover the DMA request. However, the bounce buffer size is 4096 bytes only and it's exhaused very quickly when the guest has significant disk activities on compiling 'cuda-samples'. The full log and problem description can be found from PATCH[2/3]'s commit log.
Fix the issue handled in commit 4a2e242bbb by replacing memmove() with newly added qemu_ram_move() where the aligned and small-sized accesses are handled by qatomics, and fall back to memmove() otherwise, for the directly accessible regions. With this, we can revert 4a2e242bbb to make ram device region directly accessible again and bypass the bounce buffer in address_space_map() where the guest hang happens.
PATCH[1] replaces memcpy() with memomve() for directly accessible regions PATCH[2] uses qemu_ram_move() for directly accessible regions PATCH[3] makes ram device region directly accessible again
Changelog
=========
v4 -> v5:
* https://lore.kernel.org/qemu-arm/[email protected]/
* Improved commit log of PATCH[2/3] and comments for
qemu_ram_move() (Peterm/Peterx)
v3 -> v4:
* https://lore.kernel.org/qemu-arm/[email protected]/
* New PATCH[1/3] to replace memcpy() with memmove() (Peterx)
* Unified qemu_ram_move() for all architectures (Peterx/Michael)
v2 -> v3:
* https://lore.kernel.org/qemu-arm/[email protected]/
* Documentation for qemu_ram_{copy, move} (Peterm/Michael)
* Support qemu_ram_move() for overlapped src/dest (Richard)
* Use {memcpy, memmove} if step is 16-bytes or more (Michael)
* Code improvements (Richard/Michael)
v1 -> v2:
* https://lore.kernel.org/qemu-arm/[email protected]/
* Rename address_space_{memcpy, memmove}() to qemu_ram_{copy, move}()
and move them to physmem.c and memory.h (Philippe)
* Use memcpy() and memmove() in qemu_ram_{copy, move}() for the variable
length case (Miachel)
* Handle unaligned access in qemu_ram_{copy, move}() for all archs
except i386 and x86_64 (Richard/Michael)
RFCv1 -> v1:
* https://lists.nongnu.org/archive/html/qemu-arm/2026-06/msg00307.html
* Reworked solution based on suggestions from Peter Xu, Peter Maydell and Michael S. Tsirkin
Gavin Shan (3):
system/memory: Use memmove() for directly accessible regions system/memory: Use qemu_ram_move() for directly accessible regions system/memory: Make ram device region directly accessible
hw/remote/vfio-user-obj.c | 4 ++-- include/system/memory.h | 46 ++++++++++++++++++++++++++++++--------- system/memory.c | 41 +--------------------------------- system/physmem.c | 43 ++++++++++++++++++++++++++++++++++-- system/trace-events | 2 -- 5 files changed, 80 insertions(+), 56 deletions(-)
Signature · 1 line
2.55.0
PATCH v5 1/3 system/memory: Use memmove() for directly accessible regions
Similar to what's done in commit 4a73aee88140 ("softmmu: Use memmove in flatview_write_continue"), there are more sites where the overlapping source and destination buffer are allowed for the directly accessible regions. Use memmove() in those sites, listed as below.
hw/remote/vfio-user-obj.c::vfu_object_mr_rw
include/system/memory.h::address_space_read
system/physmem.c::flatview_read_continue_step
--- hw/remote/vfio-user-obj.c | 4 ++-- include/system/memory.h | 2 +- system/physmem.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
hw/remote/vfio-user-obj.c +2 -2
@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset, ram_ptr =memory_region_get_ram_ptr (mr);if (is_write) { -memcpy ((ram_ptr + offset), buf, size); +memmove ((ram_ptr + offset), buf, size); }else { -memcpy (buf, (ram_ptr + offset), size); +memmove (buf, (ram_ptr + offset), size); }return 0 ;
include/system/memory.h +1 -1
@@ -2741,7 +2741,7 @@ MemTxResult address_space_read(const AddressSpace *as, hwaddr addr, mr =flatview_translate (fv, addr, &addr1, &l, false, attrs);if (len == l &&memory_access_is_direct (mr, false, attrs)) { ptr =qemu_map_ram_ptr (mr->ram_block, addr1); -memcpy (buf, ptr, len); +memmove (buf, ptr, len); }else { result =flatview_read_continue (fv, addr, attrs, buf, len, addr1, l, mr);
system/physmem.c +1 -1
@@ -3363,7 +3363,7 @@ static MemTxResult flatview_read_continue_step(MemTxAttrs attrs, uint8_t *buf,uint8_t *ram_ptr =qemu_ram_ptr_length (mr->ram_block, mr_addr, l, false, false); -memcpy (buf, ram_ptr, *l); +memmove (buf, ram_ptr, *l);return MEMTX_OK ; }
Signature · 1 line
2.55.0
Re: [PATCH v5 1/3] system/memory: Use memmove() for directly accessible regions
Quoted 9 lines · Gavin Shan
Similar to what's done in commit
4a73aee88140("softmmu: Use memmove inflatview_write_continue"), there are more sites where the overlapping source and destination buffer are allowed for the directly accessible regions. Usememmove()in those sites, listed as below.hw/remote/vfio-user-obj.c::vfu_object_mr_rw
include/system/memory.h::address_space_read
system/physmem.c::flatview_read_continue_step
Quoted 8 lines
Signed-off-by Gavin Shan <[email protected]>Reviewed-by Peter Maydell <[email protected]>Reviewed-by Peter Xu <[email protected]>--- hw/remote/vfio-user-obj.c | 4 ++-- include/system/memory.h | 2 +- system/physmem.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
PATCH v5 2/3 system/memory: Use qemu_ram_move() for directly accessible regions
In reply to Gavin ShanAll ram device regions were turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.
host$ lspci | grep GH100
0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1)
host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64 \
-machine virt,gic-version=host,ras=on,highmem-mmio-size=4T \
-accel kvm -cpu host -smp cpus=48 -m size=8G \
-drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0 \
-device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4 \
-device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0
:
guest$ cd cuda-samples/build
guest$ make -j 20 clean
guest$ make -j 20
:
[ 54%] Linking CUDA executable graphMemoryNodes
[ 54%] Built target graphMemoryNodes
<no more output afterwards, guest becomes frozen here>guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources
[ 555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)
When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 of the GH100 GPU card can be presented to the guest through memory hot-add. The page cache can then be allocated from the hot added memory blocks when cuda-samples is being built. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is an indirectly accessible ram device region in qemu. However, the max bounce bufer size is only 4096 bytes by default and that is exhausted quickly, leading to a reset on the virtio-blk device and frozen guest eventually.
QEMU
====
virtio_blk_handle_output
virtio_blk_handle_vq
virtio_blk_get_request
virtqueue_pop
virtqueue_split_pop
virtqueue_map_desc
address_space_map
memory_access_is_direct # Return false
memory_region_supports_direct_access
(qemu) info mtree
memory-region: pci_bridge_pci
0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci
0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4
0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4
0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]This adds qemu_ram_move() where the aligned and small-sized accesses are handled by qatomics, and fall back to memmove() otherwise. The memove() for the directly accessible regions is replaced by qemu_ram_move() so that the issue covered by commit 4a2e242bbb (MMIO access instructions were optimized to SSE instructions) is fixed. This makes 'ram_device_mem_ops' redundant, paving the way to revert that commit to make the ram device region directly accessible again in the next patch.
Besides, this also fixes the issue of the unexpected frozen reception on e1000 NIC in the scenario of DPDK due to the wrong Rx queue full indication caused by the following memcpy(), which is turned to 3 consective 'strb' instructions to the same location by glibc-2.24+ for aarch64. With this applied, the syntax of one-byte store is strictly ensured by a one-byte qatomic set.
QEMU
====
e1000_receive_iov
pci_dma_write
pci_dma_rw
dma_memory_rw
dma_memory_rw_relaxed
address_space_rw
address_space_write
flatview_write
flatview_write_continue
flatview_write_continue_step
memcpy # 3 consective 'strb' instructions
---
v5: Improved commit log and comments to qemu_ram_move()
--- hw/remote/vfio-user-obj.c | 4 ++-- include/system/memory.h | 35 ++++++++++++++++++++++++++++++- system/physmem.c | 43 +++++++++++++++++++++++++++++++++++++-- 3 files changed, 77 insertions(+), 5 deletions(-)
hw/remote/vfio-user-obj.c +2 -2
@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset, ram_ptr =memory_region_get_ram_ptr (mr);if (is_write) { -memmove ((ram_ptr + offset), buf, size); +qemu_ram_move ((ram_ptr + offset), buf, size); }else { -memmove (buf, (ram_ptr + offset), size); +qemu_ram_move (buf, (ram_ptr + offset), size); }return 0 ;
include/system/memory.h +34 -1
@@ -2668,6 +2668,39 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);void address_space_unregister_map_client (AddressSpace *as,QEMUBH *bh);/* Internal functions, part of the implementation of address_space_read. */ + +/** +* qemu_ram_move: move data from or to ramblock +* +* @dst: destination where the data is moved to +* @src: source where the data is moved from +* @n: length of data to be moved +* +* Move @n bytes from @src to @dst, the memory areas may overlap. This +* provides the same semantics as memmove(), plus an additional stronger +* guarantee: if @n is 1, 2 or 4 or 8 bytes, and @src and @dst are both +* naturally aligned for that access size, and the memory areas do not +* overlap, then both the load and the store will be done as a single +* atomic access (with the semantics of qatomic_read() and qatomic_set()). +* +* This is the underlying function that we use to implement accesses by +* a guest vCPU or a device DMA operation to a ram block. The atomic +* guarantee is needed for two major cases: (A) When the ram block is +* backed by a PCI BAR passed through from a host device (and so it might +* be hardware registers that must be accessed exactly once at the right +* width); (B) When an emulated device updates a data structure shared in +* guest memory with guest software (e.g. a network device's set of tx and +* rx descriptor blocks), if a write to memory is accidentally performed +* multiple times then it can break the guest code when it busy polls the +* guest memory. +* +* We don't attempt to perform the exact access when it would be unaligned +* because this can't be done on all host architectures. Although this is +* strictly speaking not doing what would happen on real hardware, we don't +* think there are going to be situations where that matters in practice. +*/ +void qemu_ram_move (void *dst,const void *src,size_t n); +MemTxResult address_space_read_full (const AddressSpace *as,hwaddr addr,MemTxAttrs attrs,void *buf,hwaddr len); MemTxResult flatview_read_continue(FlatView *fv, hwaddr addr,
@@ -2741,7 +2774,7 @@ MemTxResult address_space_read(const AddressSpace *as, hwaddr addr, mr =flatview_translate (fv, addr, &addr1, &l, false, attrs);if (len == l &&memory_access_is_direct (mr, false, attrs)) { ptr =qemu_map_ram_ptr (mr->ram_block, addr1); -memmove (buf, ptr, len); +qemu_ram_move (buf, ptr, len); }else { result =flatview_read_continue (fv, addr, attrs, buf, len, addr1, l, mr);
system/physmem.c +41 -2
61 lines not on this page · raw message
Signature · 1 line
2.55.0
Re: [PATCH v5 2/3] system/memory: Use qemu_ram_move() for directly accessible regions
Quoted 189 lines · Gavin Shan
All ram device regions were turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.host$ lspci | grep GH100 0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1) host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64 \ -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T \ -accel kvm -cpu host -smp cpus=48 -m size=8G \ -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0 \ -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4 \ -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0 : guest$ cd cuda-samples/build guest$ make -j 20 clean guest$ make -j 20 : [ 54%] Linking CUDA executable graphMemoryNodes [ 54%] Built target graphMemoryNodes <no more output afterwards, guest becomes frozen here>guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources
[ 555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 of the GH100 GPU card can be presented to the guest through memory hot-add. The page cache can then be allocated from the hot added memory blocks when cuda-samples is being built. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is an indirectly accessible ram device region in qemu. However, the max bounce bufer size is only 4096 bytes by default and that is exhausted quickly, leading to a reset on the virtio-blk device and frozen guest eventually.
QEMU
====
virtio_blk_handle_output
virtio_blk_handle_vq
virtio_blk_get_request
virtqueue_pop
virtqueue_split_pop
virtqueue_map_desc
address_space_map
memory_access_is_direct# Return false
memory_region_supports_direct_access(qemu) info mtree
memory-region:pci_bridge_pci0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci 0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4 0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4 0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]This adds
qemu_ram_move()where the aligned and small-sized accesses are handled by qatomics, and fall back tomemmove()otherwise. Thememove()for the directly accessible regions is replaced byqemu_ram_move()so that the issue covered by commit4a2e242bbb(MMIO access instructions were optimized to SSE instructions) is fixed. This makes 'ram_device_mem_ops' redundant, paving the way to revert that commit to make the ram device region directly accessible again in the next patch.Besides, this also fixes the issue of the unexpected frozen reception on e1000 NIC in the scenario of DPDK due to the wrong Rx queue full indication caused by the following
memcpy(), which is turned to 3 consective 'strb' instructions to the same location by glibc-2.24+ for aarch64. With this applied, the syntax of one-byte store is strictly ensured by a one-byte qatomic set.QEMU
====
e1000_receive_iov
pci_dma_write
pci_dma_rw
dma_memory_rw
dma_memory_rw_relaxed
address_space_rw
address_space_write
flatview_write
flatview_write_continue
flatview_write_continue_step
memcpy # 3 consective 'strb' instructionsReported-by Julia Graham <[email protected]>Reported-by Liu Gang <[email protected]>Reported-by Ding Hui <[email protected]>Suggested-by Michael S. Tsirkin <[email protected]>Suggested-by Peter Xu <[email protected]>Suggested-by Richard Henderson <[email protected]>Suggested-by Peter Maydell <[email protected]>Signed-off-by Gavin Shan <[email protected]>---
v5: Improved commit log and comments toqemu_ram_move()--- hw/remote/vfio-user-obj.c | 4 ++-- include/system/memory.h | 35 ++++++++++++++++++++++++++++++- system/physmem.c | 43 +++++++++++++++++++++++++++++++++++++-- 3 files changed, 77 insertions(+), 5 deletions(-)hw/remote/vfio-user-obj.c +2
-2@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset, ram_ptr =memory_region_get_ram_ptr (mr);if (is_write) { -memmove ((ram_ptr + offset), buf, size); +qemu_ram_move ((ram_ptr + offset), buf, size); }else { -memmove (buf, (ram_ptr + offset), size); +qemu_ram_move (buf, (ram_ptr + offset), size); }return 0 ;include/system/memory.h +34
-1@@ -2668,6 +2668,39 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);void address_space_unregister_map_client (AddressSpace *as,QEMUBH *bh);/* Internal functions, part of the implementation of address_space_read. */ + +/** +* qemu_ram_move: move data from or to ramblock +* +* @dst: destination where the data is moved to +* @src: source where the data is moved from +* @n: length of data to be moved +* +* Move @n bytes from @src to @dst, the memory areas may overlap. This +* provides the same semantics as memmove(), plus an additional stronger +* guarantee: if @n is 1, 2 or 4 or 8 bytes, and @src and @dst are both +* naturally aligned for that access size, and the memory areas do not +* overlap, then both the load and the store will be done as a single +* atomic access (with the semantics of qatomic_read() and qatomic_set()). +* +* This is the underlying function that we use to implement accesses by +* a guest vCPU or a device DMA operation to a ram block. The atomic +* guarantee is needed for two major cases: (A) When the ram block is +* backed by a PCI BAR passed through from a host device (and so it might +* be hardware registers that must be accessed exactly once at the right +* width); (B) When an emulated device updates a data structure shared in +* guest memory with guest software (e.g. a network device's set of tx and +* rx descriptor blocks), if a write to memory is accidentally performed +* multiple times then it can break the guest code when it busy polls the +* guest memory. +* +* We don't attempt to perform the exact access when it would be unaligned +* because this can't be done on all host architectures. Although this is +* strictly speaking not doing what would happen on real hardware, we don't +* think there are going to be situations where that matters in practice. +*/ +void qemu_ram_move (void *dst,const void *src,size_t n); +MemTxResult address_space_read_full (const AddressSpace *as,hwaddr addr,MemTxAttrs attrs,void *buf,hwaddr len); MemTxResult flatview_read_continue(FlatView *fv, hwaddr addr,@@ -2741,7 +2774,7 @@ MemTxResult address_space_read(const AddressSpace *as, hwaddr addr, mr =flatview_translate (fv, addr, &addr1, &l, false, attrs);if (len == l &&memory_access_is_direct (mr, false, attrs)) { ptr =qemu_map_ram_ptr (mr->ram_block, addr1); -memmove (buf, ptr, len); +qemu_ram_move (buf, ptr, len); }else { result =flatview_read_continue (fv, addr, attrs, buf, len, addr1, l, mr);system/physmem.c +10
-0@@ -3158,6 +3158,45 @@ void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size)invalidate_and_set_dirty (mr, addr, size); } +void qemu_ram_move (void *dst,const void *src,size_t n) +{ +uintptr_t test, len; + +if (src == dst || n ==0 ) { +return ; + } + +/* +* Maximal length of aligned access that are determined by @src,
system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3180
Quoted 10 lines
system/physmem.c +10
-0+* @dst and @n +*/ + test = (uintptr_t )src | (uintptr_t )dst | n; + len = test & -test; + +/* Overlapping buffers, unaligned or oversized access */ +if (n >8 || len != n) { +memmove (dst, src, n); +return ; + }
This part I don't get exactly. It's fine on most arches I think. x86, power...
But here is aarch64 for example:
https://codebrowser.dev/glibc/glibc/sysdeps/aarch64/memcpy.S.html
it says
It uses unaligned accesses and branchless sequences to keep the code small, simple and improve performance.
so if the point is to get rid of unaligned accesses to BARs, are you sure it achieves this?
Quoted 42 lines
system/physmem.c +21
-2+ +switch (len) { +case 1 : +qatomic_set ((uint8_t *)dst,qatomic_read ((uint8_t *)src)); +break ; +case 2 : +qatomic_set ((uint16_t *)dst,qatomic_read ((uint16_t *)src)); +break ; +case 4 : +qatomic_set ((uint32_t *)dst,qatomic_read ((uint32_t *)src)); +break ; +case 8 : +qatomic_set ((uint64_t *)dst,qatomic_read ((uint64_t *)src)); +break ; +default : +g_assert_not_reached (); + } +} +int memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr) {unsigned access_size_max = mr->ops->valid.max_access_size;@@ -3270,7 +3309,7 @@ static MemTxResult flatview_write_continue_step(MemTxAttrs attrs,uint8_t *ram_ptr =qemu_ram_ptr_length (mr->ram_block, mr_addr, l, false, true); -memmove (ram_ptr, buf, *l); +qemu_ram_move (ram_ptr, buf, *l);invalidate_and_set_dirty (mr, mr_addr, *l);return MEMTX_OK ;@@ -3363,7 +3402,7 @@ static MemTxResult flatview_read_continue_step(MemTxAttrs attrs, uint8_t *buf,uint8_t *ram_ptr =qemu_ram_ptr_length (mr->ram_block, mr_addr, l, false, false); -memmove (buf, ram_ptr, *l); +qemu_ram_move (buf, ram_ptr, *l);return MEMTX_OK ; }Signature · 1 line
2.55.0
Re: [PATCH v5 2/3] system/memory: Use qemu_ram_move() for directly accessible regions
Quoted 208 lines · Michael S. Tsirkin
Quoted 189 lines · Gavin Shan
All ram device regions were turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.host$ lspci | grep GH100 0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1) host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64 \ -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T \ -accel kvm -cpu host -smp cpus=48 -m size=8G \ -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0 \ -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4 \ -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0 : guest$ cd cuda-samples/build guest$ make -j 20 clean guest$ make -j 20 : [ 54%] Linking CUDA executable graphMemoryNodes [ 54%] Built target graphMemoryNodes <no more output afterwards, guest becomes frozen here> guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources [ 555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 of the GH100 GPU card can be presented to the guest through memory hot-add. The page cache can then be allocated from the hot added memory blocks when cuda-samples is being built. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is an indirectly accessible ram device region in qemu. However, the max bounce bufer size is only 4096 bytes by default and that is exhausted quickly, leading to a reset on the virtio-blk device and frozen guest eventually.
QEMU ==== virtio_blk_handle_output virtio_blk_handle_vq virtio_blk_get_request virtqueue_pop virtqueue_split_pop virtqueue_map_desc address_space_map memory_access_is_direct # Return false memory_region_supports_direct_access (qemu) info mtree memory-region: pci_bridge_pci 0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci 0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4 0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4 0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]This adds
qemu_ram_move()where the aligned and small-sized accesses are handled by qatomics, and fall back tomemmove()otherwise. Thememove()for the directly accessible regions is replaced byqemu_ram_move()so that the issue covered by commit4a2e242bbb(MMIO access instructions were optimized to SSE instructions) is fixed. This makes 'ram_device_mem_ops' redundant, paving the way to revert that commit to make the ram device region directly accessible again in the next patch.Besides, this also fixes the issue of the unexpected frozen reception on e1000 NIC in the scenario of DPDK due to the wrong Rx queue full indication caused by the following
memcpy(), which is turned to 3 consective 'strb' instructions to the same location by glibc-2.24+ for aarch64. With this applied, the syntax of one-byte store is strictly ensured by a one-byte qatomic set.QEMU ==== e1000_receive_iov pci_dma_write pci_dma_rw dma_memory_rw dma_memory_rw_relaxed address_space_rw address_space_write flatview_write flatview_write_continue flatview_write_continue_step memcpy # 3 consective 'strb' instructionsReported-by Julia Graham <[email protected]>Reported-by Liu Gang <[email protected]>Reported-by Ding Hui <[email protected]>Suggested-by Michael S. Tsirkin <[email protected]>Suggested-by Peter Xu <[email protected]>Suggested-by Richard Henderson <[email protected]>Suggested-by Peter Maydell <[email protected]>Signed-off-by Gavin Shan <[email protected]>---
v5: Improved commit log and comments toqemu_ram_move()--- hw/remote/vfio-user-obj.c | 4 ++-- include/system/memory.h | 35 ++++++++++++++++++++++++++++++- system/physmem.c | 43 +++++++++++++++++++++++++++++++++++++-- 3 files changed, 77 insertions(+), 5 deletions(-)hw/remote/vfio-user-obj.c +2
-2@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset, ram_ptr =memory_region_get_ram_ptr (mr);if (is_write) { -memmove ((ram_ptr + offset), buf, size); +qemu_ram_move ((ram_ptr + offset), buf, size); }else { -memmove (buf, (ram_ptr + offset), size); +qemu_ram_move (buf, (ram_ptr + offset), size); }return 0 ;include/system/memory.h +34
-1@@ -2668,6 +2668,39 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);void address_space_unregister_map_client (AddressSpace *as,QEMUBH *bh);/* Internal functions, part of the implementation of address_space_read. */ + +/** +* qemu_ram_move: move data from or to ramblock +* +* @dst: destination where the data is moved to +* @src: source where the data is moved from +* @n: length of data to be moved +* +* Move @n bytes from @src to @dst, the memory areas may overlap. This +* provides the same semantics as memmove(), plus an additional stronger +* guarantee: if @n is 1, 2 or 4 or 8 bytes, and @src and @dst are both +* naturally aligned for that access size, and the memory areas do not +* overlap, then both the load and the store will be done as a single +* atomic access (with the semantics of qatomic_read() and qatomic_set()). +* +* This is the underlying function that we use to implement accesses by +* a guest vCPU or a device DMA operation to a ram block. The atomic +* guarantee is needed for two major cases: (A) When the ram block is +* backed by a PCI BAR passed through from a host device (and so it might +* be hardware registers that must be accessed exactly once at the right +* width); (B) When an emulated device updates a data structure shared in +* guest memory with guest software (e.g. a network device's set of tx and +* rx descriptor blocks), if a write to memory is accidentally performed +* multiple times then it can break the guest code when it busy polls the +* guest memory. +* +* We don't attempt to perform the exact access when it would be unaligned +* because this can't be done on all host architectures. Although this is +* strictly speaking not doing what would happen on real hardware, we don't +* think there are going to be situations where that matters in practice. +*/ +void qemu_ram_move (void *dst,const void *src,size_t n); +MemTxResult address_space_read_full (const AddressSpace *as,hwaddr addr,MemTxAttrs attrs,void *buf,hwaddr len); MemTxResult flatview_read_continue(FlatView *fv, hwaddr addr,@@ -2741,7 +2774,7 @@ MemTxResult address_space_read(const AddressSpace *as, hwaddr addr, mr =flatview_translate (fv, addr, &addr1, &l, false, attrs);if (len == l &&memory_access_is_direct (mr, false, attrs)) { ptr =qemu_map_ram_ptr (mr->ram_block, addr1); -memmove (buf, ptr, len); +qemu_ram_move (buf, ptr, len); }else { result =flatview_read_continue (fv, addr, attrs, buf, len, addr1, l, mr);system/physmem.c +10
-0@@ -3158,6 +3158,45 @@ void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size)invalidate_and_set_dirty (mr, addr, size); } +void qemu_ram_move (void *dst,const void *src,size_t n) +{ +uintptr_t test, len; + +if (src == dst || n ==0 ) { +return ; + } + +/* +* Maximal length of aligned access that are determined by @src, system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3180
Quoted 10 lines
system/physmem.c +10
-0+* @dst and @n +*/ + test = (uintptr_t )src | (uintptr_t )dst | n; + len = test & -test; + +/* Overlapping buffers, unaligned or oversized access */ +if (n >8 || len != n) { +memmove (dst, src, n); +return ; + }This part I don't get exactly. It's fine on most arches I think. x86, power...
But here is aarch64 for example:
https://codebrowser.dev/glibc/glibc/sysdeps/aarch64/memcpy.S.html
Quoted 7 lines
it says
It uses unaligned accesses and branchless sequences to keep the code small, simple and improve performance.so if the point is to get rid of unaligned accesses to BARs, are you sure it achieves this?
No, we don't intend to avoid the unaligned accesses to BARs if they're existing, even on aarch64. The behavior is consistent to that introduced by 'ram_device_mem_ops'. When 'ram_device_mem_ops' is in place of upstream 'master' branch, the unaligned access is put through, without splitting it to aligned accesses because 'ram_device_mem_ops::{valid, impl}.unaligned' have been set to true.
Quoted 36 lines
Quoted 36 lines
system/physmem.c +20
-1+ +switch (len) { +case 1 : +qatomic_set ((uint8_t *)dst,qatomic_read ((uint8_t *)src)); +break ; +case 2 : +qatomic_set ((uint16_t *)dst,qatomic_read ((uint16_t *)src)); +break ; +case 4 : +qatomic_set ((uint32_t *)dst,qatomic_read ((uint32_t *)src)); +break ; +case 8 : +qatomic_set ((uint64_t *)dst,qatomic_read ((uint64_t *)src)); +break ; +default : +g_assert_not_reached (); + } +} +int memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr) {unsigned access_size_max = mr->ops->valid.max_access_size;@@ -3270,7 +3309,7 @@ static MemTxResult flatview_write_continue_step(MemTxAttrs attrs,uint8_t *ram_ptr =qemu_ram_ptr_length (mr->ram_block, mr_addr, l, false, true); -memmove (ram_ptr, buf, *l); +qemu_ram_move (ram_ptr, buf, *l);invalidate_and_set_dirty (mr, mr_addr, *l);return MEMTX_OK ;@@ -3363,7 +3402,7 @@ static MemTxResult flatview_read_continue_step(MemTxAttrs attrs, uint8_t *buf,uint8_t *ram_ptr =qemu_ram_ptr_length (mr->ram_block, mr_addr, l, false, false);
Quoted 5 lines
system/physmem.c static MemTxResult flatview_read_continue_step(MemTxAttrs attrs, uint8_t *buf, · line 3408
Quoted 5 lines
system/physmem.c +1
-1-memmove (buf, ram_ptr, *l); +qemu_ram_move (buf, ram_ptr, *l);return MEMTX_OK ; }
Thanks,
Gavin
Re: [PATCH v5 2/3] system/memory: Use qemu_ram_move() for directly accessible regions
In reply to Michael S. TsirkinQuoted 30 lines · Michael S. Tsirkin
Quoted 10 lines · Gavin Shan
system/physmem.c +10
-0+void qemu_ram_move (void *dst,const void *src,size_t n) +{ +uintptr_t test, len; + +if (src == dst || n ==0 ) { +return ; + } + +/* +* Maximal length of aligned access that are determined by @src, system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3180
Quoted 10 lines
system/physmem.c +10
-0+* @dst and @n +*/ + test = (uintptr_t )src | (uintptr_t )dst | n; + len = test & -test; + +/* Overlapping buffers, unaligned or oversized access */ +if (n >8 || len != n) { +memmove (dst, src, n); +return ; + }This part I don't get exactly. It's fine on most arches I think. x86, power...
But here is aarch64 for example:
https://codebrowser.dev/glibc/glibc/sysdeps/aarch64/memcpy.S.html
Quoted 6 lines
it says
It uses unaligned accesses and branchless sequences to keep the code small, simple and improve performance.so if the point is to get rid of unaligned accesses to BARs, are you sure it achieves this?
The idea is rather just to ignore the unaligned-access-to-not-RAM case as "only a theoretical issue" and default it to memmove(). If anybody reports an actual problem we can think about it then :-)
-- PMM
Re: [PATCH v5 2/3] system/memory: Use qemu_ram_move() for directly accessible regions
Quoted 35 lines · Peter Maydell
Quoted 34 lines · Michael S. Tsirkin
Quoted 10 lines · Gavin Shan
system/physmem.c +10
-0+void qemu_ram_move (void *dst,const void *src,size_t n) +{ +uintptr_t test, len; + +if (src == dst || n ==0 ) { +return ; + } + +/* +* Maximal length of aligned access that are determined by @src, system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3180
Quoted 10 lines
system/physmem.c +10
-0+* @dst and @n +*/ + test = (uintptr_t )src | (uintptr_t )dst | n; + len = test & -test; + +/* Overlapping buffers, unaligned or oversized access */ +if (n >8 || len != n) { +memmove (dst, src, n); +return ; + }This part I don't get exactly. It's fine on most arches I think. x86, power...
But here is aarch64 for example:
https://codebrowser.dev/glibc/glibc/sysdeps/aarch64/memcpy.S.htmlit says
It uses unaligned accesses and branchless sequences to keep the code small, simple and improve performance.
Quoted 8 lines
Quoted 2 lines
so if the point is to get rid of unaligned accesses to BARs, are you sure it achieves this?
The idea is rather just to ignore the unaligned-access-to-not-RAM case as "only a theoretical issue" and default it to
memmove(). If anybody reports an actual problem we can think about it then :-)-- PMM
Well given the only reported case of virtio DMA into vfio memory we have is GH100 and that one is mapped MT_NORMAL, I'm fine with that.
I feel extending vfio to report the mapping type to userspace might not be a bad idea down the road, though. "Directly accessible" is a very rough approximation to the rich variety of mappings available across architectures.
Signature · 1 line
MST
Re: [PATCH v5 2/3] system/memory: Use qemu_ram_move() for directly accessible regions
Quoted 6 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell
The idea is rather just to ignore the unaligned-access-to-not-RAM case as "only a theoretical issue" and default it to
memmove(). If anybody reports an actual problem we can think about it then :-)
Quoted 7 lines
Well given the only reported case of virtio DMA into vfio memory we have is GH100 and that one is mapped
MT_NORMAL, I'm fine with that.I feel extending vfio to report the mapping type to userspace might not be a bad idea down the road, though. "Directly accessible" is a very rough approximation to the rich variety of mappings available across architectures.
Yes, this sounds like a good longer term idea. If userspace doesn't have a way to know what the mapping behind something it's mmap()ed is then it can't know what operations it can and can't safely do.
-- PMM
Re: [PATCH v5 2/3] system/memory: Use qemu_ram_move() for directly accessible regions
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell
The idea is rather just to ignore the unaligned-access-to-not-RAM case as "only a theoretical issue" and default it to
memmove(). If anybody reports an actual problem we can think about it then :-)Well given the only reported case of virtio DMA into vfio memory we have is GH100 and that one is mapped
MT_NORMAL, I'm fine with that.
Quoted 9 lines
Quoted 4 lines
I feel extending vfio to report the mapping type to userspace might not be a bad idea down the road, though. "Directly accessible" is a very rough approximation to the rich variety of mappings available across architectures.
Yes, this sounds like a good longer term idea. If userspace doesn't have a way to know what the mapping behind something it's mmap()ed is then it can't know what operations it can and can't safely do.
Gavin, didn't you have the idea of using a VFIO region caps for that ?
This is similar to what we did on POWER9 with :
VFIO_REGION_INFO_CAP_NVLINK2_SSATGT
which exported GPU RAM to the userspace too.
Thanks,
C.
Re: [PATCH v5 2/3] system/memory: Use qemu_ram_move() for directly accessible regions
Quoted 20 lines · Cédric Le Goater
Quoted 10 lines · Peter Maydell
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Maydell
The idea is rather just to ignore the unaligned-access-to-not-RAM case as "only a theoretical issue" and default it to
memmove(). If anybody reports an actual problem we can think about it then :-)Well given the only reported case of virtio DMA into vfio memory we have is GH100 and that one is mapped
MT_NORMAL, I'm fine with that.Quoted 9 lines
Quoted 4 lines
I feel extending vfio to report the mapping type to userspace might not be a bad idea down the road, though. "Directly accessible" is a very rough approximation to the rich variety of mappings available across architectures.
Yes, this sounds like a good longer term idea. If userspace doesn't have a way to know what the mapping behind something it's mmap()ed is then it can't know what operations it can and can't safely do.
Quoted 8 lines
Gavin, didn't you have the idea of using a VFIO region caps for that ?
This is similar to what we did on POWER9 with :
VFIO_REGION_INFO_CAP_NVLINK2_SSATGTwhich exported GPU RAM to the userspace too.
The idea was initially proposed by Michael in the comments for (RFCv1) patch of inheriting the max dma bounce buffer size from the parent [1].
[1] https://lore.kernel.org/qemu-arm/[email protected]/
We can't simply call to glibc::memmove() even it's known that the MemoryRegion is mapped through MT_NORMAL due to issue that one-byte stores have been translated to triple one-byte stores by glibc::memmove() in glibc-2.24+ for aarch64 [2].
[2] https://lore.kernel.org/qemu-devel/[email protected]
Besides, I guess we have similar issue on those MemoryRegion created by memory_region_init_ram_from_{fd, file}. At present, those regions are assumed to be regular RAM (backed by MT_NORMAL). The assumption isn't always correct.
Thanks,
Gavin
PATCH v5 3/3 system/memory: Make ram device region directly accessible
In reply to Gavin ShanThis basically reverts 4a2e242bbb30 ("memory: Don't use memcpy for ram_device regions") to make ram device region directly accessible again. With this, the bounce buffer is bypassed in address_space_map() when a ram device region is involved, potentially avoid to overrun the (small) bounce buffer.
--- include/system/memory.h | 11 ++--------- system/memory.c | 41 +---------------------------------------- system/trace-events | 2 -- 3 files changed, 3 insertions(+), 51 deletions(-)
include/system/memory.h +2 -9
@@ -2718,15 +2718,8 @@ static inline bool memory_region_supports_direct_access(const MemoryRegion *mr)if (memory_region_is_romd (mr)) {return true; } -if (!memory_region_is_ram (mr)) { -return false; - } -/* -* RAM DEVICE regions can be accessed directly using memcpy, but it might -* be MMIO and access using mempy can be wrong (e.g., using instructions not -* intended for MMIO access). So we treat this as IO. -*/ -return !memory_region_is_ram_device (mr); + +return memory_region_is_ram (mr); } static inline bool memory_access_is_direct(const MemoryRegion *mr,
system/memory.c +1 -40
54 lines not on this page · raw message
system/trace-events +0 -2
@@ -20,8 +20,6 @@ memory_region_ops_read(int cpu_index, void *mr, uint64_t addr, uint64_t value, u memory_region_ops_write(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size, const char *name) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u name '%s'" memory_region_subpage_read(int cpu_index, void *mr, uint64_t offset, uint64_t value, unsigned size) "cpu %d mr %p offset 0x%"PRIx64" value 0x%"PRIx64" size %u" memory_region_subpage_write(int cpu_index, void *mr, uint64_t offset, uint64_t value, unsigned size) "cpu %d mr %p offset 0x%"PRIx64" value 0x%"PRIx64" size %u" -memory_region_ram_device_read(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u" -memory_region_ram_device_write(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u" memory_region_sync_dirty(const char *mr, const char *listener, int global) "mr '%s' listener '%s' synced (global=%d)" flatview_new(void *view, void *root) "%p (root %p)" flatview_destroy(void *view, void *root) "%p (root %p)"
Signature · 1 line
2.55.0
Quoted 17 lines · Gavin Shan
All ram device regions was turned to be indirectly accessible by commit
4a2e242bbb("memory: Don't use memcpy forram_deviceregions"). This leads to a frozen guest where a NVidia GH100 GPU is passed from host. The memory in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take DMA bounce buffer inaddress_space_map()to cover the DMA request. However, the bounce buffer size is 4096 bytes only and it's exhaused very quickly when the guest has significant disk activities on compiling 'cuda-samples'. The full log and problem description can be found from PATCH[2/3]'s commit log.Fix the issue handled in commit
4a2e242bbbby replacingmemmove()with newly addedqemu_ram_move()where the aligned and small-sized accesses are handled by qatomics, and fall back tomemmove()otherwise, for the directly accessible regions. With this, we can revert4a2e242bbbto make ram device region directly accessible again and bypass the bounce buffer inaddress_space_map()where the guest hang happens.
Quoted 3 lines
PATCH[1] replaces
memcpy()withmemomve()for directly accessible regions PATCH[2] usesqemu_ram_move()for directly accessible regions PATCH[3] makes ram device region directly accessible again
Queued for 11.2, with fixups suggested by PeterM in v4 discussions:
Fixup:
include/system/memory.h +3 -3
@@ -2677,9 +2677,9 @@ void address_space_unregister_map_client(AddressSpace *as, QEMUBH *bh);* Move @n bytes from @src to @dst, the memory areas may overlap. This * provides the same semantics as memmove(), plus an additional stronger * guarantee: if @n is 1, 2 or 4 or 8 bytes, and @src and @dst are both -* naturally aligned for that access size, and the memory areas do not -* overlap, then both the load and the store will be done as a single -* atomic access (with the semantics of qatomic_read() and qatomic_set()). +* naturally aligned for that access size, then both the load and the store +* will be done as a single atomic access (with the semantics of +* qatomic_read() and qatomic_set()). * * This is the underlying function that we use to implement accesses by * a guest vCPU or a device DMA operation to a ram block. The atomic
system/physmem.c +8 -3
@@ -3162,13 +3162,18 @@ void qemu_ram_move(void *dst, const void *src, size_t n) {uintptr_t test, len; -if (src == dst || n ==0 ) { +if (n ==0 ) {return ; }/* -* Maximal length of aligned access that are determined by @src, -* @dst and @n +* Calculate "the lowest set bit" over @src, @dst and @n, result put +* into @len (which guarantees a power-of-two). With that and the +* later check (len!=n), it makes sure that we will only do the atomic +* ops when: +* +* (1) @n is a power-of-two +* (2) @src and @dst addresses are both aligned to @n */ test = (uintptr_t )src | (uintptr_t )dst | n; len = test & -test;
Signature · 1 line
Peter Xu