Skip to contentSkip to conversation

system/memory: Make ram device region directly accessible

PATCH Superseded Gavin Shan 5 participants 24 messages Jun 12 – Jun 15

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Gavin Shan

author
#0 lore raw

cover system/memory: Make ram device region directly accessible

All ram device regions was turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to a hanged guest where a NVidia GH100 GPU is passed from host. The memory in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take DMA bounce buffer in address_space_map() to cover the DMA request. However, the bounce buffer size is 4096 bytes and we're overrunning it easily when the guest has significant disk activities on compiling 'cuda-samples'. The full log and problem description can be found from PATCH[1/2]'s commit log.

Try to fix the issue handled in commit 4a2e242bbb by replacing mem{cpy, move} with __builtin_mem{cpy, move} in the accessors to the ram device regions. With this, we can basically revert that commit to make ram device region directly accessible again and bypass the bounce buffer in address_space_map() where the guest hang is caused.

PATCH[1] replaces mem{cpy, move} with __builtin_mem{cpy, move} PATCH[2] makes ram device region directly accessible again

Changelog
=========
RFCv1 -> v1:
* https://lists.nongnu.org/archive/html/qemu-arm/2026-06/msg00307.html
* Reworked solution based on suggestions from Peter Xu, Peter Maydell and Michael S. Tsirkin

Gavin Shan (2):
system/memory: Use __builtin_mem{cpy, move} in accessors of ram device
region
system/memory: Make ram device region directly accessible

 hw/remote/vfio-user-obj.c |  4 +--
 include/system/memory.h   | 53 +++++++++++++++++++++++++++++++--------
 system/memory.c           | 41 +-----------------------------
 system/physmem.c          |  8 +++---
 system/trace-events       |  2 --
 5 files changed, 50 insertions(+), 58 deletions(-)
Signature · 1 line
2.54.0
author
#1 lore raw

PATCH 1/2 system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

All ram device regions was turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on compiling 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with a GH100 GPU card.

   host$ lspci | grep GH100
   0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1)
   host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64            \
         -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T         \
         -accel kvm -cpu host -smp cpus=48 -m size=8G                       \
         -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0    \
         -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4     \
         -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0
           :
   guest$ cd cuda-samples/build
   guest$ make -j 20 clean
   guest$ make -j 20
           :
   [ 54%] Linking CUDA executable graphMemoryNodes
   [ 54%] Built target graphMemoryNodes
   <no more output afterwards, guest becomes frozen here>

guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources

   [  555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)

When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 can be presented to the guest through memory hot-add. The page cache can be allocated from the hot added memory blocks when cuda-samples is being compiled. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is a RAM DEVICE region and indirectly accessible in qemu. However, the max bounce bufer size is only 4096 bytes by default. We're running out of that space quickly.

QEMU
====
virtio_blk_handle_output
virtio_blk_handle_vq
virtio_blk_get_request
virtqueue_pop
virtqueue_split_pop
virtqueue_map_desc
address_space_map
memory_access_is_direct # Return false
memory_region_supports_direct_access

(qemu) info mtree
memory-region: pci_bridge_pci

    0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci
      0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4
        0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4
          0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]

This replaces mem{cpy, move} with __builtin_mem{cpy, move} in the memory accessors to ram device memory region, preparatory work to make ram device region directly accessible and bypass the bounce buffer in the DMA path in next patch.

---
 hw/remote/vfio-user-obj.c |  4 ++--
 include/system/memory.h   | 42 ++++++++++++++++++++++++++++++++++++++-
 system/physmem.c          |  8 ++++----
 3 files changed, 47 insertions(+), 7 deletions(-)
hw/remote/vfio-user-obj.c +2 -2
index 87fa7b6572..fe6f661fe2 100644
@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset,
         ram_ptr = memory_region_get_ram_ptr(mr);
 
         if (is_write) {
-            memcpy((ram_ptr + offset), buf, size);
+            address_space_memcpy(ram_ptr + offset, buf, size);
         } else {
-            memcpy(buf, (ram_ptr + offset), size);
+            address_space_memcpy(buf, ram_ptr + offset, size);
         }
 
         return 0;
include/system/memory.h +41 -1

54 lines not on this page · raw message

system/physmem.c +4 -4
index 7bcbf87573..5f46a9d676 100644
@@ -3272,7 +3272,7 @@ static MemTxResult flatview_write_continue_step(MemTxAttrs attrs,
         uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l,
                                                false, true);
 
-        memmove(ram_ptr, buf, *l);
+        address_space_memmove(ram_ptr, buf, *l);
         invalidate_and_set_dirty(mr, mr_addr, *l);
 
         return MEMTX_OK;
@@ -3365,7 +3365,7 @@ static MemTxResult flatview_read_continue_step(MemTxAttrs attrs, uint8_t *buf,
         uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l,
                                                false, false);
 
-        memcpy(buf, ram_ptr, *l);
+        address_space_memcpy(buf, ram_ptr, *l);
 
         return MEMTX_OK;
     }
@@ -3503,8 +3503,8 @@ MemTxResult address_space_write_rom(AddressSpace *as, hwaddr addr,
             l = memory_access_size(mr, l, addr1);
         } else {
             /* ROM/RAM case */
-            void *ram_ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
-            memcpy(ram_ptr, buf, l);
+            address_space_memcpy(qemu_map_ram_ptr(mr->ram_block, addr1),
+                                 buf, l);
             invalidate_and_set_dirty(mr, addr1, l);
         }
         len -= l;
Signature · 1 line
2.54.0
#2 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

Quoted 126 lines · Gavin Shan

All ram device regions was turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on compiling 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with a GH100 GPU card.

   host$ lspci | grep GH100
   0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1)
   host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64            \
         -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T         \
         -accel kvm -cpu host -smp cpus=48 -m size=8G                       \
         -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0    \
         -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4     \
         -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0
           :
   guest$ cd cuda-samples/build
   guest$ make -j 20 clean
   guest$ make -j 20
           :
   [ 54%] Linking CUDA executable graphMemoryNodes
   [ 54%] Built target graphMemoryNodes
   <no more output afterwards, guest becomes frozen here>

guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources

   [  555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)

When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 can be presented to the guest through memory hot-add. The page cache can be allocated from the hot added memory blocks when cuda-samples is being compiled. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is a RAM DEVICE region and indirectly accessible in qemu. However, the max bounce bufer size is only 4096 bytes by default. We're running out of that space quickly.

QEMU
====
virtio_blk_handle_output
virtio_blk_handle_vq
virtio_blk_get_request
virtqueue_pop
virtqueue_split_pop
virtqueue_map_desc
address_space_map
memory_access_is_direct # Return false
memory_region_supports_direct_access

(qemu) info mtree
memory-region: pci_bridge_pci

    0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci
      0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4
        0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4
          0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]

This replaces mem{cpy, move} with __builtin_mem{cpy, move} in the memory accessors to ram device memory region, preparatory work to make ram device region directly accessible and bypass the bounce buffer in the DMA path in next patch.

---
 hw/remote/vfio-user-obj.c |  4 ++--
 include/system/memory.h   | 42 ++++++++++++++++++++++++++++++++++++++-
 system/physmem.c          |  8 ++++----
 3 files changed, 47 insertions(+), 7 deletions(-)
hw/remote/vfio-user-obj.c +2 -2
index 87fa7b6572..fe6f661fe2 100644
@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset,
         ram_ptr = memory_region_get_ram_ptr(mr);
 
         if (is_write) {
-            memcpy((ram_ptr + offset), buf, size);
+            address_space_memcpy(ram_ptr + offset, buf, size);
         } else {
-            memcpy(buf, (ram_ptr + offset), size);
+            address_space_memcpy(buf, ram_ptr + offset, size);
         }
 
         return 0;
include/system/memory.h +33 -0
index 1417132f6d..6bb2e13eea 100644
@@ -2938,6 +2938,46 @@ static inline bool memory_access_is_direct(const MemoryRegion *mr,
     return true;
 }
 
+static inline void address_space_memcpy(void *dest, const void *src, size_t n)
+{
+    switch (n) {
+    case 1:
+        __builtin_memcpy(dest, src, 1);
+        break;
+    case 2:
+        __builtin_memcpy(dest, src, 2);
+        break;
+    case 4:
+        __builtin_memcpy(dest, src, 4);
+        break;
+    case 8:
+        __builtin_memcpy(dest, src, 8);
+        break;
+    default:
+        __builtin_memcpy(dest, src, n);
+    }
+}
+
+static inline void address_space_memmove(void *dest, const void *src, size_t n)
+{
+    switch (n) {
+    case 1:
+        __builtin_memmove(dest, src, 1);
+        break;
+    case 2:
+        __builtin_memmove(dest, src, 2);
+        break;
+    case 4:
+        __builtin_memmove(dest, src, 4);
+        break;
+    case 8:

include/system/memory.h static inline bool memory_access_is_direct(const MemoryRegion *mr, · line 2983

Quoted 10 lines
include/system/memory.h +7 -0
+        __builtin_memmove(dest, src, 8);
+        break;
+    default:
+        __builtin_memmove(dest, src, n);
+    }
+}
+
 /**
  * address_space_read: read from an address space.
  *

The variable length probably should use the regular memcpy/memmove - no reason to bypass fortification for these.

Quoted 44 lines
include/system/memory.h +1 -1
@@ -2970,7 +3010,7 @@ MemTxResult address_space_read(AddressSpace *as, hwaddr addr,
             mr = flatview_translate(fv, addr, &addr1, &l, false, attrs);
             if (len == l && memory_access_is_direct(mr, false, attrs)) {
                 ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
-                memcpy(buf, ptr, len);
+                __builtin_memcpy(buf, ptr, len);
             } else {
                 result = flatview_read_continue(fv, addr, attrs, buf, len,
                                                 addr1, l, mr);
system/physmem.c +4 -4
index 7bcbf87573..5f46a9d676 100644
@@ -3272,7 +3272,7 @@ static MemTxResult flatview_write_continue_step(MemTxAttrs attrs,
         uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l,
                                                false, true);
 
-        memmove(ram_ptr, buf, *l);
+        address_space_memmove(ram_ptr, buf, *l);
         invalidate_and_set_dirty(mr, mr_addr, *l);
 
         return MEMTX_OK;
@@ -3365,7 +3365,7 @@ static MemTxResult flatview_read_continue_step(MemTxAttrs attrs, uint8_t *buf,
         uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l,
                                                false, false);
 
-        memcpy(buf, ram_ptr, *l);
+        address_space_memcpy(buf, ram_ptr, *l);
 
         return MEMTX_OK;
     }
@@ -3503,8 +3503,8 @@ MemTxResult address_space_write_rom(AddressSpace *as, hwaddr addr,
             l = memory_access_size(mr, l, addr1);
         } else {
             /* ROM/RAM case */
-            void *ram_ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
-            memcpy(ram_ptr, buf, l);
+            address_space_memcpy(qemu_map_ram_ptr(mr->ram_block, addr1),
+                                 buf, l);
             invalidate_and_set_dirty(mr, addr1, l);
         }
         len -= l;
Signature · 1 line
2.54.0
author
#3 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

Quoted 133 lines · Michael S. Tsirkin
Quoted 132 lines · Gavin Shan

All ram device regions was turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on compiling 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with a GH100 GPU card.

    host$ lspci | grep GH100
    0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1)
    host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64            \
          -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T         \
          -accel kvm -cpu host -smp cpus=48 -m size=8G                       \
          -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0    \
          -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4     \
          -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0
            :
    guest$ cd cuda-samples/build
    guest$ make -j 20 clean
    guest$ make -j 20
            :
    [ 54%] Linking CUDA executable graphMemoryNodes
    [ 54%] Built target graphMemoryNodes
    <no more output afterwards, guest becomes frozen here>

    guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources
    [  555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)

When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 can be presented to the guest through memory hot-add. The page cache can be allocated from the hot added memory blocks when cuda-samples is being compiled. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is a RAM DEVICE region and indirectly accessible in qemu. However, the max bounce bufer size is only 4096 bytes by default. We're running out of that space quickly.

   QEMU
   ====
   virtio_blk_handle_output
     virtio_blk_handle_vq
       virtio_blk_get_request
         virtqueue_pop
           virtqueue_split_pop
             virtqueue_map_desc
               address_space_map
                 memory_access_is_direct         # Return false
                   memory_region_supports_direct_access

   (qemu) info mtree
   memory-region: pci_bridge_pci
     0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci
       0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4
         0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4
           0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]

This replaces mem{cpy, move} with __builtin_mem{cpy, move} in the memory accessors to ram device memory region, preparatory work to make ram device region directly accessible and bypass the bounce buffer in the DMA path in next patch.

---
  hw/remote/vfio-user-obj.c |  4 ++--
  include/system/memory.h   | 42 ++++++++++++++++++++++++++++++++++++++-
  system/physmem.c          |  8 ++++----
  3 files changed, 47 insertions(+), 7 deletions(-)
hw/remote/vfio-user-obj.c +2 -2
index 87fa7b6572..fe6f661fe2 100644
@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset,
          ram_ptr = memory_region_get_ram_ptr(mr);
 
          if (is_write) {
-            memcpy((ram_ptr + offset), buf, size);
+            address_space_memcpy(ram_ptr + offset, buf, size);
          } else {
-            memcpy(buf, (ram_ptr + offset), size);
+            address_space_memcpy(buf, ram_ptr + offset, size);
          }
 
          return 0;
include/system/memory.h +39 -0
index 1417132f6d..6bb2e13eea 100644
@@ -2938,6 +2938,46 @@ static inline bool memory_access_is_direct(const MemoryRegion *mr,
      return true;
  }
 
+static inline void address_space_memcpy(void *dest, const void *src, size_t n)
+{
+    switch (n) {
+    case 1:
+        __builtin_memcpy(dest, src, 1);
+        break;
+    case 2:
+        __builtin_memcpy(dest, src, 2);
+        break;
+    case 4:
+        __builtin_memcpy(dest, src, 4);
+        break;
+    case 8:
+        __builtin_memcpy(dest, src, 8);
+        break;
+    default:
+        __builtin_memcpy(dest, src, n);
+    }
+}
+
+static inline void address_space_memmove(void *dest, const void *src, size_t n)
+{
+    switch (n) {
+    case 1:
+        __builtin_memmove(dest, src, 1);
+        break;
+    case 2:
+        __builtin_memmove(dest, src, 2);
+        break;
+    case 4:
+        __builtin_memmove(dest, src, 4);
+        break;
+    case 8:
+        __builtin_memmove(dest, src, 8);
+        break;
+    default:
+        __builtin_memmove(dest, src, n);
+    }
+}
Quoted 9 lines

include/system/memory.h static inline bool memory_access_is_direct(const MemoryRegion *mr, · line 2983

Quoted 4 lines
include/system/memory.h +1 -0
+
  /**
   * address_space_read: read from an address space.
   *

The variable length probably should use the regular memcpy/memmove - no reason to bypass fortification for these.

Fixed locally and the changes will be included in next revision.

Quoted 30 lines
Quoted 30 lines
include/system/memory.h +1 -1
@@ -2970,7 +3010,7 @@ MemTxResult address_space_read(AddressSpace *as, hwaddr addr,
              mr = flatview_translate(fv, addr, &addr1, &l, false, attrs);
              if (len == l && memory_access_is_direct(mr, false, attrs)) {
                  ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
-                memcpy(buf, ptr, len);
+                __builtin_memcpy(buf, ptr, len);
              } else {
                  result = flatview_read_continue(fv, addr, attrs, buf, len,
                                                  addr1, l, mr);
system/physmem.c +2 -2
index 7bcbf87573..5f46a9d676 100644
@@ -3272,7 +3272,7 @@ static MemTxResult flatview_write_continue_step(MemTxAttrs attrs,
          uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l,
                                                 false, true);
 
-        memmove(ram_ptr, buf, *l);
+        address_space_memmove(ram_ptr, buf, *l);
          invalidate_and_set_dirty(mr, mr_addr, *l);
 
          return MEMTX_OK;
@@ -3365,7 +3365,7 @@ static MemTxResult flatview_read_continue_step(MemTxAttrs attrs, uint8_t *buf,
          uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l,
                                                 false, false);
 
-        memcpy(buf, ram_ptr, *l);
+        address_space_memcpy(buf, ram_ptr, *l);
 
Quoted 16 lines
Quoted 15 lines
system/physmem.c +2 -2
          return MEMTX_OK;
      }
@@ -3503,8 +3503,8 @@ MemTxResult address_space_write_rom(AddressSpace *as, hwaddr addr,
              l = memory_access_size(mr, l, addr1);
          } else {
              /* ROM/RAM case */
-            void *ram_ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
-            memcpy(ram_ptr, buf, l);
+            address_space_memcpy(qemu_map_ram_ptr(mr->ram_block, addr1),
+                                 buf, l);
              invalidate_and_set_dirty(mr, addr1, l);
          }
          len -= l;
Signature · 1 line
2.54.0

Thanks,
Gavin

#4 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

In reply to Gavin Shan

Hi Gavin,

Quoted 90 lines · Gavin Shan

All ram device regions was turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on compiling 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with a GH100 GPU card.

    host$ lspci | grep GH100
    0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1)
    host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64            \
          -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T         \
          -accel kvm -cpu host -smp cpus=48 -m size=8G                       \
          -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0    \
          -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4     \
          -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0
            :
    guest$ cd cuda-samples/build
    guest$ make -j 20 clean
    guest$ make -j 20
            :
    [ 54%] Linking CUDA executable graphMemoryNodes
    [ 54%] Built target graphMemoryNodes
    <no more output afterwards, guest becomes frozen here>

    guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources
    [  555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)

When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 can be presented to the guest through memory hot-add. The page cache can be allocated from the hot added memory blocks when cuda-samples is being compiled. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is a RAM DEVICE region and indirectly accessible in qemu. However, the max bounce bufer size is only 4096 bytes by default. We're running out of that space quickly.

   QEMU
   ====
   virtio_blk_handle_output
     virtio_blk_handle_vq
       virtio_blk_get_request
         virtqueue_pop
           virtqueue_split_pop
             virtqueue_map_desc
               address_space_map
                 memory_access_is_direct         # Return false
                   memory_region_supports_direct_access

   (qemu) info mtree
   memory-region: pci_bridge_pci
     0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci
       0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4
         0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4
           0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]

This replaces mem{cpy, move} with __builtin_mem{cpy, move} in the memory accessors to ram device memory region, preparatory work to make ram device region directly accessible and bypass the bounce buffer in the DMA path in next patch.

---
  hw/remote/vfio-user-obj.c |  4 ++--
  include/system/memory.h   | 42 ++++++++++++++++++++++++++++++++++++++-
  system/physmem.c          |  8 ++++----
  3 files changed, 47 insertions(+), 7 deletions(-)
hw/remote/vfio-user-obj.c +2 -2
index 87fa7b6572..fe6f661fe2 100644
@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset,
          ram_ptr = memory_region_get_ram_ptr(mr);
 
          if (is_write) {
-            memcpy((ram_ptr + offset), buf, size);
+            address_space_memcpy(ram_ptr + offset, buf, size);
          } else {
-            memcpy(buf, (ram_ptr + offset), size);
+            address_space_memcpy(buf, ram_ptr + offset, size);
          }
 
          return 0;
include/system/memory.h +0 -0
index 1417132f6d..6bb2e13eea 100644

include/system/memory.h static inline bool memory_access_is_direct(const MemoryRegion *mr, · line 2941

Quoted 4 lines
include/system/memory.h +1 -0
@@ -2938,6 +2938,46 @@ static inline bool memory_access_is_direct(const MemoryRegion *mr,
      return true;
  }
 
+static inline void address_space_memcpy(void *dest, const void *src, size_t n)

'address_space_' prefix for something that doesn't use neither AddressSpace nor MemoryRegion is odd.

Maybe prefix 'qemu_ram_' or 'qemu_ram_ptr_' instead? (since the address is returned by memory_region_get_ram_ptr)
Add the definitions in "system/ramblock.h" with that declaration?

Quoted 84 lines
include/system/memory.h +40 -1
+{
+    switch (n) {
+    case 1:
+        __builtin_memcpy(dest, src, 1);
+        break;
+    case 2:
+        __builtin_memcpy(dest, src, 2);
+        break;
+    case 4:
+        __builtin_memcpy(dest, src, 4);
+        break;
+    case 8:
+        __builtin_memcpy(dest, src, 8);
+        break;
+    default:
+        __builtin_memcpy(dest, src, n);
+    }
+}
+
+static inline void address_space_memmove(void *dest, const void *src, size_t n)
+{
+    switch (n) {
+    case 1:
+        __builtin_memmove(dest, src, 1);
+        break;
+    case 2:
+        __builtin_memmove(dest, src, 2);
+        break;
+    case 4:
+        __builtin_memmove(dest, src, 4);
+        break;
+    case 8:
+        __builtin_memmove(dest, src, 8);
+        break;
+    default:
+        __builtin_memmove(dest, src, n);
+    }
+}
+
  /**
   * address_space_read: read from an address space.
   *
@@ -2970,7 +3010,7 @@ MemTxResult address_space_read(AddressSpace *as, hwaddr addr,
              mr = flatview_translate(fv, addr, &addr1, &l, false, attrs);
              if (len == l && memory_access_is_direct(mr, false, attrs)) {
                  ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
-                memcpy(buf, ptr, len);
+                __builtin_memcpy(buf, ptr, len);
              } else {
                  result = flatview_read_continue(fv, addr, attrs, buf, len,
                                                  addr1, l, mr);
system/physmem.c +4 -4
index 7bcbf87573..5f46a9d676 100644
@@ -3272,7 +3272,7 @@ static MemTxResult flatview_write_continue_step(MemTxAttrs attrs,
          uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l,
                                                 false, true);
 
-        memmove(ram_ptr, buf, *l);
+        address_space_memmove(ram_ptr, buf, *l);
          invalidate_and_set_dirty(mr, mr_addr, *l);
 
          return MEMTX_OK;
@@ -3365,7 +3365,7 @@ static MemTxResult flatview_read_continue_step(MemTxAttrs attrs, uint8_t *buf,
          uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l,
                                                 false, false);
 
-        memcpy(buf, ram_ptr, *l);
+        address_space_memcpy(buf, ram_ptr, *l);
 
          return MEMTX_OK;
      }
@@ -3503,8 +3503,8 @@ MemTxResult address_space_write_rom(AddressSpace *as, hwaddr addr,
              l = memory_access_size(mr, l, addr1);
          } else {
              /* ROM/RAM case */
-            void *ram_ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
-            memcpy(ram_ptr, buf, l);
+            address_space_memcpy(qemu_map_ram_ptr(mr->ram_block, addr1),
+                                 buf, l);
              invalidate_and_set_dirty(mr, addr1, l);
          }
          len -= l;
author
#5 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

Hi Philippe,

Quoted 98 lines · Philippe Mathieu-Daudé

Hi Gavin,

Quoted 88 lines · Gavin Shan

All ram device regions was turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on compiling 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with a GH100 GPU card.

    host$ lspci | grep GH100
    0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1) host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64            \
          -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T         \
          -accel kvm -cpu host -smp cpus=48 -m size=8G                       \
          -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0    \
          -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4     \
          -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0
            :
    guest$ cd cuda-samples/build
    guest$ make -j 20 clean
    guest$ make -j 20
            :
    [ 54%] Linking CUDA executable graphMemoryNodes
    [ 54%] Built target graphMemoryNodes
    <no more output afterwards, guest becomes frozen here>

    guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources
    [  555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)

When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 can be presented to the guest through memory hot-add. The page cache can be allocated from the hot added memory blocks when cuda-samples is being compiled. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is a RAM DEVICE region and indirectly accessible in qemu. However, the max bounce bufer size is only 4096 bytes by default. We're running out of that space quickly.

   QEMU
   ====
   virtio_blk_handle_output
     virtio_blk_handle_vq
       virtio_blk_get_request
         virtqueue_pop
           virtqueue_split_pop
             virtqueue_map_desc
               address_space_map memory_access_is_direct         # Return false memory_region_supports_direct_access

   (qemu) info mtree
   memory-region: pci_bridge_pci
     0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci 0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4 0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4 0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]

This replaces mem{cpy, move} with __builtin_mem{cpy, move} in the memory accessors to ram device memory region, preparatory work to make ram device region directly accessible and bypass the bounce buffer in the DMA path in next patch.

---
  hw/remote/vfio-user-obj.c |  4 ++-- include/system/memory.h   | 42 ++++++++++++++++++++++++++++++++++++++- system/physmem.c          |  8 ++++----

  3 files changed, 47 insertions(+), 7 deletions(-)
hw/remote/vfio-user-obj.c +2 -2
index 87fa7b6572..fe6f661fe2 100644
@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset,
           ram_ptr = memory_region_get_ram_ptr(mr);
           if (is_write) {
-            memcpy((ram_ptr + offset), buf, size);
+            address_space_memcpy(ram_ptr + offset, buf, size);
           } else {
-            memcpy(buf, (ram_ptr + offset), size);
+            address_space_memcpy(buf, ram_ptr + offset, size);
           }
           return 0;
include/system/memory.h +0 -0
index 1417132f6d..6bb2e13eea 100644

include/system/memory.h static inline bool memory_access_is_direct(const MemoryRegion *mr, · line 2940

Quoted 3 lines
include/system/memory.h +1 -0
@@ -2938,6 +2938,46 @@ static inline bool memory_access_is_direct(const MemoryRegion *mr,
       return true;
   }
+static inline void address_space_memcpy(void *dest, const void *src, size_t n)

'address_space_' prefix for something that doesn't use neither AddressSpace nor MemoryRegion is odd.

Quoted 4 lines

Maybe prefix 'qemu_ram_' or 'qemu_ram_ptr_' instead? (since the address is returned by memory_region_get_ram_ptr)
Add the definitions in "system/ramblock.h" with that declaration?

Yes, they have been renamed to qemu_ram_{copy, move}() and moved to system/ramblock.h in local repo. The changes will be seen in the next revision.

Quoted 72 lines
Quoted 72 lines
include/system/memory.h +40 -1
+{
+    switch (n) {
+    case 1:
+        __builtin_memcpy(dest, src, 1);
+        break;
+    case 2:
+        __builtin_memcpy(dest, src, 2);
+        break;
+    case 4:
+        __builtin_memcpy(dest, src, 4);
+        break;
+    case 8:
+        __builtin_memcpy(dest, src, 8);
+        break;
+    default:
+        __builtin_memcpy(dest, src, n);
+    }
+}
+
+static inline void address_space_memmove(void *dest, const void *src, size_t n)
+{
+    switch (n) {
+    case 1:
+        __builtin_memmove(dest, src, 1);
+        break;
+    case 2:
+        __builtin_memmove(dest, src, 2);
+        break;
+    case 4:
+        __builtin_memmove(dest, src, 4);
+        break;
+    case 8:
+        __builtin_memmove(dest, src, 8);
+        break;
+    default:
+        __builtin_memmove(dest, src, n);
+    }
+}
+
   /**
    * address_space_read: read from an address space.
    *
@@ -2970,7 +3010,7 @@ MemTxResult address_space_read(AddressSpace *as, hwaddr addr,
               mr = flatview_translate(fv, addr, &addr1, &l, false, attrs);
               if (len == l && memory_access_is_direct(mr, false, attrs)) {
                   ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
-                memcpy(buf, ptr, len);
+                __builtin_memcpy(buf, ptr, len);
               } else {
                   result = flatview_read_continue(fv, addr, attrs, buf, len,
                                                   addr1, l, mr);
system/physmem.c +2 -2
index 7bcbf87573..5f46a9d676 100644
@@ -3272,7 +3272,7 @@ static MemTxResult flatview_write_continue_step(MemTxAttrs attrs,
           uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l,
                                                  false, true);
-        memmove(ram_ptr, buf, *l);
+        address_space_memmove(ram_ptr, buf, *l);
           invalidate_and_set_dirty(mr, mr_addr, *l);
           return MEMTX_OK;
@@ -3365,7 +3365,7 @@ static MemTxResult flatview_read_continue_step(MemTxAttrs attrs, uint8_t *buf,
           uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l,
                                                  false, false);
-        memcpy(buf, ram_ptr, *l);
+        address_space_memcpy(buf, ram_ptr, *l);
           return MEMTX_OK;
       }
@@ -3503,8 +3503,8 @@ MemTxResult address_space_write_rom(AddressSpace *as, hwaddr addr,
               l = memory_access_size(mr, l, addr1);
           } else {
Quoted 9 lines

system/physmem.c MemTxResult address_space_write_rom(AddressSpace *as, hwaddr addr, · line 3510

Quoted 8 lines
system/physmem.c +2 -2
               /* ROM/RAM case */
-            void *ram_ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
-            memcpy(ram_ptr, buf, l);
+            address_space_memcpy(qemu_map_ram_ptr(mr->ram_block, addr1),
+                                 buf, l);
               invalidate_and_set_dirty(mr, addr1, l);
           }
           len -= l;

Thanks,
Gavin

#6 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

In reply to Gavin Shan
Quoted 54 lines · Gavin Shan

All ram device regions was turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on compiling 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with a GH100 GPU card.

    host$ lspci | grep GH100
    0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1)
    host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64            \
          -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T         \
          -accel kvm -cpu host -smp cpus=48 -m size=8G                       \
          -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0    \
          -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4     \
          -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0
            :
    guest$ cd cuda-samples/build
    guest$ make -j 20 clean
    guest$ make -j 20
            :
    [ 54%] Linking CUDA executable graphMemoryNodes
    [ 54%] Built target graphMemoryNodes
    <no more output afterwards, guest becomes frozen here>

    guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources
    [  555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)

When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 can be presented to the guest through memory hot-add. The page cache can be allocated from the hot added memory blocks when cuda-samples is being compiled. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is a RAM DEVICE region and indirectly accessible in qemu. However, the max bounce bufer size is only 4096 bytes by default. We're running out of that space quickly.

   QEMU
   ====
   virtio_blk_handle_output
     virtio_blk_handle_vq
       virtio_blk_get_request
         virtqueue_pop
           virtqueue_split_pop
             virtqueue_map_desc
               address_space_map
                 memory_access_is_direct         # Return false
                   memory_region_supports_direct_access

   (qemu) info mtree
   memory-region: pci_bridge_pci
     0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci
       0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4
         0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4
           0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]
Quoted 4 lines

This replaces mem{cpy, move} with __builtin_mem{cpy, move} in the memory accessors to ram device memory region, preparatory work to make ram device region directly accessible and bypass the bounce buffer in the DMA path in next patch.

memcpy/memmove *always* compile to __builtin_memcpy/memmove, and the compiler later decides whether or not to expand inline.

So, this doesn't do what you think it does.
My real question is: what are you attempting to achieve?

(1) is the problem unaligned access to a mapped physical device?
(2) is the problem vector access to a mapped physical device?
(3) something else?

r~

#7 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

Quoted 9 lines · Richard Henderson
Quoted 4 lines · Gavin Shan

This replaces mem{cpy, move} with __builtin_mem{cpy, move} in the memory accessors to ram device memory region, preparatory work to make ram device region directly accessible and bypass the bounce buffer in the DMA path in next patch.

memcpy/memmove *always* compile to __builtin_memcpy/memmove, and the compiler later decides whether or not to expand inline.

Yes, but if you pass it a fixed small integer, then it is likely to expand it inline, whereas if you pass it a variable then it is likely not to... The patch is attempting to persuade the compiler to definitely do an inline access for 1, 2, 4, 8 byte access.

Quoted 6 lines

So, this doesn't do what you think it does.
My real question is: what are you attempting to achieve?

(1) is the problem unaligned access to a mapped physical device?
(2) is the problem vector access to a mapped physical device?
(3) something else?

I think there are two problems we're trying to fix here:

(1) If a device does e.g. a pci_dma_write() with size 1, we want this to turn into exactly 1 byte write into guest memory, for the normal case where the guest memory is real host RAM. This deals with the e1000 bug where the pci_dma_write() turns into a call to glibc memmove() with size 1 and glibc's implementation turns that into 3 writes of the byte to the same address, and then a guest write/read might interleave badly with the extra writes.[*] Similarly some devices specify that they do definitely do DMA accesses in a particular way (e.g. the Arm SMMU spec says that the SMMU reads 64-bit page table entries as single-copy atomic accesses).

(2) If a vCPU (emulated or KVM) does a 4 byte access to an address which is in the PCI BAR of a vfio-passthrough device, we want this to turn into exactly a 4-byte access to the mmap()ed memory which is the BAR of the host device. This is because that address might be a real hardware device register, so it's important to access it exactly the way the guest vCPU asked for, and not do multiple accesses or misaligned accesses or whatever.

(The reason we make pass-through BARs not "direct access" but instead go via the bounce buffer is that we were working around (2).)

[*] I have not fully investigated the e1000 situation so it's possible that there is some other issue also there, e.g. guest or device model getting barrier semantics wrong. But writing the byte with the "done with this descriptor" bit 3 times rather than once seems like asking for trouble.

thanks
-- PMM

#8 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

Quoted 9 lines · Peter Maydell
Quoted 7 lines · Richard Henderson
Quoted 4 lines · Gavin Shan

This replaces mem{cpy, move} with __builtin_mem{cpy, move} in the memory accessors to ram device memory region, preparatory work to make ram device region directly accessible and bypass the bounce buffer in the DMA path in next patch.

Quoted 8 lines
Quoted 2 lines

memcpy/memmove *always* compile to __builtin_memcpy/memmove, and the compiler later decides whether or not to expand inline.

Yes, but if you pass it a fixed small integer, then it is likely to expand it inline, whereas if you pass it a variable then it is likely not to... The patch is attempting to persuade the compiler to definitely do an inline access for 1, 2, 4, 8 byte access.

Sure, for hosts with unaligned accesses. We still have sparc64 and (some?) riscv64 that don't automatically have such and will compile to more than one host instruction.

Quoted 6 lines
Quoted 5 lines

My real question is: what are you attempting to achieve?

(1) is the problem unaligned access to a mapped physical device?
(2) is the problem vector access to a mapped physical device?
(3) something else?

Quoted 8 lines

I think there are two problems we're trying to fix here:

(1) If a device does e.g. a pci_dma_write() with size 1, we want this to turn into exactly 1 byte write into guest memory, for the normal case where the guest memory is real host RAM. This deals with the e1000 bug where the pci_dma_write() turns into a call to glibc memmove() with size 1 and glibc's implementation turns that into 3 writes of the byte to the same address...

Gotcha. Easily handled by not using memcpy/memmove at all.

	*(char *)ptr = val;

is sufficient for all hosts.

Quoted 9 lines

(2) If a vCPU (emulated or KVM) does a 4 byte access to an address which is in the PCI BAR of a vfio-passthrough device, we want this to turn into exactly a 4-byte access to the mmap()ed memory which is the BAR of the host device. This is because that address might be a real hardware device register, so it's important to access it exactly the way the guest vCPU asked for, and not do multiple accesses or misaligned accesses or whatever.

Quoted 3 lines

(The reason we make pass-through BARs not "direct access" but instead go via the bounce buffer is that we were working around (2).)

That's going to require qatomic and alignment checks.

That's also going to beg the question of the intended behaviour anything that isn't a naturally aligned access of size in {1,2,4,8}: fall back to N operations of the minimum of alignment and size?

r~

#9 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

Quoted 14 lines · Richard Henderson
Quoted 12 lines · Peter Maydell
Quoted 9 lines · Richard Henderson
Quoted 4 lines · Gavin Shan

This replaces mem{cpy, move} with __builtin_mem{cpy, move} in the memory accessors to ram device memory region, preparatory work to make ram device region directly accessible and bypass the bounce buffer in the DMA path in next patch.

memcpy/memmove *always* compile to __builtin_memcpy/memmove, and the compiler later decides whether or not to expand inline.

Quoted 8 lines
Quoted 5 lines

Yes, but if you pass it a fixed small integer, then it is likely to expand it inline, whereas if you pass it a variable then it is likely not to... The patch is attempting to persuade the compiler to definitely do an inline access for 1, 2, 4, 8 byte access.

Sure, for hosts with unaligned accesses. We still have sparc64 and (some?) riscv64 that don't automatically have such and will compile to more than one host instruction.

Good point.

Quoted 15 lines
Quoted 6 lines
Quoted 5 lines

My real question is: what are you attempting to achieve?

(1) is the problem unaligned access to a mapped physical device?
(2) is the problem vector access to a mapped physical device?
(3) something else?

Quoted 8 lines

I think there are two problems we're trying to fix here:

(1) If a device does e.g. a pci_dma_write() with size 1, we want this to turn into exactly 1 byte write into guest memory, for the normal case where the guest memory is real host RAM. This deals with the e1000 bug where the pci_dma_write() turns into a call to glibc memmove() with size 1 and glibc's implementation turns that into 3 writes of the byte to the same address...

Quoted 5 lines

Gotcha. Easily handled by not using memcpy/memmove at all.

        *(char *)ptr = val;

is sufficient for all hosts.

That helps e1000 because it happens to be doing byte writes, but the same effect probably occurs for word writes in other devices...

Quoted 13 lines
Quoted 9 lines

(2) If a vCPU (emulated or KVM) does a 4 byte access to an address which is in the PCI BAR of a vfio-passthrough device, we want this to turn into exactly a 4-byte access to the mmap()ed memory which is the BAR of the host device. This is because that address might be a real hardware device register, so it's important to access it exactly the way the guest vCPU asked for, and not do multiple accesses or misaligned accesses or whatever.

Quoted 3 lines

(The reason we make pass-through BARs not "direct access" but instead go via the bounce buffer is that we were working around (2).)

Quoted 5 lines

That's going to require qatomic and alignment checks.

That's also going to beg the question of the intended behaviour anything that isn't a naturally aligned access of size in {1,2,4,8}:
fall back to N operations of the minimum of alignment and size?

I think in practice we can probably say we don't care and fall back to memmove: anything unaligned here is almost certainly going to real RAM and not going to have any particular ordering requirements (and/or might be caused by the guest misprogramming a device DMA address).

I feel like we handle "unaligned access, maybe need to split it" already in multiple parts of QEMU (for TCG accesses we do that in accel/tcg; system/memory.c has a TODO comment about unaligned accesses at the IO MemoryRegion read/writefn level and there's a patchseries that tries to fix that), so if we find ourselves really needing to do "handle this unaligned access by splitting it into pieces" again here we ought to take a wider look at whether we can consolidate that at all...

thanks
-- PMM

#10 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

In reply to Richard Henderson
Quoted 38 lines · Richard Henderson
Quoted 9 lines · Peter Maydell
Quoted 7 lines · Richard Henderson
Quoted 4 lines · Gavin Shan

This replaces mem{cpy, move} with __builtin_mem{cpy, move} in the memory accessors to ram device memory region, preparatory work to make ram device region directly accessible and bypass the bounce buffer in the DMA path in next patch.

Quoted 8 lines
Quoted 2 lines

memcpy/memmove *always* compile to __builtin_memcpy/memmove, and the compiler later decides whether or not to expand inline.

Yes, but if you pass it a fixed small integer, then it is likely to expand it inline, whereas if you pass it a variable then it is likely not to... The patch is attempting to persuade the compiler to definitely do an inline access for 1, 2, 4, 8 byte access.

Sure, for hosts with unaligned accesses. We still have sparc64 and (some?) riscv64 that don't automatically have such and will compile to more than one host instruction.

Quoted 6 lines
Quoted 5 lines

My real question is: what are you attempting to achieve?

(1) is the problem unaligned access to a mapped physical device?
(2) is the problem vector access to a mapped physical device?
(3) something else?

Quoted 8 lines

I think there are two problems we're trying to fix here:

(1) If a device does e.g. a pci_dma_write() with size 1, we want this to turn into exactly 1 byte write into guest memory, for the normal case where the guest memory is real host RAM. This deals with the e1000 bug where the pci_dma_write() turns into a call to glibc memmove() with size 1 and glibc's implementation turns that into 3 writes of the byte to the same address...

Quoted 5 lines

Gotcha. Easily handled by not using memcpy/memmove at all.

	*(char *)ptr = val;

is sufficient for all hosts.

Yes, I think it does work because we use -fno-strict-aliasing. For bigger sizes we'll need packed because the addresses could be unaligned.

But again, qemu simply already relies on this in bswap.h

I kind of dislike muddying the waters by making several unrelated changes here. If we do we should change bwap too.

Quoted 13 lines
Quoted 9 lines

(2) If a vCPU (emulated or KVM) does a 4 byte access to an address which is in the PCI BAR of a vfio-passthrough device, we want this to turn into exactly a 4-byte access to the mmap()ed memory which is the BAR of the host device. This is because that address might be a real hardware device register, so it's important to access it exactly the way the guest vCPU asked for, and not do multiple accesses or misaligned accesses or whatever.

Quoted 3 lines

(The reason we make pass-through BARs not "direct access" but instead go via the bounce buffer is that we were working around (2).)

Quoted 5 lines

That's going to require qatomic and alignment checks.

That's also going to beg the question of the intended behaviour anything that isn't a naturally aligned access of size in {1,2,4,8}: fall back to N operations of the minimum of alignment and size?

For most host/guest pairs things simply work even for unaligned.

And yes, guest drivers do do this.

On classical pci, there are no transactions as such and an unaligned access will be split anyway.

Quoted 3 lines

r~

#11 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

Quoted 42 lines · Michael S. Tsirkin
Quoted 40 lines · Richard Henderson
Quoted 9 lines · Peter Maydell
Quoted 7 lines · Richard Henderson
Quoted 4 lines · Gavin Shan

This replaces mem{cpy, move} with __builtin_mem{cpy, move} in the memory accessors to ram device memory region, preparatory work to make ram device region directly accessible and bypass the bounce buffer in the DMA path in next patch.

Quoted 8 lines
Quoted 2 lines

memcpy/memmove *always* compile to __builtin_memcpy/memmove, and the compiler later decides whether or not to expand inline.

Yes, but if you pass it a fixed small integer, then it is likely to expand it inline, whereas if you pass it a variable then it is likely not to... The patch is attempting to persuade the compiler to definitely do an inline access for 1, 2, 4, 8 byte access.

Sure, for hosts with unaligned accesses. We still have sparc64 and (some?) riscv64 that don't automatically have such and will compile to more than one host instruction.

Quoted 6 lines
Quoted 5 lines

My real question is: what are you attempting to achieve?

(1) is the problem unaligned access to a mapped physical device?
(2) is the problem vector access to a mapped physical device?
(3) something else?

Quoted 8 lines

I think there are two problems we're trying to fix here:

(1) If a device does e.g. a pci_dma_write() with size 1, we want this to turn into exactly 1 byte write into guest memory, for the normal case where the guest memory is real host RAM. This deals with the e1000 bug where the pci_dma_write() turns into a call to glibc memmove() with size 1 and glibc's implementation turns that into 3 writes of the byte to the same address...

Gotcha. Easily handled by not using memcpy/memmove at all.

Quoted 7 lines
Quoted 3 lines
      *(char *)ptr = val;

is sufficient for all hosts.

Yes, I think it does work because we use -fno-strict-aliasing. For bigger sizes we'll need packed because the addresses could be unaligned.

IIRC "packed" will cause architectures that can't do unaligned word accesses to emit code to do byte accesses, so you don't want that. You need to explicitly check alignment, I think.

Quoted 4 lines

But again, qemu simply already relies on this in bswap.h

I kind of dislike muddying the waters by making several unrelated changes here. If we do we should change bwap too.

The ldl_p etc functions in bswap.h provide different semantics:
ldl_p() is "do a load of a 32-bit quantity, even if the address is not 4-aligned". We don't care if the compiler or the memcpy ends up doing that as 4 byte accesses, which on some hosts it must do.

thanks
-- PMM

#12 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

Quoted 47 lines · Peter Maydell
Quoted 46 lines · Michael S. Tsirkin
Quoted 43 lines · Richard Henderson
Quoted 9 lines · Peter Maydell
Quoted 7 lines · Richard Henderson
Quoted 4 lines · Gavin Shan

This replaces mem{cpy, move} with __builtin_mem{cpy, move} in the memory accessors to ram device memory region, preparatory work to make ram device region directly accessible and bypass the bounce buffer in the DMA path in next patch.

Quoted 8 lines
Quoted 2 lines

memcpy/memmove *always* compile to __builtin_memcpy/memmove, and the compiler later decides whether or not to expand inline.

Yes, but if you pass it a fixed small integer, then it is likely to expand it inline, whereas if you pass it a variable then it is likely not to... The patch is attempting to persuade the compiler to definitely do an inline access for 1, 2, 4, 8 byte access.

Sure, for hosts with unaligned accesses. We still have sparc64 and (some?) riscv64 that don't automatically have such and will compile to more than one host instruction.

Quoted 6 lines
Quoted 5 lines

My real question is: what are you attempting to achieve?

(1) is the problem unaligned access to a mapped physical device?
(2) is the problem vector access to a mapped physical device?
(3) something else?

Quoted 8 lines

I think there are two problems we're trying to fix here:

(1) If a device does e.g. a pci_dma_write() with size 1, we want this to turn into exactly 1 byte write into guest memory, for the normal case where the guest memory is real host RAM. This deals with the e1000 bug where the pci_dma_write() turns into a call to glibc memmove() with size 1 and glibc's implementation turns that into 3 writes of the byte to the same address...

Gotcha. Easily handled by not using memcpy/memmove at all.

      *(char *)ptr = val;

is sufficient for all hosts.

Quoted 7 lines
Quoted 3 lines

Yes, I think it does work because we use -fno-strict-aliasing. For bigger sizes we'll need packed because the addresses could be unaligned.

IIRC "packed" will cause architectures that can't do unaligned word accesses to emit code to do byte accesses, so you don't want that.

I mean, what's left to do if guest wants an unaligned access? Crash? On classical PCI, unaligned accesses are split up by the bus, so there's a decent chance most devices will be ok if qemu does it.

Quoted 8 lines

You need to explicitly check alignment,
I think.

Quoted 4 lines

But again, qemu simply already relies on this in bswap.h

I kind of dislike muddying the waters by making several unrelated changes here. If we do we should change bwap too.

Quoted 8 lines

The ldl_p etc functions in bswap.h provide different semantics:
ldl_p() is "do a load of a 32-bit quantity, even if the address is not 4-aligned". We don't care if the compiler or the memcpy ends up doing that as 4 byte accesses, which on some hosts it must do.

thanks
-- PMM

I mean the ram device ops trick is using bounce buffers precisely because it happens to give you a single load.
And yes on hosts where it does not work, these devices won't work, and qemu can't fix that.

Signature · 1 line
MST
#13 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

In reply to Peter Maydell
Quoted 47 lines · Peter Maydell
Quoted 46 lines · Michael S. Tsirkin
Quoted 43 lines · Richard Henderson
Quoted 9 lines · Peter Maydell
Quoted 7 lines · Richard Henderson
Quoted 4 lines · Gavin Shan

This replaces mem{cpy, move} with __builtin_mem{cpy, move} in the memory accessors to ram device memory region, preparatory work to make ram device region directly accessible and bypass the bounce buffer in the DMA path in next patch.

Quoted 8 lines
Quoted 2 lines

memcpy/memmove *always* compile to __builtin_memcpy/memmove, and the compiler later decides whether or not to expand inline.

Yes, but if you pass it a fixed small integer, then it is likely to expand it inline, whereas if you pass it a variable then it is likely not to... The patch is attempting to persuade the compiler to definitely do an inline access for 1, 2, 4, 8 byte access.

Sure, for hosts with unaligned accesses. We still have sparc64 and (some?) riscv64 that don't automatically have such and will compile to more than one host instruction.

Quoted 6 lines
Quoted 5 lines

My real question is: what are you attempting to achieve?

(1) is the problem unaligned access to a mapped physical device?
(2) is the problem vector access to a mapped physical device?
(3) something else?

Quoted 8 lines

I think there are two problems we're trying to fix here:

(1) If a device does e.g. a pci_dma_write() with size 1, we want this to turn into exactly 1 byte write into guest memory, for the normal case where the guest memory is real host RAM. This deals with the e1000 bug where the pci_dma_write() turns into a call to glibc memmove() with size 1 and glibc's implementation turns that into 3 writes of the byte to the same address...

Gotcha. Easily handled by not using memcpy/memmove at all.

      *(char *)ptr = val;

is sufficient for all hosts.

Quoted 7 lines
Quoted 3 lines

Yes, I think it does work because we use -fno-strict-aliasing. For bigger sizes we'll need packed because the addresses could be unaligned.

IIRC "packed" will cause architectures that can't do unaligned word accesses to emit code to do byte accesses, so you don't want that.

I checked arm32 and while it does that, so does memcpy. We'd have to explicitly code up aligned/unaligned usecases.

But do we really care about device assignment on those hosts?

Maybe, the thing to do is just to ignore the issue.

Quoted 16 lines

You need to explicitly check alignment,
I think.

Quoted 4 lines

But again, qemu simply already relies on this in bswap.h

I kind of dislike muddying the waters by making several unrelated changes here. If we do we should change bwap too.

The ldl_p etc functions in bswap.h provide different semantics:
ldl_p() is "do a load of a 32-bit quantity, even if the address is not 4-aligned". We don't care if the compiler or the memcpy ends up doing that as 4 byte accesses, which on some hosts it must do.

thanks
-- PMM

#14 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

In reply to Michael S. Tsirkin
Quoted 3 lines · Michael S. Tsirkin

Yes, I think it does work because we use -fno-strict-aliasing. For bigger sizes we'll need packed because the addresses could be unaligned.

...

Quoted 6 lines

For most host/guest pairs things simply work even for unaligned.

And yes, guest drivers do do this.

On classical pci, there are no transactions as such and an unaligned access will be split anyway.

I'm saying, if you're talking about pass-through to real devices, that won't work. For instance, AArch64 will trap unaligned accesses to Device memory.

You need to actually handle unaligned. Perhaps something like

    /* Find unit to fit size and alignment of dst */
    uintptr_t test = (uintptr_t)dst | size;
    uintptr_t lsb = test & -test;

    switch (lsb) {
    case 1:   // loop over uint8_t
    case 2:   // loop over uint16_t
    case 4:   // loop over uint32_t
    default:  // loop over uint64_t
    }

with the expectation that normally we'll have aligned addresses and size such that the loop will iterate once.

r~

#15 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

Quoted 7 lines · Richard Henderson
Quoted 3 lines · Michael S. Tsirkin

Yes, I think it does work because we use -fno-strict-aliasing. For bigger sizes we'll need packed because the addresses could be unaligned.

...

Quoted 2 lines

For most host/guest pairs things simply work even for unaligned.

Quoted 7 lines
Quoted 4 lines

And yes, guest drivers do do this.

On classical pci, there are no transactions as such and an unaligned access will be split anyway.

I'm saying, if you're talking about pass-through to real devices, that won't work. For instance, AArch64 will trap unaligned accesses to Device memory.

Presumably, AArch64 drivers don't do unaligned at all then?

Quoted 13 lines

You need to actually handle unaligned. Perhaps something like

    /* Find unit to fit size and alignment of dst */
    uintptr_t test = (uintptr_t)dst | size;
    uintptr_t lsb = test & -test;

    switch (lsb) {
    case 1:   // loop over uint8_t
    case 2:   // loop over uint16_t
    case 4:   // loop over uint32_t
    default:  // loop over uint64_t
    }
Quoted 5 lines

with the expectation that normally we'll have aligned addresses and size such that the loop will iterate once.

r~

And ifdef for arches without unaligned support?

Signature · 1 line
MST
author
#16 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Richard Henderson
Quoted 3 lines · Michael S. Tsirkin

Yes, I think it does work because we use -fno-strict-aliasing. For bigger sizes we'll need packed because the addresses could be unaligned.

...

Quoted 4 lines

For most host/guest pairs things simply work even for unaligned.

And yes, guest drivers do do this.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

On classical pci, there are no transactions as such and an unaligned access will be split anyway.

I'm saying, if you're talking about pass-through to real devices, that won't work. For instance, AArch64 will trap unaligned accesses to Device memory.

Presumably, AArch64 drivers don't do unaligned at all then?

I think Michael is correct because the unaligned access isn't a concern to ram device region. MemroyRegionOPs::impl::unaligned is true for ram device region, which means no unaligned access concerns.

static const MemoryRegionOps ram_device_mem_ops = {
    .read = memory_region_ram_device_read,
    .write = memory_region_ram_device_write,
    .endianness = HOST_BIG_ENDIAN ? DEVICE_BIG_ENDIAN : DEVICE_LITTLE_ENDIAN,
    .valid = {
        .min_access_size = 1,
        .max_access_size = 8,
        .unaligned = true,
    },
    .impl = {
        .min_access_size = 1,
        .max_access_size = 8,
        .unaligned = true,
    },
};

system/physmem.c::memory_access_size
====================================
int memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr)

{
        :
    /* Bound the maximum access by the alignment of the address.  */
    if (!mr->ops->impl.unaligned) {
        unsigned align_size_max = addr & -addr;
        if (align_size_max != 0 && align_size_max < access_size_max) {
            access_size_max = align_size_max;
        }
    }
}

Please help to confirm if we really want to cover the unaligned access for all architectures. After it's confirmed, I can send (v2) for further review.

Quoted 13 lines
Quoted 13 lines

You need to actually handle unaligned. Perhaps something like

     /* Find unit to fit size and alignment of dst */
     uintptr_t test = (uintptr_t)dst | size;
     uintptr_t lsb = test & -test;

     switch (lsb) {
     case 1:   // loop over uint8_t
     case 2:   // loop over uint16_t
     case 4:   // loop over uint32_t
     default:  // loop over uint64_t
     }
Quoted 3 lines
Quoted 3 lines

with the expectation that normally we'll have aligned addresses and size such that the loop will iterate once.

Thanks, Richard. I think it should work if we really want to cover the unaligned access. I tried the following snippet based on yours, my issue can be fixed and no other issues are seen in my environment.

-----> system/physmem.c

+/*
+ * qemu_ram_copy - copy data from ram block
+ *
+ * @dest: destination into which data is copied
+ * @src: source of the data
+ * @n: length of the data to be copied
+ *
+ * This function is friendly to unaligned access.
+ */
+void qemu_ram_copy(void *dest, const void *src, size_t n) +{
+ uintptr_t test, lsb;
+
+ do {

+        test = (uintptr_t)dest | n;
+        lsb = test & -test;
+        switch (lsb) {
+        case 1:
+            *(uint8_t *)dest = *(uint8_t *)src;
+            src += 1;
+            dest += 1;
+            n -= 1;
+            break;
+        case 2:
+            *(uint16_t *)dest = *(uint16_t *)src;
+            src += 2;
+            dest += 2;
+            n -= 2;
+            break;
+        case 4:
+            *(uint32_t *)dest = *(uint32_t *)src;
+            src += 4;
+            dest += 4;
+            n -= 4;
+            break;
+        default:
+            *(uint64_t *)dest = *(uint64_t *)src;
+            src += 8;
+            dest += 8;
+            n -= 8;

+ }
+ } while (n != 0);
+}

-----> include/system/memory.h

+void qemu_ram_copy(void *dest, const void *src, size_t n); +static inline void qemu_ram_move(void *dest, const void *src, size_t n) +{
+ qemu_ram_copy(dest, src, n);
+}

Quoted 5 lines
Quoted 2 lines

r~

And ifdef for arches without unaligned support?

I guess we probably support unaligned access for all architectures or none of them. It will depend on guest if the unaligned access will be triggered :-)

Thanks,
Gavin

#17 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

In reply to Michael S. Tsirkin
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Richard Henderson
Quoted 3 lines · Michael S. Tsirkin

Yes, I think it does work because we use -fno-strict-aliasing. For bigger sizes we'll need packed because the addresses could be unaligned.

...

Quoted 4 lines

For most host/guest pairs things simply work even for unaligned.

And yes, guest drivers do do this.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

On classical pci, there are no transactions as such and an unaligned access will be split anyway.

I'm saying, if you're talking about pass-through to real devices, that won't work. For instance, AArch64 will trap unaligned accesses to Device memory.

Presumably, AArch64 drivers don't do unaligned at all then?

Yes.

Quoted 13 lines
Quoted 13 lines

You need to actually handle unaligned. Perhaps something like

     /* Find unit to fit size and alignment of dst */
     uintptr_t test = (uintptr_t)dst | size;
     uintptr_t lsb = test & -test;

     switch (lsb) {
     case 1:   // loop over uint8_t
     case 2:   // loop over uint16_t
     case 4:   // loop over uint32_t
     default:  // loop over uint64_t
     }
Quoted 7 lines
Quoted 5 lines

with the expectation that normally we'll have aligned addresses and size such that the loop will iterate once.

r~

And ifdef for arches without unaligned support?

No ifdef. All accesses produced by the above are aligned.

r~

#18 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

Quoted 14 lines · Richard Henderson
Quoted 13 lines · Michael S. Tsirkin
Quoted 12 lines · Richard Henderson
Quoted 3 lines · Michael S. Tsirkin

Yes, I think it does work because we use -fno-strict-aliasing. For bigger sizes we'll need packed because the addresses could be unaligned.

...

Quoted 6 lines

For most host/guest pairs things simply work even for unaligned.

And yes, guest drivers do do this.

On classical pci, there are no transactions as such and an unaligned access will be split anyway.

Quoted 22 lines
Quoted 4 lines
Quoted 2 lines

I'm saying, if you're talking about pass-through to real devices, that won't work. For instance, AArch64 will trap unaligned accesses to Device memory.

Presumably, AArch64 drivers don't do unaligned at all then?

Yes.

Quoted 15 lines
Quoted 15 lines

You need to actually handle unaligned. Perhaps something like

     /* Find unit to fit size and alignment of dst */
     uintptr_t test = (uintptr_t)dst | size;
     uintptr_t lsb = test & -test;

     switch (lsb) {
     case 1:   // loop over uint8_t
     case 2:   // loop over uint16_t
     case 4:   // loop over uint32_t
     default:  // loop over uint64_t
     }

with the expectation that normally we'll have aligned addresses and size such that the loop will iterate once.

OK though it is worth looking at assembly and checking how to make it ooptimal.

I don't get why we have default here either, for that we really should use memcpy for a better perf, I think VCPUs can't initiate MMIO transactions >8 bytes.

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

r~

And ifdef for arches without unaligned support?

No ifdef. All accesses produced by the above are aligned.

r~

but I don't think it's a good idea to have this on x86.

x86 does not need this pile of branches, has a popular closed source guest we are all familiar with, famous for it's rich ecosystem of drivers) and it's just barely possible that an x86 guest on x86 host could work as long as we do not break transaction boundaries.

Is

#ifdef HOST_X86_64
#define QEMU_UNALIGNED 1
#else
#define QEMU_UNALIGNED 0
#endif

And then if (QEMU_UNALIGNED) a big deal really?

Signature · 1 line
MST
#19 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

In reply to Richard Henderson
Quoted 38 lines · Richard Henderson
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Richard Henderson
Quoted 3 lines · Michael S. Tsirkin

Yes, I think it does work because we use -fno-strict-aliasing. For bigger sizes we'll need packed because the addresses could be unaligned.

...

Quoted 4 lines

For most host/guest pairs things simply work even for unaligned.

And yes, guest drivers do do this.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

On classical pci, there are no transactions as such and an unaligned access will be split anyway.

I'm saying, if you're talking about pass-through to real devices, that won't work. For instance, AArch64 will trap unaligned accesses to Device memory.

Presumably, AArch64 drivers don't do unaligned at all then?

Yes.

Quoted 17 lines
Quoted 17 lines

You need to actually handle unaligned. Perhaps something like

     /* Find unit to fit size and alignment of dst */
     uintptr_t test = (uintptr_t)dst | size;
     uintptr_t lsb = test & -test;

     switch (lsb) {
     case 1:   // loop over uint8_t
     case 2:   // loop over uint16_t
     case 4:   // loop over uint32_t
     default:  // loop over uint64_t
     }

with the expectation that normally we'll have aligned addresses and size such that the loop will iterate once.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

r~

And ifdef for arches without unaligned support?

No ifdef. All accesses produced by the above are aligned.

r~

So something like:

#if defined(__i386__) || defined(__x86_64__)
#define HOST_UNALIGNED_MMIO_OK 1
#else
#define HOST_UNALIGNED_MMIO_OK 0
#endif

and then we can check that.

Signature · 1 line
MST
author
#20 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

Quoted 50 lines · Michael S. Tsirkin
Quoted 38 lines · Richard Henderson
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Richard Henderson
Quoted 3 lines · Michael S. Tsirkin

Yes, I think it does work because we use -fno-strict-aliasing. For bigger sizes we'll need packed because the addresses could be unaligned.

...

Quoted 4 lines

For most host/guest pairs things simply work even for unaligned.

And yes, guest drivers do do this.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

On classical pci, there are no transactions as such and an unaligned access will be split anyway.

I'm saying, if you're talking about pass-through to real devices, that won't work. For instance, AArch64 will trap unaligned accesses to Device memory.

Presumably, AArch64 drivers don't do unaligned at all then?

Yes.

Quoted 17 lines
Quoted 17 lines

You need to actually handle unaligned. Perhaps something like

      /* Find unit to fit size and alignment of dst */
      uintptr_t test = (uintptr_t)dst | size;
      uintptr_t lsb = test & -test;

      switch (lsb) {
      case 1:   // loop over uint8_t
      case 2:   // loop over uint16_t
      case 4:   // loop over uint32_t
      default:  // loop over uint64_t
      }

with the expectation that normally we'll have aligned addresses and size such that the loop will iterate once.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

r~

And ifdef for arches without unaligned support?

No ifdef. All accesses produced by the above are aligned.

r~

So something like:

Quoted 9 lines

#if defined(__i386__) || defined(__x86_64__)
#define HOST_UNALIGNED_MMIO_OK 1
#else
#define HOST_UNALIGNED_MMIO_OK 0
#endif

and then we can check that.

Thanks for all the comments. A new revision (v2) has been sent for further comments, thanks.

https://lore.kernel.org/qemu-arm/[email protected]/

Thanks,
Gavin

#21 lore raw

Re: [PATCH 1/2] system/memory: Use __builtin_mem{cpy, move} in accessors of ram device region

In reply to Richard Henderson
Quoted 55 lines · Richard Henderson
Quoted 54 lines · Gavin Shan

All ram device regions was turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on compiling 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with a GH100 GPU card.

    host$ lspci | grep GH100
    0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1)
    host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64            \
          -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T         \
          -accel kvm -cpu host -smp cpus=48 -m size=8G                       \
          -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0    \
          -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4     \
          -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0
            :
    guest$ cd cuda-samples/build
    guest$ make -j 20 clean
    guest$ make -j 20
            :
    [ 54%] Linking CUDA executable graphMemoryNodes
    [ 54%] Built target graphMemoryNodes
    <no more output afterwards, guest becomes frozen here>

    guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources
    [  555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)

When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 can be presented to the guest through memory hot-add. The page cache can be allocated from the hot added memory blocks when cuda-samples is being compiled. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is a RAM DEVICE region and indirectly accessible in qemu. However, the max bounce bufer size is only 4096 bytes by default. We're running out of that space quickly.

   QEMU
   ====
   virtio_blk_handle_output
     virtio_blk_handle_vq
       virtio_blk_get_request
         virtqueue_pop
           virtqueue_split_pop
             virtqueue_map_desc
               address_space_map
                 memory_access_is_direct         # Return false
                   memory_region_supports_direct_access

   (qemu) info mtree
   memory-region: pci_bridge_pci
     0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci
       0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4
         0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4
           0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]
Quoted 7 lines
Quoted 4 lines

This replaces mem{cpy, move} with __builtin_mem{cpy, move} in the memory accessors to ram device memory region, preparatory work to make ram device region directly accessible and bypass the bounce buffer in the DMA path in next patch.

memcpy/memmove *always* compile to __builtin_memcpy/memmove, and the compiler later decides whether or not to expand inline.

Did some research.

The issue that prompted use of __builtin_memcpy in bswap.h is musl's fortify headers that Alpine Linux was using back in 2019.

Try this:
git clone --depth=1 https://github.com/jvoisin/fortify-headers /tmp/fortify-headers git -C /tmp/fortify-headers checkout 5aabc7e~1

now:

musl-gcc -O2 -D_FORTIFY_SOURCE=2 -isystem /tmp/fortify-headers/include -S -o- test.c where test.c is:

#include <string.h>
#include <stdint.h>

void test_memcpy(void *dst, const void *src) {
    memcpy(dst, src, 4);
}
void test_builtin_memcpy(void *dst, const void *src) {
    __builtin_memcpy(dst, src, 4);
}

On x86:

 test_memcpy:

     .LFB13:
        .cfi_startproc
        cmpq    %rsi, %rdi
        jnb     .L2
        leaq    4(%rdi), %rax
        cmpq    %rax, %rsi
        jb      .L6
     .L4:                                                                                            
        movl    $4, %edx
        jmp     memcpy
        .p2align 4,,10
        .p2align 3
     .L2:
        cmpq    %rdi, %rsi
        jnb     .L4
        leaq    4(%rsi), %rax
        cmpq    %rax, %rdi
        jb      .L3
        movl    $4, %edx
        jmp     memcpy
     .L6:
        jmp     .L3
        .cfi_endproc
        .section        .text.unlikely
        .cfi_startproc
        .type   test_memcpy.cold, @function
     test_memcpy.cold:
     .LFSB13:
     .L3:
        ud2
        .cfi_endproc
     .LFE13:
        .text
        .size   test_memcpy, .-test_memcpy

A jmp so a function call.
While builtin:

test_builtin_memcpy:
.LFB14:
.cfi_startproc
movl (%rsi), %eax
movl %eax, (%rdi)
ret
.cfi_endproc
.LFE14:
.size test_builtin_memcpy, .-test_builtin_memcpy

So it's a fortify headers Version 1.0 bug. Version 1.1 has the fix.

I guess we should rewrite bswap.h then.

Quoted 9 lines

So, this doesn't do what you think it does.
My real question is: what are you attempting to achieve?

(1) is the problem unaligned access to a mapped physical device?
(2) is the problem vector access to a mapped physical device?
(3) something else?

r~

author
#22 lore raw

PATCH 2/2 system/memory: Make ram device region directly accessible

In reply to Gavin Shan

This basically reverts 4a2e242bbb30 ("memory: Don't use memcpy for ram_device regions") to make ram device region directly accessible again. With this, the bounce buffer is bypassed in address_space_map() when a ram device region is involved, potentially avoid to overrun the bounce buffer.

---
 include/system/memory.h | 11 ++---------
 system/memory.c         | 41 +----------------------------------------
 system/trace-events     |  2 --
 3 files changed, 3 insertions(+), 51 deletions(-)
include/system/memory.h +2 -9
index 6bb2e13eea..3ca6155805 100644
@@ -2914,15 +2914,8 @@ static inline bool memory_region_supports_direct_access(const MemoryRegion *mr)
     if (memory_region_is_romd(mr)) {
         return true;
     }
-    if (!memory_region_is_ram(mr)) {
-        return false;
-    }
-    /*
-     * RAM DEVICE regions can be accessed directly using memcpy, but it might
-     * be MMIO and access using mempy can be wrong (e.g., using instructions not
-     * intended for MMIO access). So we treat this as IO.
-     */
-    return !memory_region_is_ram_device(mr);
+
+    return memory_region_is_ram(mr);
 }
 
 static inline bool memory_access_is_direct(const MemoryRegion *mr,
system/memory.c +1 -40

54 lines not on this page · raw message

system/trace-events +0 -2
index e6e1b61279..34af0a3a1e 100644
@@ -20,8 +20,6 @@ memory_region_ops_read(int cpu_index, void *mr, uint64_t addr, uint64_t value, u
 memory_region_ops_write(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size, const char *name) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u name '%s'"
 memory_region_subpage_read(int cpu_index, void *mr, uint64_t offset, uint64_t value, unsigned size) "cpu %d mr %p offset 0x%"PRIx64" value 0x%"PRIx64" size %u"
 memory_region_subpage_write(int cpu_index, void *mr, uint64_t offset, uint64_t value, unsigned size) "cpu %d mr %p offset 0x%"PRIx64" value 0x%"PRIx64" size %u"
-memory_region_ram_device_read(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u"
-memory_region_ram_device_write(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u"
 memory_region_sync_dirty(const char *mr, const char *listener, int global) "mr '%s' listener '%s' synced (global=%d)"
 flatview_new(void *view, void *root) "%p (root %p)"
 flatview_destroy(void *view, void *root) "%p (root %p)"
Signature · 1 line
2.54.0
author
#23 lore raw
In reply to Gavin Shan

[Cc Liu Gang and Ding hui]

Quoted 10 lines · Gavin Shan

All ram device regions was turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to a hanged guest where a NVidia GH100 GPU is passed from host. The memory in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take DMA bounce buffer in address_space_map() to cover the DMA request. However, the bounce buffer size is 4096 bytes and we're overrunning it easily when the guest has significant disk activities on compiling 'cuda-samples'. The full log and problem description can be found from PATCH[1/2]'s commit log.

Quoted 9 lines

Try to fix the issue handled in commit 4a2e242bbb by replacing mem{cpy, move} with __builtin_mem{cpy, move} in the accessors to the ram device regions. With this, we can basically revert that commit to make ram device region directly accessible again and bypass the bounce buffer in address_space_map() where the guest hang is caused.

PATCH[1] replaces mem{cpy, move} with __builtin_mem{cpy, move} PATCH[2] makes ram device region directly accessible again

Liu and Ding, Could you give this series a try to see if your e1000 issue gets fixed by this?

https://lore.kernel.org/qemu-devel/[email protected]/

Quoted 12 lines

Changelog
=========
RFCv1 -> v1:
* https://lists.nongnu.org/archive/html/qemu-arm/2026-06/msg00307.html
* Reworked solution based on suggestions from Peter Xu, Peter Maydell and Michael S. Tsirkin

Gavin Shan (2):
system/memory: Use __builtin_mem{cpy, move} in accessors of ram device
region
system/memory: Make ram device region directly accessible

Quoted 7 lines
  hw/remote/vfio-user-obj.c |  4 +--
  include/system/memory.h   | 53 +++++++++++++++++++++++++++++++--------
  system/memory.c           | 41 +-----------------------------
  system/physmem.c          |  8 +++---
  system/trace-events       |  2 --
  5 files changed, 50 insertions(+), 58 deletions(-)

Thanks,
Gavin