Skip to contentSkip to conversation

system/memory: Make ram device region directly accessible

PATCH Superseded Gavin Shan 4 participants 33 messages Jun 15 – Jun 16

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Gavin Shan

author
#0 lore raw

v2 cover system/memory: Make ram device region directly accessible

All ram device regions was turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to a hanged guest where a NVidia GH100 GPU is passed from host. The memory in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take DMA bounce buffer in address_space_map() to cover the DMA request. However, the bounce buffer size is 4096 bytes and we're overrunning it easily when the guest has significant disk activities on compiling 'cuda-samples'. The full log and problem description can be found from PATCH[1/2]'s commit log.

Try to fix the issue handled in commit 4a2e242bbb by replacing memcopy()/ memmove() with newly added helpers qemu_ram_{copy, move}() that works on top of __builtin_{memcpy, memmove} or unaligned access friendly memory movement in the accessors to the ram device regions. With this, we can basically revert that commit to make ram device region directly accessible again and bypass the bounce buffer in address_space_map() where the guest hang is caused.

PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors PATCH[2] makes ram device region directly accessible again

Changelog
=========
v1 -> v2:
* https://lore.kernel.org/qemu-arm/[email protected]/
* Rename address_space_{memcpy, memmove}() to qemu_ram_{copy, move}()
and move them to physmem.c and memory.h (Philippe)
* Use memcpy() and memmove() in qemu_ram_{copy, move}() for the variable
length case (Miachel)
* Handle unaligned access in qemu_ram_{copy, move}() for all archs
except i386 and x86_64 (Richard/Michael)
RFCv1 -> v1:
* https://lists.nongnu.org/archive/html/qemu-arm/2026-06/msg00307.html
* Reworked solution based on suggestions from Peter Xu, Peter Maydell and Michael S. Tsirkin

Gavin Shan (2):
system/memory: Use qemu_ram_{copy, move}() in ram device region
accessors
system/memory: Make ram device region directly accessible

 hw/remote/vfio-user-obj.c |  4 +-
 include/system/memory.h   | 15 +++----
 system/memory.c           | 41 +-----------------
 system/physmem.c          | 91 +++++++++++++++++++++++++++++++++++++--
 system/trace-events       |  2 -
 5 files changed, 95 insertions(+), 58 deletions(-)
Signature · 1 line
2.54.0
author
#1 lore raw

PATCH v2 1/2 system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

All ram device regions were turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.

   host$ lspci | grep GH100
   0009:01:00.0 3D controller: NVIDIA Corporation GH100 [GH200 120GB / 480GB] (rev a1)
   host$ /home/sandbox/gavin/qemu.main/build/qemu-system-aarch64            \
         -machine virt,gic-version=host,ras=on,highmem-mmio-size=4T         \
         -accel kvm -cpu host -smp cpus=48 -m size=8G                       \
         -drive file=/home/gavin/sandbox/images/disk.qcow2,if=none,id=d0    \
         -device virtio-blk-pci,id=vb0,bus=pcie.0,drive=d0,num-queues=4     \
         -device vfio-pci-nohotplug,host=0009:01:00.0,bus=pcie.1.0
           :
   guest$ cd cuda-samples/build
   guest$ make -j 20 clean
   guest$ make -j 20
           :
   [ 54%] Linking CUDA executable graphMemoryNodes
   [ 54%] Built target graphMemoryNodes
   <no more output afterwards, guest becomes frozen here>

guest$ qemu-system-aarch64: virtio: bogus descriptor or out of resources

   [  555.814025] virtio_blk virtio0: [vda] new size: 268435456 512-byte logical blocks (137 GB/128 GiB)

When the GPU's driver (NVidia open driver) is loaded on guest bootup, the memory blocks residing in the PCI BAR#4 of the GH100 GPU card can be presented to the guest through memory hot-add. The page cache can then be allocated from the hot added memory blocks when cuda-samples is being built. Afterwards, the page cache is sent to QEMU's virtio-blk device as part of the DMA request, the bounce buffer has to be used to accomodate the request as the corresponding memory region (MemoryRegion) is an indirectly accessible ram device region in qemu. However, the max bounce bufer size is only 4096 bytes by default and that is exhausted quickly, leading to a reset on the virtio-blk device and frozen guest eventually.

QEMU
====
virtio_blk_handle_output
virtio_blk_handle_vq
virtio_blk_get_request
virtqueue_pop
virtqueue_split_pop
virtqueue_map_desc
address_space_map
memory_access_is_direct # Return false
memory_region_supports_direct_access

(qemu) info mtree
memory-region: pci_bridge_pci

    0000000000000000-ffffffffffffffff (prio 0, container): pci_bridge_pci
      0000042000000000-0000043fffffffff (prio 1, i/o): 0009:01:00.0 base BAR 4
        0000042000000000-0000043fffffffff (prio 0, i/o): 0009:01:00.0 BAR 4
          0000042000000000-000004379fffffff (prio 0, ramd): 0009:01:00.0 BAR 4 mmaps[0]

This adds qemu_ram_{copy, move}() and replaces {memcpy, memmove}() with them in the ram device memory region accessors, similar to what's done in commit 4a2e242bbb so that the issue (MMIO access instructions were optimized to SSE instructions) covered by that commit is fixed. This makes 'ram_device_mem_ops' redundant, paving the way to revert that commit to make ram device region directly accessible again in the next patch.

---
v2: Rename address_space_{memcpy, memmove}() to qemu_ram_{copy, move}() and move them to physmem.c and memory.h, the unaligned access is also covered for all architectures except i386 and x86_64 as Richard suggested.

---
 hw/remote/vfio-user-obj.c |  4 +-
 include/system/memory.h   |  4 +-
 system/physmem.c          | 91 +++++++++++++++++++++++++++++++++++++--
 3 files changed, 92 insertions(+), 7 deletions(-)
hw/remote/vfio-user-obj.c +2 -2
index 87fa7b6572..97a6c88780 100644
@@ -375,9 +375,9 @@ static int vfu_object_mr_rw(MemoryRegion *mr, uint8_t *buf, hwaddr offset,
         ram_ptr = memory_region_get_ram_ptr(mr);
 
         if (is_write) {
-            memcpy((ram_ptr + offset), buf, size);
+            qemu_ram_copy(ram_ptr + offset, buf, size);
         } else {
-            memcpy(buf, (ram_ptr + offset), size);
+            qemu_ram_copy(buf, ram_ptr + offset, size);
         }
 
         return 0;
include/system/memory.h +3 -1
index 1417132f6d..5878727d09 100644
@@ -2897,6 +2897,8 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);
 void address_space_unregister_map_client(AddressSpace *as, QEMUBH *bh);
 
 /* Internal functions, part of the implementation of address_space_read.  */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+void qemu_ram_move(void *dest, const void *src, size_t n);
 MemTxResult address_space_read_full(AddressSpace *as, hwaddr addr,
                                     MemTxAttrs attrs, void *buf, hwaddr len);
 MemTxResult flatview_read_continue(FlatView *fv, hwaddr addr,
@@ -2970,7 +2972,7 @@ MemTxResult address_space_read(AddressSpace *as, hwaddr addr,
             mr = flatview_translate(fv, addr, &addr1, &l, false, attrs);
             if (len == l && memory_access_is_direct(mr, false, attrs)) {
                 ptr = qemu_map_ram_ptr(mr->ram_block, addr1);
-                memcpy(buf, ptr, len);
+                qemu_ram_copy(buf, ptr, len);
             } else {
                 result = flatview_read_continue(fv, addr, attrs, buf, len,
                                                 addr1, l, mr);
system/physmem.c +87 -4

115 lines not on this page · raw message

Signature · 1 line
2.54.0
#2 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 12 lines · Gavin Shan

All ram device regions were turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.

include/system/memory.h +0 -0
index 1417132f6d..5878727d09 100644

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2901

Quoted 5 lines
include/system/memory.h +2 -0
@@ -2897,6 +2897,8 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);
 void address_space_unregister_map_client(AddressSpace *as, QEMUBH *bh);
 
 /* Internal functions, part of the implementation of address_space_read.  */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+void qemu_ram_move(void *dest, const void *src, size_t n);

New function prototypes in include headers need documentation comments.

In particular for these, it's really important that we clearly say what semantics we are attempting to provide with them, so that
(a) when we're reviewing or later updating the implementation we know what we are trying to provide
(b) when we're looking at the callsites we know what the function is guaranteeing to us and what it is not, and thus whether it's OK to use it or we need something els.

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);

Quoted 5 lines
include/system/memory.h +5 -0
+#if defined(__i386__) || defined(__x86_64__)
+#define HOST_UNALIGNED_MMIO_OK 1
+#else
+#define HOST_UNALIGNED_MMIO_OK 0
+#endif

We need to do something better than this. We can't just say "oh, we trust that on x86 this works": it is neither actually true that the compiler guarantees it even on x86, nor is it the case that only x86 can do unaligned accesses to normal RAM.

thanks
-- PMM

#3 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 41 lines · Peter Maydell
Quoted 12 lines · Gavin Shan

All ram device regions were turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.

include/system/memory.h +0 -0
index 1417132f6d..5878727d09 100644

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2901

Quoted 5 lines
include/system/memory.h +2 -0
@@ -2897,6 +2897,8 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);
 void address_space_unregister_map_client(AddressSpace *as, QEMUBH *bh);
 
 /* Internal functions, part of the implementation of address_space_read.  */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+void qemu_ram_move(void *dest, const void *src, size_t n);

New function prototypes in include headers need documentation comments.

In particular for these, it's really important that we clearly say what semantics we are attempting to provide with them, so that
(a) when we're reviewing or later updating the implementation we know what we are trying to provide
(b) when we're looking at the callsites we know what the function is guaranteeing to us and what it is not, and thus whether it's OK to use it or we need something els.

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);

Quoted 5 lines
include/system/memory.h +5 -0
+#if defined(__i386__) || defined(__x86_64__)
+#define HOST_UNALIGNED_MMIO_OK 1
+#else
+#define HOST_UNALIGNED_MMIO_OK 0
+#endif

We need to do something better than this. We can't just say "oh, we trust that on x86 this works": it is neither actually true that the compiler guarantees it even on x86, nor is it the case that only x86 can do unaligned accesses to normal RAM.

Quoted 2 lines

thanks
-- PMM

Why normal RAM? It explicitly says MMIO.
x86 seems to be the only arch that can do unaligned MMIO?

#4 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

In reply to Peter Maydell
Quoted 29 lines · Peter Maydell
Quoted 12 lines · Gavin Shan

All ram device regions were turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.

include/system/memory.h +0 -0
index 1417132f6d..5878727d09 100644

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2901

Quoted 5 lines
include/system/memory.h +2 -0
@@ -2897,6 +2897,8 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);
 void address_space_unregister_map_client(AddressSpace *as, QEMUBH *bh);
 
 /* Internal functions, part of the implementation of address_space_read.  */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+void qemu_ram_move(void *dest, const void *src, size_t n);

New function prototypes in include headers need documentation comments.

In particular for these, it's really important that we clearly say what semantics we are attempting to provide with them, so that
(a) when we're reviewing or later updating the implementation we know what we are trying to provide
(b) when we're looking at the callsites we know what the function is guaranteeing to us and what it is not, and thus whether it's OK to use it or we need something els.

Quoted 9 lines

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);

Quoted 5 lines
include/system/memory.h +5 -0
+#if defined(__i386__) || defined(__x86_64__)
+#define HOST_UNALIGNED_MMIO_OK 1
+#else
+#define HOST_UNALIGNED_MMIO_OK 0
+#endif

We need to do something better than this. We can't just say "oh, we trust that on x86 this works": it is neither actually true that the compiler guarantees it

sorry guarantee what?

Quoted 6 lines

even
on x86, nor is it the case that only x86 can do unaligned accesses to normal RAM.

thanks
-- PMM

#5 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 41 lines · Michael S. Tsirkin
Quoted 29 lines · Peter Maydell
Quoted 12 lines · Gavin Shan

All ram device regions were turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.

include/system/memory.h +0 -0
index 1417132f6d..5878727d09 100644

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2901

Quoted 5 lines
include/system/memory.h +2 -0
@@ -2897,6 +2897,8 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);
 void address_space_unregister_map_client(AddressSpace *as, QEMUBH *bh);
 
 /* Internal functions, part of the implementation of address_space_read.  */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+void qemu_ram_move(void *dest, const void *src, size_t n);

New function prototypes in include headers need documentation comments.

In particular for these, it's really important that we clearly say what semantics we are attempting to provide with them, so that
(a) when we're reviewing or later updating the implementation we know what we are trying to provide
(b) when we're looking at the callsites we know what the function is guaranteeing to us and what it is not, and thus whether it's OK to use it or we need something els.

Quoted 9 lines

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);

Quoted 5 lines
include/system/memory.h +5 -0
+#if defined(__i386__) || defined(__x86_64__)
+#define HOST_UNALIGNED_MMIO_OK 1
+#else
+#define HOST_UNALIGNED_MMIO_OK 0
+#endif

We need to do something better than this. We can't just say "oh, we trust that on x86 this works": it is neither actually true that the compiler guarantees it

Quoted 1 line

sorry guarantee what?

Well, that's part of my point about "we need a doc comment":
what exactly are we trying to guarantee ? But whatever it is, "hardcoded ifdef that privileges x86" is definitely the wrong answer.

-- PMM

author
#6 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Hi Peter and Michael,

Quoted 42 lines · Peter Maydell
Quoted 41 lines · Michael S. Tsirkin
Quoted 38 lines · Peter Maydell
Quoted 12 lines · Gavin Shan

All ram device regions were turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.

include/system/memory.h +0 -0
index 1417132f6d..5878727d09 100644

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2901

Quoted 5 lines
include/system/memory.h +2 -0
@@ -2897,6 +2897,8 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);
  void address_space_unregister_map_client(AddressSpace *as, QEMUBH *bh);
 
  /* Internal functions, part of the implementation of address_space_read.  */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+void qemu_ram_move(void *dest, const void *src, size_t n);

New function prototypes in include headers need documentation comments.

In particular for these, it's really important that we clearly say what semantics we are attempting to provide with them, so that
(a) when we're reviewing or later updating the implementation we know what we are trying to provide
(b) when we're looking at the callsites we know what the function is guaranteeing to us and what it is not, and thus whether it's OK to use it or we need something els.

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);

Quoted 5 lines
include/system/memory.h +5 -0
+#if defined(__i386__) || defined(__x86_64__)
+#define HOST_UNALIGNED_MMIO_OK 1
+#else
+#define HOST_UNALIGNED_MMIO_OK 0
+#endif

We need to do something better than this. We can't just say "oh, we trust that on x86 this works": it is neither actually true that the compiler guarantees it

Quoted 7 lines
Quoted 1 line

sorry guarantee what?

Well, that's part of my point about "we need a doc comment":
what exactly are we trying to guarantee ? But whatever it is, "hardcoded ifdef that privileges x86" is definitely the wrong answer.

Could you please check the following comments (documentation context) for the added functions look good to you? Please let me know if there are anything can be improved.

+
+/**
+ * qemu_ram_copy: copy data to a RAMBlock
+ *
+ * @dest: destination where the data is copied to. It's the pointer returned
+ * by ramblock_ptr() and its variants.
+ * @src: source where the data is copied from. It can be a regular memory block
+ * or a pointer returned by ramblock_ptr() and its variants.
+ * @n: length of data to be copied
+ *
+ * The destination is always a pointer to data area of a RAMBlock which can
+ * be for a regular memory block or a MMIO region. The source can be a pointer
+ * to a regular memory block or a MMIO region. Atomicity isn't guranteed by
+ * memcpy() to copy data between those MMIO regions as we do in this function.
+ * Besides, unaligned accesses are well handled on all architectures except
+ * i386 and x86_64 where the unaligned accesses are supported by the CPUs.
+ *
+ * The ensured atomicity and alignment consideration, which aren't needed
+ * by data copy between two regular memory blocks. So performance penalty
+ * is introduced by this function in such circumstance.
+ */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+
+/**
+ * qemu_ram_move: move data to a RAMBlock
+ *
+ * @dest: destination where the data is moved to. It's the pointer returned
+ * by ramblock_ptr() and its variants.
+ * @src: source where the data is moved from. It can be a regular memory block
+ * or a pointer returned by ramblock_ptr() and its variants.
+ * @n: length of data to be moved
+ *
+ * The destination is always a pointer to data area of a RAMBlock which can
+ * be for a regular memory block or a MMIO region. The source can be a pointer
+ * to a regular memory block or a MMIO region. Atomicity isn't guranteed by
+ * memmove() to move data from or to those MMIO regions as we do in this
+ * function. Besides, unaligned accesses are well handled on all architectures
+ * except i386 and x86_64 where the unaligned accesses are supported by the
+ * CPUs.
+ *
+ * The ensured atomicity and alignment consideration, which aren't needed
+ * by data movement between two regular memory blocks. So performance penalty
+ * is introduced by this function in such circumstance.
+ */
+void qemu_ram_move(void *dest, const void *src, size_t n);
+

Quoted 2 lines

-- PMM

Thanks,
Gavin

#7 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 93 lines · Gavin Shan

Hi Peter and Michael,

Quoted 42 lines · Peter Maydell
Quoted 41 lines · Michael S. Tsirkin
Quoted 38 lines · Peter Maydell
Quoted 12 lines · Gavin Shan

All ram device regions were turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.

include/system/memory.h +0 -0
index 1417132f6d..5878727d09 100644

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2901

Quoted 5 lines
include/system/memory.h +2 -0
@@ -2897,6 +2897,8 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);
  void address_space_unregister_map_client(AddressSpace *as, QEMUBH *bh);
 
  /* Internal functions, part of the implementation of address_space_read.  */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+void qemu_ram_move(void *dest, const void *src, size_t n);

New function prototypes in include headers need documentation comments.

In particular for these, it's really important that we clearly say what semantics we are attempting to provide with them, so that
(a) when we're reviewing or later updating the implementation we know what we are trying to provide
(b) when we're looking at the callsites we know what the function is guaranteeing to us and what it is not, and thus whether it's OK to use it or we need something els.

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);

Quoted 5 lines
include/system/memory.h +5 -0
+#if defined(__i386__) || defined(__x86_64__)
+#define HOST_UNALIGNED_MMIO_OK 1
+#else
+#define HOST_UNALIGNED_MMIO_OK 0
+#endif

We need to do something better than this. We can't just say "oh, we trust that on x86 this works": it is neither actually true that the compiler guarantees it

Quoted 7 lines
Quoted 1 line

sorry guarantee what?

Well, that's part of my point about "we need a doc comment":
what exactly are we trying to guarantee ? But whatever it is, "hardcoded ifdef that privileges x86" is definitely the wrong answer.

Could you please check the following comments (documentation context) for the added functions look good to you? Please let me know if there are anything can be improved.

(quoted diff) +36 -0
+
+/**
+ * qemu_ram_copy: copy data to a RAMBlock
+ *
+ * @dest: destination where the data is copied to. It's the pointer returned
+ *        by ramblock_ptr() and its variants.
+ * @src: source where the data is copied from. It can be a regular memory block
+ *       or a pointer returned by ramblock_ptr() and its variants.
+ * @n: length of data to be copied
+ *
+ * The destination is always a pointer to data area of a RAMBlock which can
+ * be for a regular memory block or a MMIO region. The source can be a pointer
+ * to a regular memory block or a MMIO region. Atomicity isn't guranteed by
+ * memcpy() to copy data between those MMIO regions as we do in this function.
+ * Besides, unaligned accesses are well handled on all architectures except
+ * i386 and x86_64 where the unaligned accesses are supported by the CPUs.
+ *
+ * The ensured atomicity and alignment consideration, which aren't needed
+ * by data copy between two regular memory blocks. So performance penalty
+ * is introduced by this function in such circumstance.
+ */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+
+/**
+ * qemu_ram_move: move data to a RAMBlock
+ *
+ * @dest: destination where the data is moved to. It's the pointer returned
+ *        by ramblock_ptr() and its variants.
+ * @src: source where the data is moved from. It can be a regular memory block
+ *       or a pointer returned by ramblock_ptr() and its variants.
+ * @n: length of data to be moved
+ *
+ * The destination is always a pointer to data area of a RAMBlock which can
+ * be for a regular memory block or a MMIO region. The source can be a pointer
+ * to a regular memory block or a MMIO region. Atomicity isn't guranteed by
+ * memmove() to move data from or to those MMIO regions as we do in this

Review

Quoted 10 lines
(quoted diff) +10 -0
+ * function. Besides, unaligned accesses are well handled on all architectures
+ * except i386 and x86_64 where the unaligned accesses are supported by the
+ * CPUs.
+ *
+ * The ensured atomicity and alignment consideration, which aren't needed
+ * by data movement between two regular memory blocks. So performance penalty
+ * is introduced by this function in such circumstance.
+ */
+void qemu_ram_move(void *dest, const void *src, size_t n);
+

I apologize, I don't understand what these comments are saying, at all.

Quoted 5 lines
Quoted 2 lines

-- PMM

Thanks,
Gavin

author
#8 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 99 lines · Michael S. Tsirkin
Quoted 98 lines · Gavin Shan

Hi Peter and Michael,

Quoted 42 lines · Peter Maydell
Quoted 41 lines · Michael S. Tsirkin
Quoted 38 lines · Peter Maydell
Quoted 12 lines · Gavin Shan

All ram device regions were turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.

include/system/memory.h +0 -0
index 1417132f6d..5878727d09 100644

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2901

Quoted 5 lines
include/system/memory.h +2 -0
@@ -2897,6 +2897,8 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);
   void address_space_unregister_map_client(AddressSpace *as, QEMUBH *bh);
 
   /* Internal functions, part of the implementation of address_space_read.  */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+void qemu_ram_move(void *dest, const void *src, size_t n);

New function prototypes in include headers need documentation comments.

In particular for these, it's really important that we clearly say what semantics we are attempting to provide with them, so that
(a) when we're reviewing or later updating the implementation we know what we are trying to provide
(b) when we're looking at the callsites we know what the function is guaranteeing to us and what it is not, and thus whether it's OK to use it or we need something els.

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);

Quoted 5 lines
include/system/memory.h +5 -0
+#if defined(__i386__) || defined(__x86_64__)
+#define HOST_UNALIGNED_MMIO_OK 1
+#else
+#define HOST_UNALIGNED_MMIO_OK 0
+#endif

We need to do something better than this. We can't just say "oh, we trust that on x86 this works": it is neither actually true that the compiler guarantees it

Quoted 7 lines
Quoted 1 line

sorry guarantee what?

Well, that's part of my point about "we need a doc comment":
what exactly are we trying to guarantee ? But whatever it is, "hardcoded ifdef that privileges x86" is definitely the wrong answer.

Could you please check the following comments (documentation context) for the added functions look good to you? Please let me know if there are anything can be improved.

(quoted diff) +41 -0
+
+/**
+ * qemu_ram_copy: copy data to a RAMBlock
+ *
+ * @dest: destination where the data is copied to. It's the pointer returned
+ *        by ramblock_ptr() and its variants.
+ * @src: source where the data is copied from. It can be a regular memory block
+ *       or a pointer returned by ramblock_ptr() and its variants.
+ * @n: length of data to be copied
+ *
+ * The destination is always a pointer to data area of a RAMBlock which can
+ * be for a regular memory block or a MMIO region. The source can be a pointer
+ * to a regular memory block or a MMIO region. Atomicity isn't guranteed by
+ * memcpy() to copy data between those MMIO regions as we do in this function.
+ * Besides, unaligned accesses are well handled on all architectures except
+ * i386 and x86_64 where the unaligned accesses are supported by the CPUs.
+ *
+ * The ensured atomicity and alignment consideration, which aren't needed
+ * by data copy between two regular memory blocks. So performance penalty
+ * is introduced by this function in such circumstance.
+ */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+
+/**
+ * qemu_ram_move: move data to a RAMBlock
+ *
+ * @dest: destination where the data is moved to. It's the pointer returned
+ *        by ramblock_ptr() and its variants.
+ * @src: source where the data is moved from. It can be a regular memory block
+ *       or a pointer returned by ramblock_ptr() and its variants.
+ * @n: length of data to be moved
+ *
+ * The destination is always a pointer to data area of a RAMBlock which can
+ * be for a regular memory block or a MMIO region. The source can be a pointer
+ * to a regular memory block or a MMIO region. Atomicity isn't guranteed by
+ * memmove() to move data from or to those MMIO regions as we do in this
+ * function. Besides, unaligned accesses are well handled on all architectures
+ * except i386 and x86_64 where the unaligned accesses are supported by the
+ * CPUs.
+ *
+ * The ensured atomicity and alignment consideration, which aren't needed
Quoted 9 lines

Review

Quoted 5 lines

+ * by data movement between two regular memory blocks. So performance penalty
+ * is introduced by this function in such circumstance.
+ */

(quoted diff) +2 -0
+void qemu_ram_move(void *dest, const void *src, size_t n);
+

I apologize, I don't understand what these comments are saying, at all.

Sorry to hear that, Michael. There are two things done in the newly added function qemu_ram_{copy, move}(), comparing to the original memcpy() and memmove():

- Data copy or movement on MMIO region(s) using __bultin_{memcopy, memmove}() on x86, or qatomic_set() on other architecutures, to avoid the unexpected optimization done by the compiler on memcpy/memmove. Unsafe instructions can be produced by the compiler's optimization. This fixes the issue resolved by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions")

- The unaligned access is handled by forcing the access size aligned to (src | dest | n) on non-x86 architectures, comparing to the original memcpy() and memmove().

Please let me know if we're on same page. If we do, I can revise the comments accordingly.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

-- PMM

Thanks,
Gavin

author
#9 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 120 lines · Gavin Shan
Quoted 99 lines · Michael S. Tsirkin
Quoted 98 lines · Gavin Shan

Hi Peter and Michael,

Quoted 42 lines · Peter Maydell
Quoted 41 lines · Michael S. Tsirkin
Quoted 38 lines · Peter Maydell
Quoted 12 lines · Gavin Shan

All ram device regions were turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.

include/system/memory.h +0 -0
index 1417132f6d..5878727d09 100644

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2901

Quoted 5 lines
include/system/memory.h +2 -0
@@ -2897,6 +2897,8 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);
    void address_space_unregister_map_client(AddressSpace *as, QEMUBH *bh);
 
    /* Internal functions, part of the implementation of address_space_read.  */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+void qemu_ram_move(void *dest, const void *src, size_t n);

New function prototypes in include headers need documentation comments.

In particular for these, it's really important that we clearly say what semantics we are attempting to provide with them, so that
(a) when we're reviewing or later updating the implementation we know what we are trying to provide
(b) when we're looking at the callsites we know what the function is guaranteeing to us and what it is not, and thus whether it's OK to use it or we need something els.

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);

Quoted 5 lines
include/system/memory.h +5 -0
+#if defined(__i386__) || defined(__x86_64__)
+#define HOST_UNALIGNED_MMIO_OK 1
+#else
+#define HOST_UNALIGNED_MMIO_OK 0
+#endif

We need to do something better than this. We can't just say "oh, we trust that on x86 this works": it is neither actually true that the compiler guarantees it

Quoted 7 lines
Quoted 1 line

sorry guarantee what?

Well, that's part of my point about "we need a doc comment":
what exactly are we trying to guarantee ? But whatever it is, "hardcoded ifdef that privileges x86" is definitely the wrong answer.

Could you please check the following comments (documentation context) for the added functions look good to you? Please let me know if there are anything can be improved.

(quoted diff) +41 -0
+
+/**
+ * qemu_ram_copy: copy data to a RAMBlock
+ *
+ * @dest: destination where the data is copied to. It's the pointer returned
+ *        by ramblock_ptr() and its variants.
+ * @src: source where the data is copied from. It can be a regular memory block
+ *       or a pointer returned by ramblock_ptr() and its variants.
+ * @n: length of data to be copied
+ *
+ * The destination is always a pointer to data area of a RAMBlock which can
+ * be for a regular memory block or a MMIO region. The source can be a pointer
+ * to a regular memory block or a MMIO region. Atomicity isn't guranteed by
+ * memcpy() to copy data between those MMIO regions as we do in this function.
+ * Besides, unaligned accesses are well handled on all architectures except
+ * i386 and x86_64 where the unaligned accesses are supported by the CPUs.
+ *
+ * The ensured atomicity and alignment consideration, which aren't needed
+ * by data copy between two regular memory blocks. So performance penalty
+ * is introduced by this function in such circumstance.
+ */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+
+/**
+ * qemu_ram_move: move data to a RAMBlock
+ *
+ * @dest: destination where the data is moved to. It's the pointer returned
+ *        by ramblock_ptr() and its variants.
+ * @src: source where the data is moved from. It can be a regular memory block
+ *       or a pointer returned by ramblock_ptr() and its variants.
+ * @n: length of data to be moved
+ *
+ * The destination is always a pointer to data area of a RAMBlock which can
+ * be for a regular memory block or a MMIO region. The source can be a pointer
+ * to a regular memory block or a MMIO region. Atomicity isn't guranteed by
+ * memmove() to move data from or to those MMIO regions as we do in this
+ * function. Besides, unaligned accesses are well handled on all architectures
+ * except i386 and x86_64 where the unaligned accesses are supported by the
+ * CPUs.
+ *
+ * The ensured atomicity and alignment consideration, which aren't needed
Quoted 9 lines

Review

Quoted 5 lines

+ * by data movement between two regular memory blocks. So performance penalty
+ * is introduced by this function in such circumstance.
+ */

(quoted diff) +2 -0
+void qemu_ram_move(void *dest, const void *src, size_t n);
+

I apologize, I don't understand what these comments are saying, at all.

Sorry to hear that, Michael. There are two things done in the newly added function qemu_ram_{copy, move}(), comparing to the original memcpy() and memmove():

- Data copy or movement on MMIO region(s) using __bultin_{memcopy, memmove}() on x86, or qatomic_set() on other architecutures, to avoid the unexpected optimization done by the compiler on memcpy/memmove. Unsafe instructions can be produced by the compiler's optimization. This fixes the issue resolved by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions")

Quoted 6 lines

- The unaligned access is handled by forcing the access size aligned to (src | dest | n) on non-x86 architectures, comparing to the original memcpy() and memmove().

Please let me know if we're on same page. If we do, I can revise the comments accordingly.

I've modified the comments to something as below. Please let me know if it's better. The point is to emphasize qemu_ram_{copy, memmove}() are friendly to IO regions :-)

-----> include/system/memory.h

+/**
+ * qemu_ram_copy: copy data to a guest memory block
+ *
+ * @dest: destination where the data is copied to. A pointer to a guest memory
+ * block returned from ramblock_ptr() or its variants
+ * @src: source where the data is copied from. A pointer to host memory block
+ * or guest memory block returned from ramblock_ptr() or its variants.
+ * @n: length of data to be copied
+ *
+ * When the source pointer, destinatoin pointer or both dereference guest
+ * memory blocks, which can be IO blocks (regions). Under this cirtumstance,
+ * data copy between those blocks is equivalent to IO accesses. It's unsafe
+ * to use memcpy(), which could be translated to IO unfriendly instructions
+ * by the compiler due to code level optimization, resulting in the unexpected
+ * behavior. This function is ensured to be friendly to both IO and memory
+ * accesses. No optimized instructions should be generated by the compiler,
+ * and no unexpected behavior should be observed for IO accesses.
+ */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+
+/**
+ * qemu_ram_move: move data to a guest memory block
+ *
+ * @dest: destination where the data is moved to. A pointer to a guest memory
+ * block returned from ramblock_ptr() or its variants
+ * @src: source where the data is moved from. A pointer to host memory block
+ * or guest memory block returned from ramblock_ptr() or its variants.
+ * @n: length of data to be copied
+ *
+ * Similar to qemu_ram_copy(), this function is ensured to be friendly to both
+ * IO and memory accesses. No optimized instructions should be generated by
+ * the compiler, and no unexpected behavior should be observed for IO accesses.
+ */
+void qemu_ram_move(void *dest, const void *src, size_t n);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

-- PMM

Thanks,
Gavin

#10 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 132 lines · Gavin Shan
Quoted 120 lines · Gavin Shan
Quoted 99 lines · Michael S. Tsirkin
Quoted 98 lines · Gavin Shan

Hi Peter and Michael,

Quoted 42 lines · Peter Maydell
Quoted 41 lines · Michael S. Tsirkin
Quoted 38 lines · Peter Maydell
Quoted 12 lines · Gavin Shan

All ram device regions were turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.

include/system/memory.h +0 -0
index 1417132f6d..5878727d09 100644

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2901

Quoted 5 lines
include/system/memory.h +2 -0
@@ -2897,6 +2897,8 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);
    void address_space_unregister_map_client(AddressSpace *as, QEMUBH *bh);
 
    /* Internal functions, part of the implementation of address_space_read.  */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+void qemu_ram_move(void *dest, const void *src, size_t n);

New function prototypes in include headers need documentation comments.

In particular for these, it's really important that we clearly say what semantics we are attempting to provide with them, so that
(a) when we're reviewing or later updating the implementation we know what we are trying to provide
(b) when we're looking at the callsites we know what the function is guaranteeing to us and what it is not, and thus whether it's OK to use it or we need something els.

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);

Quoted 5 lines
include/system/memory.h +5 -0
+#if defined(__i386__) || defined(__x86_64__)
+#define HOST_UNALIGNED_MMIO_OK 1
+#else
+#define HOST_UNALIGNED_MMIO_OK 0
+#endif

We need to do something better than this. We can't just say "oh, we trust that on x86 this works": it is neither actually true that the compiler guarantees it

Quoted 7 lines
Quoted 1 line

sorry guarantee what?

Well, that's part of my point about "we need a doc comment":
what exactly are we trying to guarantee ? But whatever it is, "hardcoded ifdef that privileges x86" is definitely the wrong answer.

Could you please check the following comments (documentation context) for the added functions look good to you? Please let me know if there are anything can be improved.

(quoted diff) +41 -0
+
+/**
+ * qemu_ram_copy: copy data to a RAMBlock
+ *
+ * @dest: destination where the data is copied to. It's the pointer returned
+ *        by ramblock_ptr() and its variants.
+ * @src: source where the data is copied from. It can be a regular memory block
+ *       or a pointer returned by ramblock_ptr() and its variants.
+ * @n: length of data to be copied
+ *
+ * The destination is always a pointer to data area of a RAMBlock which can
+ * be for a regular memory block or a MMIO region. The source can be a pointer
+ * to a regular memory block or a MMIO region. Atomicity isn't guranteed by
+ * memcpy() to copy data between those MMIO regions as we do in this function.
+ * Besides, unaligned accesses are well handled on all architectures except
+ * i386 and x86_64 where the unaligned accesses are supported by the CPUs.
+ *
+ * The ensured atomicity and alignment consideration, which aren't needed
+ * by data copy between two regular memory blocks. So performance penalty
+ * is introduced by this function in such circumstance.
+ */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+
+/**
+ * qemu_ram_move: move data to a RAMBlock
+ *
+ * @dest: destination where the data is moved to. It's the pointer returned
+ *        by ramblock_ptr() and its variants.
+ * @src: source where the data is moved from. It can be a regular memory block
+ *       or a pointer returned by ramblock_ptr() and its variants.
+ * @n: length of data to be moved
+ *
+ * The destination is always a pointer to data area of a RAMBlock which can
+ * be for a regular memory block or a MMIO region. The source can be a pointer
+ * to a regular memory block or a MMIO region. Atomicity isn't guranteed by
+ * memmove() to move data from or to those MMIO regions as we do in this
+ * function. Besides, unaligned accesses are well handled on all architectures
+ * except i386 and x86_64 where the unaligned accesses are supported by the
+ * CPUs.
+ *
+ * The ensured atomicity and alignment consideration, which aren't needed
Quoted 9 lines

Review

Quoted 5 lines

+ * by data movement between two regular memory blocks. So performance penalty
+ * is introduced by this function in such circumstance.
+ */

(quoted diff) +2 -0
+void qemu_ram_move(void *dest, const void *src, size_t n);
+

I apologize, I don't understand what these comments are saying, at all.

Sorry to hear that, Michael. There are two things done in the newly added function qemu_ram_{copy, move}(), comparing to the original memcpy() and memmove():

- Data copy or movement on MMIO region(s) using __bultin_{memcopy, memmove}() on x86, or qatomic_set() on other architecutures, to avoid the unexpected optimization done by the compiler on memcpy/memmove. Unsafe instructions can be produced by the compiler's optimization. This fixes the issue resolved by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions")

Quoted 6 lines

- The unaligned access is handled by forcing the access size aligned to (src | dest | n) on non-x86 architectures, comparing to the original memcpy() and memmove().

Please let me know if we're on same page. If we do, I can revise the comments accordingly.

I've modified the comments to something as below. Please let me know if it's better. The point is to emphasize qemu_ram_{copy, memmove}() are friendly to IO regions :-)

Quoted 5 lines

-----> include/system/memory.h

+/**
+ * qemu_ram_copy: copy data to a guest memory block
+ *

what is a memory block? ramblock?

don't we need something like this for reads?

Quoted 1 line

+ * @dest: destination where the data is copied to. A pointer to a guest memory

@dst?

Quoted 7 lines

+ * block returned from ramblock_ptr() or its variants
+ * @src: source where the data is copied from. A pointer to host memory block
+ * or guest memory block returned from ramblock_ptr() or its variants.
+ * @n: length of data to be copied
+ *
+ * When the source pointer, destinatoin pointer or both dereference guest
+ * memory blocks, which can be IO blocks (regions).

IO block?

let's not invent terminology pls.

Quoted 7 lines

Under this cirtumstance,
+ * data copy between those blocks is equivalent to IO accesses. It's unsafe
+ * to use memcpy(), which could be translated to IO unfriendly instructions
+ * by the compiler due to code level optimization, resulting in the unexpected
+ * behavior. This function is ensured to be friendly to both IO and memory
+ * accesses. No optimized instructions should be generated by the compiler,
+ * and no unexpected behavior should be observed for IO accesses.

Let's be specific, and brief, and tell user what this does:
Maybe:

Copy @n bytes from @src to @dst.
Assumes @src and @dst do not overlap.
Handles special cases such as uncacheable ramblocks correctly. Use this for accessing ramblock in response to DMA/VCPU IO and in preference to memcpy.

Quoted 8 lines

+ */

(quoted diff) +4 -0
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+
+/**
+ * qemu_ram_move: move data to a guest memory block
 
 
 

Review

Quoted 10 lines
(quoted diff) +10 -0
+ *
+ * @dest: destination where the data is moved to. A pointer to a guest memory
+ *        block returned from ramblock_ptr() or its variants
+ * @src: source where the data is moved from. A pointer to host memory block
+ *       or guest memory block returned from ramblock_ptr() or its variants.
+ * @n: length of data to be copied
+ *
+ * Similar to qemu_ram_copy(), this function is ensured to be friendly to both
+ * IO and memory accesses. No optimized instructions should be generated by
+ * the compiler, and no unexpected behavior should be observed for IO accesses.

unclear how it is different from copy.

Quoted 13 lines

+ */
+void qemu_ram_move(void *dest, const void *src, size_t n);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

-- PMM

Thanks,
Gavin

#11 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

In reply to Gavin Shan
Quoted 114 lines · Gavin Shan
Quoted 99 lines · Michael S. Tsirkin
Quoted 98 lines · Gavin Shan

Hi Peter and Michael,

Quoted 42 lines · Peter Maydell
Quoted 41 lines · Michael S. Tsirkin
Quoted 38 lines · Peter Maydell
Quoted 12 lines · Gavin Shan

All ram device regions were turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.

include/system/memory.h +0 -0
index 1417132f6d..5878727d09 100644

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2901

Quoted 5 lines
include/system/memory.h +2 -0
@@ -2897,6 +2897,8 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);
   void address_space_unregister_map_client(AddressSpace *as, QEMUBH *bh);
 
   /* Internal functions, part of the implementation of address_space_read.  */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+void qemu_ram_move(void *dest, const void *src, size_t n);

New function prototypes in include headers need documentation comments.

In particular for these, it's really important that we clearly say what semantics we are attempting to provide with them, so that
(a) when we're reviewing or later updating the implementation we know what we are trying to provide
(b) when we're looking at the callsites we know what the function is guaranteeing to us and what it is not, and thus whether it's OK to use it or we need something els.

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);

Quoted 5 lines
include/system/memory.h +5 -0
+#if defined(__i386__) || defined(__x86_64__)
+#define HOST_UNALIGNED_MMIO_OK 1
+#else
+#define HOST_UNALIGNED_MMIO_OK 0
+#endif

We need to do something better than this. We can't just say "oh, we trust that on x86 this works": it is neither actually true that the compiler guarantees it

Quoted 7 lines
Quoted 1 line

sorry guarantee what?

Well, that's part of my point about "we need a doc comment":
what exactly are we trying to guarantee ? But whatever it is, "hardcoded ifdef that privileges x86" is definitely the wrong answer.

Could you please check the following comments (documentation context) for the added functions look good to you? Please let me know if there are anything can be improved.

(quoted diff) +41 -0
+
+/**
+ * qemu_ram_copy: copy data to a RAMBlock
+ *
+ * @dest: destination where the data is copied to. It's the pointer returned
+ *        by ramblock_ptr() and its variants.
+ * @src: source where the data is copied from. It can be a regular memory block
+ *       or a pointer returned by ramblock_ptr() and its variants.
+ * @n: length of data to be copied
+ *
+ * The destination is always a pointer to data area of a RAMBlock which can
+ * be for a regular memory block or a MMIO region. The source can be a pointer
+ * to a regular memory block or a MMIO region. Atomicity isn't guranteed by
+ * memcpy() to copy data between those MMIO regions as we do in this function.
+ * Besides, unaligned accesses are well handled on all architectures except
+ * i386 and x86_64 where the unaligned accesses are supported by the CPUs.
+ *
+ * The ensured atomicity and alignment consideration, which aren't needed
+ * by data copy between two regular memory blocks. So performance penalty
+ * is introduced by this function in such circumstance.
+ */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+
+/**
+ * qemu_ram_move: move data to a RAMBlock
+ *
+ * @dest: destination where the data is moved to. It's the pointer returned
+ *        by ramblock_ptr() and its variants.
+ * @src: source where the data is moved from. It can be a regular memory block
+ *       or a pointer returned by ramblock_ptr() and its variants.
+ * @n: length of data to be moved
+ *
+ * The destination is always a pointer to data area of a RAMBlock which can
+ * be for a regular memory block or a MMIO region. The source can be a pointer
+ * to a regular memory block or a MMIO region. Atomicity isn't guranteed by
+ * memmove() to move data from or to those MMIO regions as we do in this
+ * function. Besides, unaligned accesses are well handled on all architectures
+ * except i386 and x86_64 where the unaligned accesses are supported by the
+ * CPUs.
+ *
+ * The ensured atomicity and alignment consideration, which aren't needed
Quoted 9 lines

Review

Quoted 5 lines

+ * by data movement between two regular memory blocks. So performance penalty
+ * is introduced by this function in such circumstance.
+ */

(quoted diff) +2 -0
+void qemu_ram_move(void *dest, const void *src, size_t n);
+

I apologize, I don't understand what these comments are saying, at all.

Sorry to hear that, Michael. There are two things done in the newly added function qemu_ram_{copy, move}(), comparing to the original memcpy() and memmove():

Quoted 8 lines

- Data copy or movement on MMIO region(s) using __bultin_{memcopy, memmove}() on x86, or qatomic_set() on other architecutures, to avoid the unexpected optimization done by the compiler on memcpy/memmove. Unsafe instructions can be produced by the compiler's optimization. This fixes the issue resolved by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions")

- The unaligned access is handled by forcing the access size aligned to (src | dest | n) on non-x86 architectures, comparing to the original memcpy() and memmove().

Well first of all let's start with listing what the problems are. Below is my understanding of the issues:

1. On x86, memcpy is different from __builtin_memcpy if one uses old 1.0 force-headers from 2019. Likely no longer relevant.

2. variable length memcpy can translate 2,4,8 byte guest access into multiple byte accesses. doing this for mmio is guaranteed to break devices.

3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can
in theory break devices.

4. also on x86, vector instructions for large (>16 byte) writes into pgprot_noncached memory are safe and faster than multiple 8 byte ones.

5. also on x86 it so happens that if you write a fixed-size memcpy this gets optimized to a single store/load and it works for aligned and unaligned addresses on that architecture. How to ensure this keeps being correct is left as an excerise for the reader. But qemu already relies on this and did for years.

6. on non-x86 both unaligned accesses and vector instructions for accessing UC memory are illegal.

7. standard vfio gives KVM VM_ALLOW_ANY_UNCACHED, so even on non x86 guest can
map the memory as as pgprot_noncached/ioremap or pgprot_writecombine/ioremap_uc. If it does the second then it can use unaligned or vector for access. This is why normal passthrough tends to work - it never traps to qemu at all.

But for qemu, vfio uses pgprot_noncached unconditionally so qemu can't use unaligned or vector instructions on non-x86.

8. But for nvgrace RAM, vfio has a driver that uses pgprot_writecombine/ioremap_uc. so qemu could safely use unaligned/vector instructioons even on non-x86.

9. Except sadly, vfio currently does not tell qemu how it maps the memory, so qemu can not know what is safe on non-x86.

Now, what is to be done?

A. on x86, we must avoid converting 2,4,8 byte accesses into byte accesses. At least for aligned, perferably for unaligned accesses too. Fixed width memcpy seems to work for this. Whether we should bother with __builtin to work around broken old fortify headers, I donnu.
I do not have any answer how to check that compiler does this correctly. If anyone is motivated enough, adding a GCC builtin could be possible. Given qemu did this for years, I think we can leave solving this for another day.

B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf.

C. on non-x86, we currently must not memcpy since we do not know if it is pgprot_noncached. yes, performance will be bad for DMA into device RAM.

D. It goes without saying that casting an unaligned address to unint32_t (be it for qatomic_set or whatever) is undefined behaviour in C and so a bad idea on any architecture.

E. also for non-x86, we really should teach vfio to tell qemu whether it maps device pgprot_noncached or pgprot_writecombine.
we will then be able to use memcpy for >8 accesses.

Anyone, correct me if I'm wrong? Maybe I should start a new thread with this summary?

Quoted 11 lines

Please let me know if we're on same page. If we do, I can revise the comments accordingly.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

-- PMM

Thanks,
Gavin

#12 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 2 lines · Michael S. Tsirkin

B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf.

We have an example where memcpy writes to the same location 3 times. This is not appropriate for any host.

r~

#13 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 9 lines · Richard Henderson
Quoted 2 lines · Michael S. Tsirkin

B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf.

We have an example where memcpy writes to the same location 3 times. This is not appropriate for any host.

r~

as in, same byte? could you share the details pls?

author
#14 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Richard Henderson
Quoted 2 lines · Michael S. Tsirkin

B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf.

Quoted 8 lines
Quoted 5 lines

We have an example where memcpy writes to the same location 3 times. This is not appropriate for any host.

r~

as in, same byte? could you share the details pls?

This issue happened on aarch64 only.

https://lore.kernel.org/qemu-devel/CAFEAcA8dwHV8F48kb-013rxkG9kKcZhym9_qarKmoeUfeh0YWw@mail.gmail.com/

Thanks,
Gavin

#15 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 15 lines · Gavin Shan
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Richard Henderson
Quoted 2 lines · Michael S. Tsirkin

B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf.

Quoted 8 lines
Quoted 5 lines

We have an example where memcpy writes to the same location 3 times. This is not appropriate for any host.

r~

as in, same byte? could you share the details pls?

Quoted 6 lines

This issue happened on aarch64 only.

https://lore.kernel.org/qemu-devel/CAFEAcA8dwHV8F48kb-013rxkG9kKcZhym9_qarKmoeUfeh0YWw@mail.gmail.com/

Thanks,
Gavin

oh wow. thanks for sharing!

Still I think for sizes outside of 2,4,8 it's fine? That is where the perf is.

Signature · 1 line
MST
author
#16 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Gavin Shan
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Richard Henderson
Quoted 2 lines · Michael S. Tsirkin

B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf.

Quoted 8 lines
Quoted 5 lines

We have an example where memcpy writes to the same location 3 times. This is not appropriate for any host.

r~

as in, same byte? could you share the details pls?

This issue happened on aarch64 only.

https://lore.kernel.org/qemu-devel/CAFEAcA8dwHV8F48kb-013rxkG9kKcZhym9_qarKmoeUfeh0YWw@mail.gmail.com/

Quoted 8 lines
Quoted 2 lines

Thanks,
Gavin

oh wow. thanks for sharing!

Still I think for sizes outside of 2,4,8 it's fine? That is where the perf is.

For that point, I agree with you that it's fine to use memcpy/memmove when the length is 16-bytes or more, unless Richard is concerned by this.

I'm posting (v3) and lets continue the discussion on (v3) series. Sorry for taking your guys so many time to review and discuss ;-)

Thanks,
Gavin

Quoted 1 line
#17 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 26 lines · Gavin Shan
Quoted 25 lines · Michael S. Tsirkin
Quoted 15 lines · Gavin Shan
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Richard Henderson
Quoted 2 lines · Michael S. Tsirkin

B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf.

Quoted 8 lines
Quoted 5 lines

We have an example where memcpy writes to the same location 3 times. This is not appropriate for any host.

r~

as in, same byte? could you share the details pls?

Quoted 6 lines

This issue happened on aarch64 only.

https://lore.kernel.org/qemu-devel/CAFEAcA8dwHV8F48kb-013rxkG9kKcZhym9_qarKmoeUfeh0YWw@mail.gmail.com/

Thanks,
Gavin

oh wow. thanks for sharing!

Quoted 8 lines
Quoted 3 lines

Still I think for sizes outside of 2,4,8 it's fine? That is where the perf is.

For that point, I agree with you that it's fine to use memcpy/memmove when the length is 16-bytes or more,
unless Richard is concerned by
this.

yea size cutoff is a heuristic but i guess it's the best we can do. or disable memcpy completely if Richard feels strongly about it. But it seems like a waste.

Quoted 6 lines

I'm posting (v3) and lets continue the discussion on (v3) series. Sorry for taking your guys so many time to review and discuss ;-)

Thanks,
Gavin

pls do include the list of issues and high level solution in the cover letter though.

#18 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

In reply to Richard Henderson
Quoted 9 lines · Richard Henderson
Quoted 2 lines · Michael S. Tsirkin

B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf.

We have an example where memcpy writes to the same location 3 times. This is not appropriate for any host.

r~

Ah, checked libc and sure enough, it does it. E.g. it uses 2 overlapping SSE stores to do a 17 byte write. Not sure how we get 3 but whatevs.

But just to clarify, I am talking about DMA accesses, that are not initiated by the VCPU. I am not so sure we care about multiple stores in this instance? Do we? We do care about speed, for sure.

Signature · 1 line
MST
author
#19 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 11 lines · Michael S. Tsirkin
Quoted 9 lines · Richard Henderson
Quoted 2 lines · Michael S. Tsirkin

B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf.

We have an example where memcpy writes to the same location 3 times. This is not appropriate for any host.

r~

Quoted 8 lines

Ah, checked libc and sure enough, it does it. E.g. it uses 2 overlapping SSE stores to do a 17 byte write. Not sure how we get 3 but whatevs.

But just to clarify, I am talking about DMA accesses, that are not initiated by the VCPU. I am not so sure we care about multiple stores in this instance? Do we? We do care about speed, for sure.

In current implementation, qemu_ram_copy/move are differentiated on x86 and other architectures. Do we need to unify the implementations (qemu_ram_copy/move) on all architectures to avoid using memcpy() and memmove()?

Maybe it's time for me to post (v3) for a new round of discussions.

Thanks,
Gavin

#20 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 16 lines · Gavin Shan
Quoted 15 lines · Michael S. Tsirkin
Quoted 9 lines · Richard Henderson
Quoted 2 lines · Michael S. Tsirkin

B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf.

We have an example where memcpy writes to the same location 3 times. This is not appropriate for any host.

r~

Ah, checked libc and sure enough, it does it. E.g. it uses 2 overlapping SSE stores to do a 17 byte write. Not sure how we get 3 but whatevs.

Quoted 8 lines
Quoted 4 lines

But just to clarify, I am talking about DMA accesses, that are not initiated by the VCPU. I am not so sure we care about multiple stores in this instance? Do we? We do care about speed, for sure.

In current implementation, qemu_ram_copy/move are differentiated on x86 and other architectures. Do we need to unify the implementations (qemu_ram_copy/move) on all architectures to avoid using memcpy() and memmove()?

I am not sure for anything outside 1,2,4,8 bytes the issues are not theoretical.

I'd be care

Quoted 4 lines

Maybe it's time for me to post (v3) for a new round of discussions.

Thanks,
Gavin

maybe start a toppost with the list of issues and solutions first of all.

Let's add more to the list:

10. on x86 memcpy will sometimes do multiple overlapping stores when size is not a power of 2. for example, a 15 byte write is done with 2 8-byte stores. This is theoretically an issue
if guest does something super clever with ordering,
but does not seem to be in practice.

10. on non-x86 memcpy will do multiple overlapping stores even for single byte writes. E.g. it does it to avoid extra branches. This is causing issues in practice.

Signature · 1 line
MST
#21 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

In reply to Peter Maydell
Quoted 42 lines · Peter Maydell
Quoted 41 lines · Michael S. Tsirkin
Quoted 38 lines · Peter Maydell
Quoted 12 lines · Gavin Shan

All ram device regions were turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to guest hang on attempt to build 'cuda-samples' as reported by Julia. The guest is started by the following command lines, with GH100 GPU card passed from the host.

include/system/memory.h +0 -0
index 1417132f6d..5878727d09 100644

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh); · line 2901

Quoted 5 lines
include/system/memory.h +2 -0
@@ -2897,6 +2897,8 @@ void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);
 void address_space_unregister_map_client(AddressSpace *as, QEMUBH *bh);
 
 /* Internal functions, part of the implementation of address_space_read.  */
+void qemu_ram_copy(void *dest, const void *src, size_t n);
+void qemu_ram_move(void *dest, const void *src, size_t n);

New function prototypes in include headers need documentation comments.

In particular for these, it's really important that we clearly say what semantics we are attempting to provide with them, so that
(a) when we're reviewing or later updating the implementation we know what we are trying to provide
(b) when we're looking at the callsites we know what the function is guaranteeing to us and what it is not, and thus whether it's OK to use it or we need something els.

include/system/memory.h void address_space_register_map_client(AddressSpace *as, QEMUBH *bh);

Quoted 5 lines
include/system/memory.h +5 -0
+#if defined(__i386__) || defined(__x86_64__)
+#define HOST_UNALIGNED_MMIO_OK 1
+#else
+#define HOST_UNALIGNED_MMIO_OK 0
+#endif

We need to do something better than this. We can't just say "oh, we trust that on x86 this works": it is neither actually true that the compiler guarantees it

Quoted 4 lines
Quoted 1 line

sorry guarantee what?

Well, that's part of my point about "we need a doc comment":
what exactly are we trying to guarantee ?

we are trying to guarantee that host transaction sizes match what guest requested as closely as possible.

Quoted 5 lines

But whatever it is,
"hardcoded ifdef that privileges x86" is definitely the wrong answer.

-- PMM

why is it "definitely a wrong answer" if a specific host architecture supports originating transactions that other host architectures lack? should we intentionally break setups that could initiate unaligned accesses and work fine just because some other setups have hardware that simply can not initiate unaligned accesses?

Signature · 1 line
MST
#22 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

In reply to Gavin Shan
Quoted 16 lines · Gavin Shan
system/physmem.c +16 -0
+void qemu_ram_copy(void *dest, const void *src, size_t n)
+{
+    if (HOST_UNALIGNED_MMIO_OK) {
+        switch (n) {
+        case 1:
+            __builtin_memcpy(dest, src, 1);
+            break;
+        case 2:
+            __builtin_memcpy(dest, src, 2);
+            break;
+        case 4:
+            __builtin_memcpy(dest, src, 4);
+            break;
+        case 8:
+            __builtin_memcpy(dest, src, 8);
+            break;

system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3194

Quoted 10 lines
system/physmem.c +10 -0
+        default:
+            memcpy(dest, src, n);
+        }
+    } else {
+        uintptr_t test, lsb;
+
+        do {
+            test = (uintptr_t)dest | n;
+            lsb = test & -test;
+            switch (lsb) {

Either assert n != 0 to start, or use while not do/while. Because the body of the loop won't handle n == 0 correctly.

Quoted 13 lines
system/physmem.c +13 -0
+            case 1:
+                *(uint8_t *)dest = *(uint8_t *)src;
+                src += 1;
+                dest += 1;
+                n -= 1;
+                break;
+            case 2:
+                *(uint16_t *)dest = *(uint16_t *)src;
+                src += 2;
+                dest += 2;
+                n -= 2;
+                break;
+            case 4:

system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3217

Quoted 10 lines
system/physmem.c +10 -0
+                *(uint32_t *)dest = *(uint32_t *)src;
+                src += 4;
+                dest += 4;
+                n -= 4;
+                break;
+            default:
+                *(uint64_t *)dest = *(uint64_t *)src;
+                src += 8;
+                dest += 8;
+                n -= 8;

Use qatomic_set for the stores.

src is not aligned, so except for case 1, you need ld{uw,l,q}_he_p.

Quoted 13 lines
system/physmem.c +13 -0
+void qemu_ram_move(void *dest, const void *src, size_t n)
+{
+    if (HOST_UNALIGNED_MMIO_OK) {
+        switch (n) {
+        case 1:
+            __builtin_memmove(dest, src, 1);
+            break;
+        case 2:
+            __builtin_memmove(dest, src, 2);
+            break;
+        case 4:
+            __builtin_memmove(dest, src, 4);
+            break;

system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3245

Quoted 10 lines
system/physmem.c +10 -0
+        case 8:
+            __builtin_memmove(dest, src, 8);
+            break;
+        default:
+            memmove(dest, src, n);
+        }
+    } else {
+        qemu_ram_copy(dest, src, n);
+    }
+}

The qemu_ram_copy implementation above does not work with overlapping blocks.

r~

author
#23 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 22 lines · Richard Henderson
Quoted 21 lines · Gavin Shan
system/physmem.c +21 -0
+void qemu_ram_copy(void *dest, const void *src, size_t n)
+{
+    if (HOST_UNALIGNED_MMIO_OK) {
+        switch (n) {
+        case 1:
+            __builtin_memcpy(dest, src, 1);
+            break;
+        case 2:
+            __builtin_memcpy(dest, src, 2);
+            break;
+        case 4:
+            __builtin_memcpy(dest, src, 4);
+            break;
+        case 8:
+            __builtin_memcpy(dest, src, 8);
+            break;
+        default:
+            memcpy(dest, src, n);
+        }
+    } else {
+        uintptr_t test, lsb;
Quoted 9 lines
Quoted 5 lines

+
+        do {
+            test = (uintptr_t)dest | n;
+            lsb = test & -test;
+            switch (lsb) {

Either assert n != 0 to start, or use while not do/while. Because the body of the loop won't handle n == 0 correctly.

I will change this to "while (n > 0)" since we're not expecting "n < 0" either.

Quoted 24 lines
Quoted 15 lines

+            case 1:
+                *(uint8_t *)dest = *(uint8_t *)src; +                src += 1;
+                dest += 1;
+                n -= 1;
+                break;
+            case 2:
+                *(uint16_t *)dest = *(uint16_t *)src; +                src += 2;
+                dest += 2;
+                n -= 2;
+                break;
+            case 4:
+                *(uint32_t *)dest = *(uint32_t *)src; +                src += 4;

Quoted 8 lines

+                dest += 4;
+                n -= 4;
+                break;
+            default:
+                *(uint64_t *)dest = *(uint64_t *)src; +                src += 8;
+                dest += 8;
+                n -= 8;

Quoted 2 lines

Use qatomic_set for the stores.

Could you confirm which stores need qatomic_set()? There are two sets of stores as below. I guess you're asking have qatomic_set() for (a)? Could you explain a bit why qatomic_set() is needed?

    // (a)
    *(uint64_t *)dest = *(uint64_t *)src;


    // (b)
    src += 8;
    dest += 8;
    n -= 8;
Quoted 2 lines

src is not aligned, so except for case 1, you need ld{uw,l,q}_he_p.

Yeah, I realized this after this patch was sent. ldub_p() will be used for case 1 either.

Another unrelated question: why 'int' value is returned from ldub_p() and ld{uw, l}_he_p() in bswap.h? They would return 'uint{8, 16, 32}_t' values?

static inline int ldub_p(const void *ptr)
static inline int lduw_he_p(const void *ptr)
static inline int ldl_he_p(const void *ptr)
static inline uint64_t ldq_he_p(const void *ptr) // return uint64_t

Quoted 17 lines
Quoted 17 lines
system/physmem.c +17 -0
+void qemu_ram_move(void *dest, const void *src, size_t n)
+{
+    if (HOST_UNALIGNED_MMIO_OK) {
+        switch (n) {
+        case 1:
+            __builtin_memmove(dest, src, 1);
+            break;
+        case 2:
+            __builtin_memmove(dest, src, 2);
+            break;
+        case 4:
+            __builtin_memmove(dest, src, 4);
+            break;
+        case 8:
+            __builtin_memmove(dest, src, 8);
+            break;
+        default:
Quoted 9 lines
Quoted 6 lines

+            memmove(dest, src, n);
+        }
+    } else {
+        qemu_ram_copy(dest, src, n);
+    }
+}

The qemu_ram_copy implementation above does not work with overlapping blocks.

Yeah, it's something I need to figure out for next revision. Thanks for your review and comments.

Quoted 3 lines

r~

Thanks,
Gavin

#24 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 6 lines · Gavin Shan
Quoted 3 lines

Either assert n != 0 to start, or use while not do/while. Because the body of the loop won't handle n == 0 correctly.

I will change this to "while (n > 0)" since we're not expecting "n < 0" either.

size_t is unsigned.

Quoted 6 lines

Could you confirm which stores need qatomic_set()? There are two sets of stores as below. I guess you're asking have qatomic_set() for (a)? Could you explain a bit why qatomic_set() is needed?

    // (a)
    *(uint64_t *)dest = *(uint64_t *)src;

Of course (a).

Using qatomic_set prevents the compiler from merging the stores, kinda like volatile. Not especially likely with the way you've written this, but if you had exchanged the switch and loop,

    switch (lsb) {
    case 1:
        for (; n ; n--, dst++, src++) {
            *(uint8_t *)dst = *(uint8_t *)src;
        }

then the compiler is quite likely to "optimize" the loop back to memcpy.

It's also self-documenting what we're intending with the store.

I guess it's also worth asking if this copy is also used for copying *from* device memory.
The commit that added memmove (4a73aee8814) suggests that dma from the device uses these
paths. In which case you'll either want a separate function for that, or both source and destination must be aligned.

Quoted 3 lines

Another unrelated question: why 'int' value is returned from ldub_p() and ld{uw, l}_he_p() in bswap.h? They would return 'uint{8, 16, 32}_t' values?

A bad choice 20 years ago, and the need to audit all uses in order to change it now. Newer functions use uintN_t as you suggest.

r~

author
#25 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 10 lines · Richard Henderson
Quoted 6 lines · Gavin Shan
Quoted 3 lines

Either assert n != 0 to start, or use while not do/while. Because the body of the loop won't handle n == 0 correctly.

I will change this to "while (n > 0)" since we're not expecting "n < 0" either.

size_t is unsigned.

oh, yes.

Quoted 17 lines
Quoted 6 lines

Could you confirm which stores need qatomic_set()? There are two sets of stores as below. I guess you're asking have qatomic_set() for (a)? Could you explain a bit why qatomic_set() is needed?

     // (a)
     *(uint64_t *)dest = *(uint64_t *)src;

Of course (a).

Using qatomic_set prevents the compiler from merging the stores, kinda like volatile.  Not especially likely with the way you've written this, but if you had exchanged the switch and loop,

    switch (lsb) {
    case 1:
        for (; n ; n--, dst++, src++) {
            *(uint8_t *)dst = *(uint8_t *)src;
        }

Quoted 4 lines

then the compiler is quite likely to "optimize" the loop back to memcpy.

It's also self-documenting what we're intending with the store.

ok, thanks for your explanation.

Quoted 2 lines

I guess it's also worth asking if this copy is also used for copying *from* device memory.  The commit that added memmove (4a73aee8814) suggests that dma from the device uses these paths.  In which case you'll either want a separate function for that, or both source and destination must be aligned.

I think it's a valid case. Lets handle this by limiting the 'step' based on src/dest/n, something like below. atomic_read() instead of ldxxx_he_p() is used for loads. Please take a look if there are anything we can improve further.

// Not compiled and tested yet
static void qemu_ram_copy_unaligned(void *dest, const void *src, size_t n)

{
    uintptr_t test, step;

    while (n != 0) {
        test = (uintptr_t)src | n;       /* alignment enforced by source */
        step = test & -test;
        test = (uintptr_t)dest | n;      /* alignment enforced by destination */
        step = MIN(step, test & -test);

        switch (step) {
        case 1:
            qatomic_set((uint8_t *)dest, qatomic_read((uint8_t *)src));
            src += 1;
            dest += 1;
            n -= 1;
            break;
        case 2:
            qatomic_set((uint16_t *)dest, qatomic_read((uint16_t *)src));
            src += 2;
            dest += 2;
            n -= 2;
            break;
        case 4:
            qatomic_set((uint32_t *)dest, qatomic_read((uint32_t *)src));
            src += 4;
            dest += 4;
            n -= 4;
            break;
        default:
            qatomic_set((uint64_t *)dest, qatomic_read((uint64_t *)src);
            src += 8;
            dest += 8;
            n -= 8;
        }
    }
}
Quoted 7 lines
Quoted 3 lines

Another unrelated question: why 'int' value is returned from ldub_p() and ld{uw, l}_he_p() in bswap.h? They would return 'uint{8, 16, 32}_t' values?

A bad choice 20 years ago, and the need to audit all uses in order to change it now. Newer functions use uintN_t as you suggest.

Ok :-)

Quoted 2 lines

r~

Thanks,
Gavin

#26 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 13 lines · Gavin Shan
Quoted 2 lines

I guess it's also worth asking if this copy is also used for copying *from* device memory.  The commit that added memmove (4a73aee8814) suggests that dma from the device uses these paths.  In which case you'll either want a separate function for that, or both source and destination must be aligned.

I think it's a valid case. Lets handle this by limiting the 'step' based on src/dest/n, something like below. atomic_read() instead of ldxxx_he_p() is used for loads. Please take a look if there are anything we can improve further.

// Not compiled and tested yet
static void qemu_ram_copy_unaligned(void *dest, const void *src, size_t n) {
    uintptr_t test, step;

Quoted 5 lines

    while (n != 0) {
        test = (uintptr_t)src | n;       /* alignment enforced by source */ step = test & -test;
        test = (uintptr_t)dest | n;      /* alignment enforced by destination */ step = MIN(step, test & -test);

    while (n) {
        uintptr_t test = (uintptr_t)src | (uintptr_t)dst | n;
        uintptr_t step = test & -test;

Otherwise it looks good.

r~

#27 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

In reply to Gavin Shan
Quoted 74 lines · Gavin Shan
Quoted 10 lines · Richard Henderson
Quoted 6 lines · Gavin Shan
Quoted 3 lines

Either assert n != 0 to start, or use while not do/while. Because the body of the loop won't handle n == 0 correctly.

I will change this to "while (n > 0)" since we're not expecting "n < 0" either.

size_t is unsigned.

oh, yes.

Quoted 17 lines
Quoted 6 lines

Could you confirm which stores need qatomic_set()? There are two sets of stores as below. I guess you're asking have qatomic_set() for (a)? Could you explain a bit why qatomic_set() is needed?

     // (a)
     *(uint64_t *)dest = *(uint64_t *)src;

Of course (a).

Using qatomic_set prevents the compiler from merging the stores, kinda like volatile.  Not especially likely with the way you've written this, but if you had exchanged the switch and loop,

    switch (lsb) {
    case 1:
        for (; n ; n--, dst++, src++) {
            *(uint8_t *)dst = *(uint8_t *)src;
        }

Quoted 4 lines

then the compiler is quite likely to "optimize" the loop back to memcpy.

It's also self-documenting what we're intending with the store.

ok, thanks for your explanation.

Quoted 2 lines

I guess it's also worth asking if this copy is also used for copying *from* device memory.  The commit that added memmove (4a73aee8814) suggests that dma from the device uses these paths.  In which case you'll either want a separate function for that, or both source and destination must be aligned.

I think it's a valid case. Lets handle this by limiting the 'step' based on src/dest/n, something like below. atomic_read() instead of ldxxx_he_p() is used for loads. Please take a look if there are anything we can improve further.

// Not compiled and tested yet
static void qemu_ram_copy_unaligned(void *dest, const void *src, size_t n)

{
    uintptr_t test, step;

    while (n != 0) {
        test = (uintptr_t)src | n;       /* alignment enforced by source */
        step = test & -test;
        test = (uintptr_t)dest | n;      /* alignment enforced by destination */
        step = MIN(step, test & -test);

        switch (step) {
        case 1:
            qatomic_set((uint8_t *)dest, qatomic_read((uint8_t *)src));
            src += 1;
            dest += 1;
            n -= 1;
            break;
        case 2:
            qatomic_set((uint16_t *)dest, qatomic_read((uint16_t *)src));
            src += 2;
            dest += 2;
            n -= 2;
            break;
        case 4:
            qatomic_set((uint32_t *)dest, qatomic_read((uint32_t *)src));
            src += 4;
            dest += 4;
Quoted 8 lines
            n -= 4;
            break;
        default:
            qatomic_set((uint64_t *)dest, qatomic_read((uint64_t *)src);
            src += 8;
            dest += 8;
            n -= 8;
        }

If the length is > 8, I feel it's important to just use memcpy/memmove normally.

Because at that point it's a large data transfer not MMIO, and performance is important.

Quoted 19 lines
    }
}
Quoted 7 lines
Quoted 3 lines

Another unrelated question: why 'int' value is returned from ldub_p() and ld{uw, l}_he_p() in bswap.h? They would return 'uint{8, 16, 32}_t' values?

A bad choice 20 years ago, and the need to audit all uses in order to change it now. Newer functions use uintN_t as you suggest.

Ok :-)

Quoted 2 lines

r~

Thanks,
Gavin

author
#28 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 85 lines · Michael S. Tsirkin
Quoted 74 lines · Gavin Shan
Quoted 10 lines · Richard Henderson
Quoted 6 lines · Gavin Shan
Quoted 3 lines

Either assert n != 0 to start, or use while not do/while. Because the body of the loop won't handle n == 0 correctly.

I will change this to "while (n > 0)" since we're not expecting "n < 0" either.

size_t is unsigned.

oh, yes.

Quoted 17 lines
Quoted 6 lines

Could you confirm which stores need qatomic_set()? There are two sets of stores as below. I guess you're asking have qatomic_set() for (a)? Could you explain a bit why qatomic_set() is needed?

     // (a)
     *(uint64_t *)dest = *(uint64_t *)src;

Of course (a).

Using qatomic_set prevents the compiler from merging the stores, kinda like volatile.  Not especially likely with the way you've written this, but if you had exchanged the switch and loop,

      switch (lsb) {
      case 1:
          for (; n ; n--, dst++, src++) {
              *(uint8_t *)dst = *(uint8_t *)src;
          }
Quoted 4 lines

then the compiler is quite likely to "optimize" the loop back to memcpy.

It's also self-documenting what we're intending with the store.

ok, thanks for your explanation.

Quoted 2 lines

I guess it's also worth asking if this copy is also used for copying *from* device memory.  The commit that added memmove (4a73aee8814) suggests that dma from the device uses these paths.  In which case you'll either want a separate function for that, or both source and destination must be aligned.

I think it's a valid case. Lets handle this by limiting the 'step' based on src/dest/n, something like below. atomic_read() instead of ldxxx_he_p() is used for loads. Please take a look if there are anything we can improve further.

// Not compiled and tested yet
static void qemu_ram_copy_unaligned(void *dest, const void *src, size_t n)

{
     uintptr_t test, step;

     while (n != 0) {
         test = (uintptr_t)src | n;       /* alignment enforced by source */
         step = test & -test;
         test = (uintptr_t)dest | n;      /* alignment enforced by destination */
         step = MIN(step, test & -test);

         switch (step) {
         case 1:
             qatomic_set((uint8_t *)dest, qatomic_read((uint8_t *)src));
             src += 1;
             dest += 1;
             n -= 1;
             break;
         case 2:
             qatomic_set((uint16_t *)dest, qatomic_read((uint16_t *)src));
             src += 2;
             dest += 2;
             n -= 2;
             break;
         case 4:
             qatomic_set((uint32_t *)dest, qatomic_read((uint32_t *)src));
             src += 4;
             dest += 4;
Quoted 8 lines
             n -= 4;
             break;
         default:
             qatomic_set((uint64_t *)dest, qatomic_read((uint64_t *)src);
             src += 8;
             dest += 8;
             n -= 8;
         }
Quoted 6 lines

If the length is > 8, I feel it's important to just use memcpy/memmove normally.

Because at that point it's a large data transfer not MMIO, and performance is important.

Yes, making sense to me. it shouldn't be a MMIO transaction if the length is 16 bytes or more, we can use memcpy/memmove() for this specific case, to improve the performance.

Quoted 13 lines
Quoted 13 lines
     }
}
Quoted 7 lines
Quoted 3 lines

Another unrelated question: why 'int' value is returned from ldub_p() and ld{uw, l}_he_p() in bswap.h? They would return 'uint{8, 16, 32}_t' values?

A bad choice 20 years ago, and the need to audit all uses in order to change it now. Newer functions use uintN_t as you suggest.

Quoted 5 lines
Quoted 5 lines

Ok :-)

Quoted 2 lines

r~

Thanks,
Gavin

#29 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

In reply to Richard Henderson
Quoted 55 lines · Richard Henderson
Quoted 16 lines · Gavin Shan
system/physmem.c +16 -0
+void qemu_ram_copy(void *dest, const void *src, size_t n)
+{
+    if (HOST_UNALIGNED_MMIO_OK) {
+        switch (n) {
+        case 1:
+            __builtin_memcpy(dest, src, 1);
+            break;
+        case 2:
+            __builtin_memcpy(dest, src, 2);
+            break;
+        case 4:
+            __builtin_memcpy(dest, src, 4);
+            break;
+        case 8:
+            __builtin_memcpy(dest, src, 8);
+            break;

system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3194

Quoted 10 lines
system/physmem.c +10 -0
+        default:
+            memcpy(dest, src, n);
+        }
+    } else {
+        uintptr_t test, lsb;
+
+        do {
+            test = (uintptr_t)dest | n;
+            lsb = test & -test;
+            switch (lsb) {

Either assert n != 0 to start, or use while not do/while. Because the body of the loop won't handle n == 0 correctly.

Quoted 13 lines
system/physmem.c +13 -0
+            case 1:
+                *(uint8_t *)dest = *(uint8_t *)src;
+                src += 1;
+                dest += 1;
+                n -= 1;
+                break;
+            case 2:
+                *(uint16_t *)dest = *(uint16_t *)src;
+                src += 2;
+                dest += 2;
+                n -= 2;
+                break;
+            case 4:

system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3217

Quoted 10 lines
system/physmem.c +10 -0
+                *(uint32_t *)dest = *(uint32_t *)src;
+                src += 4;
+                dest += 4;
+                n -= 4;
+                break;
+            default:
+                *(uint64_t *)dest = *(uint64_t *)src;
+                src += 8;
+                dest += 8;
+                n -= 8;
Quoted 1 line

Use qatomic_set for the stores.

Won't this do things like locked on x86?

Quoted 2 lines

src is not aligned, so except for case 1, you need ld{uw,l,q}_he_p.

These just map back to memcpy. What is the point?

Quoted 17 lines
Quoted 17 lines
system/physmem.c +17 -0
+void qemu_ram_move(void *dest, const void *src, size_t n)
+{
+    if (HOST_UNALIGNED_MMIO_OK) {
+        switch (n) {
+        case 1:
+            __builtin_memmove(dest, src, 1);
+            break;
+        case 2:
+            __builtin_memmove(dest, src, 2);
+            break;
+        case 4:
+            __builtin_memmove(dest, src, 4);
+            break;
+        case 8:
+            __builtin_memmove(dest, src, 8);
+            break;
+        default:
Quoted 8 lines

system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3245

Quoted 6 lines
system/physmem.c +6 -0
+            memmove(dest, src, n);
+        }
+    } else {
+        qemu_ram_copy(dest, src, n);
+    }
+}

The qemu_ram_copy implementation above does not work with overlapping blocks.

IIUC copy is not supposed to. this is what move is for.

Quoted 2 lines

r~

#30 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Richard Henderson
Quoted 21 lines · Gavin Shan
system/physmem.c +21 -0
+void qemu_ram_copy(void *dest, const void *src, size_t n)
+{
+    if (HOST_UNALIGNED_MMIO_OK) {
+        switch (n) {
+        case 1:
+            __builtin_memcpy(dest, src, 1);
+            break;
+        case 2:
+            __builtin_memcpy(dest, src, 2);
+            break;
+        case 4:
+            __builtin_memcpy(dest, src, 4);
+            break;
+        case 8:
+            __builtin_memcpy(dest, src, 8);
+            break;
+        default:
+            memcpy(dest, src, n);
+        }
+    } else {
+        uintptr_t test, lsb;
Quoted 8 lines
Quoted 8 lines

system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3194

Quoted 5 lines
system/physmem.c +5 -0
+
+        do {
+            test = (uintptr_t)dest | n;
+            lsb = test & -test;
+            switch (lsb) {

Either assert n != 0 to start, or use while not do/while. Because the body of the loop won't handle n == 0 correctly.

I think n!=0 performs better than a loop. Worth checking asm though.

Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
system/physmem.c +23 -0
+            case 1:
+                *(uint8_t *)dest = *(uint8_t *)src;
+                src += 1;
+                dest += 1;
+                n -= 1;
+                break;
+            case 2:
+                *(uint16_t *)dest = *(uint16_t *)src;
+                src += 2;
+                dest += 2;
+                n -= 2;
+                break;
+            case 4:
+                *(uint32_t *)dest = *(uint32_t *)src;
+                src += 4;
+                dest += 4;
+                n -= 4;
+                break;
+            default:
+                *(uint64_t *)dest = *(uint64_t *)src;
+                src += 8;
+                dest += 8;
+                n -= 8;
Quoted 4 lines
Quoted 1 line

Use qatomic_set for the stores.

Won't this do things like locked on x86?

Hmm I didn't check. It's relaxed - so no locked. Fine by me.

Quoted 37 lines
Quoted 4 lines

src is not aligned, so except for case 1, you need ld{uw,l,q}_he_p.

These just map back to memcpy. What is the point?

Quoted 17 lines
Quoted 17 lines
system/physmem.c +17 -0
+void qemu_ram_move(void *dest, const void *src, size_t n)
+{
+    if (HOST_UNALIGNED_MMIO_OK) {
+        switch (n) {
+        case 1:
+            __builtin_memmove(dest, src, 1);
+            break;
+        case 2:
+            __builtin_memmove(dest, src, 2);
+            break;
+        case 4:
+            __builtin_memmove(dest, src, 4);
+            break;
+        case 8:
+            __builtin_memmove(dest, src, 8);
+            break;
+        default:
Quoted 8 lines

system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3245

Quoted 6 lines
system/physmem.c +6 -0
+            memmove(dest, src, n);
+        }
+    } else {
+        qemu_ram_copy(dest, src, n);
+    }
+}

The qemu_ram_copy implementation above does not work with overlapping blocks.

IIUC copy is not supposed to. this is what move is for.

Quoted 2 lines

r~

#31 lore raw

Re: [PATCH v2 1/2] system/memory: Use qemu_ram_{copy, move}() in ram device region accessors

In reply to Michael S. Tsirkin
Quoted 26 lines · Michael S. Tsirkin
Quoted 17 lines
Quoted 17 lines
system/physmem.c +17 -0
+void qemu_ram_move(void *dest, const void *src, size_t n)
+{
+    if (HOST_UNALIGNED_MMIO_OK) {
+        switch (n) {
+        case 1:
+            __builtin_memmove(dest, src, 1);
+            break;
+        case 2:
+            __builtin_memmove(dest, src, 2);
+            break;
+        case 4:
+            __builtin_memmove(dest, src, 4);
+            break;
+        case 8:
+            __builtin_memmove(dest, src, 8);
+            break;
+        default:
Quoted 8 lines

system/physmem.c void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size) · line 3245

Quoted 6 lines
system/physmem.c +6 -0
+            memmove(dest, src, n);
+        }
+    } else {
+        qemu_ram_copy(dest, src, n);
+    }
+}

The qemu_ram_copy implementation above does not work with overlapping blocks.

Quoted 1 line

IIUC copy is not supposed to. this is what move is for.

My point exactly. You therefore can't use copy to implement move.

r~

author
#32 lore raw

PATCH v2 2/2 system/memory: Make ram device region directly accessible

In reply to Gavin Shan

This basically reverts 4a2e242bbb30 ("memory: Don't use memcpy for ram_device regions") to make ram device region directly accessible again. With this, the bounce buffer is bypassed in address_space_map() when a ram device region is involved, potentially avoid to overrun the (small) bounce buffer.

---
 include/system/memory.h | 11 ++---------
 system/memory.c         | 41 +----------------------------------------
 system/trace-events     |  2 --
 3 files changed, 3 insertions(+), 51 deletions(-)
include/system/memory.h +2 -9
index 5878727d09..7d6ef9b32e 100644
@@ -2916,15 +2916,8 @@ static inline bool memory_region_supports_direct_access(const MemoryRegion *mr)
     if (memory_region_is_romd(mr)) {
         return true;
     }
-    if (!memory_region_is_ram(mr)) {
-        return false;
-    }
-    /*
-     * RAM DEVICE regions can be accessed directly using memcpy, but it might
-     * be MMIO and access using mempy can be wrong (e.g., using instructions not
-     * intended for MMIO access). So we treat this as IO.
-     */
-    return !memory_region_is_ram_device(mr);
+
+    return memory_region_is_ram(mr);
 }
 
 static inline bool memory_access_is_direct(const MemoryRegion *mr,
system/memory.c +1 -40

54 lines not on this page · raw message

system/trace-events +0 -2
index e6e1b61279..34af0a3a1e 100644
@@ -20,8 +20,6 @@ memory_region_ops_read(int cpu_index, void *mr, uint64_t addr, uint64_t value, u
 memory_region_ops_write(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size, const char *name) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u name '%s'"
 memory_region_subpage_read(int cpu_index, void *mr, uint64_t offset, uint64_t value, unsigned size) "cpu %d mr %p offset 0x%"PRIx64" value 0x%"PRIx64" size %u"
 memory_region_subpage_write(int cpu_index, void *mr, uint64_t offset, uint64_t value, unsigned size) "cpu %d mr %p offset 0x%"PRIx64" value 0x%"PRIx64" size %u"
-memory_region_ram_device_read(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u"
-memory_region_ram_device_write(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u"
 memory_region_sync_dirty(const char *mr, const char *listener, int global) "mr '%s' listener '%s' synced (global=%d)"
 flatview_new(void *view, void *root) "%p (root %p)"
 flatview_destroy(void *view, void *root) "%p (root %p)"
Signature · 1 line
2.54.0