Skip to contentSkip to conversation

plugins: add API to read guest CPU memory from hwaddr

PATCH Superseded Rowan Hart 3 participants 8 messages Aug 28, 2024 – Jan 9, 2025

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Rowan Hart

author
#0 lore raw

cover plugins: add API to read guest CPU memory from hwaddr

This patch adds a single API function which allows reading from a guest CPU physical address.

I don't know of a good way to add a self-contained test for this feature to tests/tcg/plugins, but I did come up with a small test case to demonstrate the functionality using peiyuanix/riscv-os:

First, grab and build the firmware code:

curl -o firmware.S https://raw.githubusercontent.com/peiyuanix/riscv-os/main/03-Bare-Metal-Hello-RISC-V/firmware.s
curl -o firmware.x https://raw.githubusercontent.com/peiyuanix/riscv-os/main/03-Bare-Metal-Hello-RISC-V/firmware.ld
riscv64-linux-gnu-as firmware.S -o firmware.o
riscv64-linux-gnu-ld -T firmware.x -o firmare firmware.o
riscv64-linux-gnu-objcopy -O binary -S firmware firmware.bin

Next, grab and build the plugin (just dumps from phys address on first instruction executed):

curl -o dump-riscv-firmware.c https://gist.githubusercontent.com/novafacing/5abc08052fab671a0fb26547810b4c55/raw/33772d614d6e36eae30e3405af34f149d7cc608b/dump-riscv-firmware.c
gcc -rdynamic -shared -fPIC -Iinclude/qemu $(pkg-config --cflags --libs glib-2.0) -o libdump-riscv-firmware.so dump-riscv-firmware.c

Finally, run the plugin:

qemu-system-riscv64 -display none -machine virt -serial stdio -bios firmware.bin -plugin $(pwd)libdump-riscv-firmware.so -d plugin

This outputs as expected -- the hexdump of the running firmware:

b7 01 00 10 a3 80 01 00 93 02 00 08 a3 81 51 00  | ..............Q.
93 02 50 00 23 80 51 00 93 02 00 00 a3 80 51 00  | ..P.#.Q.......Q.
93 02 30 00 a3 81 51 00 93 02 10 00 23 81 51 00  | ..0...Q.....#.Q.
23 82 01 00 83 82 51 00 83 82 01 00 a3 83 01 00  | #.....Q.........
93 02 80 04 23 80 51 00 93 02 50 06 23 80 51 00  | ....#.Q...P.#.Q.
93 02 c0 06 23 80 51 00 93 02 c0 06 23 80 51 00  | ....#.Q.....#.Q.
93 02 f0 06 23 80 51 00 93 02 c0 02 23 80 51 00  | ....#.Q.....#.Q.
93 02 00 02 23 80 51 00 93 02 20 05 23 80 51 00  | ....#.Q.....#.Q.
93 02 90 04 23 80 51 00 93 02 30 05 23 80 51 00  | ....#.Q...0.#.Q.
93 02 30 04 23 80 51 00 93 02 d0 02 23 80 51 00  | ..0.#.Q.....#.Q.
93 02 60 05 23 80 51 00 93 02 10 02 23 80 51 00  | ..`.#.Q.....#.Q.
93 02 a0 00 23 80 51 00 6f 00 00 00 00 00 00 00  | ....#.Q.o.......
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  | ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  | ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  | ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  | ................

Hello, RISC-V!

Rowan Hart (1):
plugins: add API to read guest CPU memory from hwaddr

 include/qemu/qemu-plugin.h   | 22 ++++++++++++++++++++++
 plugins/api.c                | 17 +++++++++++++++++
 plugins/qemu-plugins.symbols |  2 ++
 3 files changed, 41 insertions(+)
Signature · 1 line
2.46.0
author
#1 lore raw

PATCH 1/1 plugins: add API to read guest CPU memory from hwaddr

---
 include/qemu/qemu-plugin.h   | 22 ++++++++++++++++++++++
 plugins/api.c                | 17 +++++++++++++++++
 plugins/qemu-plugins.symbols |  2 ++
 3 files changed, 41 insertions(+)
include/qemu/qemu-plugin.h +22 -0
index c71c705b69..25f39c0960 100644
@@ -868,6 +868,28 @@ QEMU_PLUGIN_API
 int qemu_plugin_read_register(struct qemu_plugin_register *handle,
                               GByteArray *buf);
 
+/**
+ * qemu_plugin_read_cpu_memory_hwaddr() - read CPU memory from hwaddr
+ *
+ * @addr: A virtual address to read from
+ * @data: A byte array to store data into
+ * @len: The number of bytes to read, starting from @addr
+ *
+ * @len bytes of data is read starting at @addr and stored into @data. If @data
+ * is not large enough to hold @len bytes, it will be expanded to the necessary
+ * size, reallocating if necessary. @len must be greater than 0.
+ *
+ * This function does not ensure writes are flushed prior to reading, so
+ * callers should take care when calling this function in plugin callbacks to
+ * avoid attempting to read data which may not yet be written and should use
+ * the memory callback API instead.
+ *
+ * Returns true on success and false on failure.
+ */
+QEMU_PLUGIN_API
+bool qemu_plugin_read_cpu_memory_hwaddr(uint64_t addr,
+                                          GByteArray *data, size_t len);
+
 /**
  * qemu_plugin_scoreboard_new() - alloc a new scoreboard
  *
plugins/api.c +17 -0
index 2ff13d09de..c87bed6641 100644
@@ -527,6 +527,22 @@ GArray *qemu_plugin_get_registers(void)
     return create_register_handles(regs);
 }
 
+bool qemu_plugin_read_cpu_memory_hwaddr(uint64_t addr,
+                                        GByteArray *data, uint64_t len)
+{
+#ifndef CONFIG_USER_ONLY
+    if (len == 0) {
+        return false;
+    }
+
+    g_byte_array_set_size(data, len);
+    cpu_physical_memory_rw(addr, (void *)data->data, len, 0);
+    return true;
+#else
+    return false;
+#endif
+}
+
 int qemu_plugin_read_register(struct qemu_plugin_register *reg, GByteArray *buf)
 {
     g_assert(current_cpu);
@@ -534,6 +550,7 @@ int qemu_plugin_read_register(struct qemu_plugin_register *reg, GByteArray *buf)
     return gdb_read_register(current_cpu, buf, GPOINTER_TO_INT(reg) - 1);
 }
 
+
 struct qemu_plugin_scoreboard *qemu_plugin_scoreboard_new(size_t element_size)
 {
     return plugin_scoreboard_new(element_size);
plugins/qemu-plugins.symbols +2 -0
index ca773d8d9f..5d9cfd71bb 100644
@@ -20,6 +20,8 @@
   qemu_plugin_num_vcpus;
   qemu_plugin_outs;
   qemu_plugin_path_to_binary;
+  qemu_plugin_read_cpu_memory_hwaddr;
+  qemu_plugin_read_io_memory_hwaddr;
   qemu_plugin_read_register;
   qemu_plugin_register_atexit_cb;
   qemu_plugin_register_flush_cb;
Signature · 1 line
2.46.0
author
#2 lore raw

plugins/qemu-plugins.symbols line 24

Quoted 2 lines
plugins/qemu-plugins.symbols +2 -0
+  qemu_plugin_read_cpu_memory_hwaddr;
+  qemu_plugin_read_io_memory_hwaddr;

This second symbol name should be removed, I initially wanted to implement for IO as well but there is no good generic way I can see to access a list of IO AddressSpace to read from.

#3 lore raw
In reply to Rowan Hart

Hi Rowan,

thanks for this good complement on the virt address read function.

However, to be able to merge a new plugins API function, we must have a concrete usage of it, through one of the existing plugin. What could be a good demonstration of value brought by being able to read a physical address?

Thanks,
Pierrick

Quoted 80 lines · Rowan Hart
---
  include/qemu/qemu-plugin.h   | 22 ++++++++++++++++++++++
  plugins/api.c                | 17 +++++++++++++++++
  plugins/qemu-plugins.symbols |  2 ++
  3 files changed, 41 insertions(+)
include/qemu/qemu-plugin.h +22 -0
index c71c705b69..25f39c0960 100644
@@ -868,6 +868,28 @@ QEMU_PLUGIN_API
  int qemu_plugin_read_register(struct qemu_plugin_register *handle,
                                GByteArray *buf);
 
+/**
+ * qemu_plugin_read_cpu_memory_hwaddr() - read CPU memory from hwaddr
+ *
+ * @addr: A virtual address to read from
+ * @data: A byte array to store data into
+ * @len: The number of bytes to read, starting from @addr
+ *
+ * @len bytes of data is read starting at @addr and stored into @data. If @data
+ * is not large enough to hold @len bytes, it will be expanded to the necessary
+ * size, reallocating if necessary. @len must be greater than 0.
+ *
+ * This function does not ensure writes are flushed prior to reading, so
+ * callers should take care when calling this function in plugin callbacks to
+ * avoid attempting to read data which may not yet be written and should use
+ * the memory callback API instead.
+ *
+ * Returns true on success and false on failure.
+ */
+QEMU_PLUGIN_API
+bool qemu_plugin_read_cpu_memory_hwaddr(uint64_t addr,
+                                          GByteArray *data, size_t len);
+
  /**
   * qemu_plugin_scoreboard_new() - alloc a new scoreboard
   *
plugins/api.c +17 -0
index 2ff13d09de..c87bed6641 100644
@@ -527,6 +527,22 @@ GArray *qemu_plugin_get_registers(void)
      return create_register_handles(regs);
  }
 
+bool qemu_plugin_read_cpu_memory_hwaddr(uint64_t addr,
+                                        GByteArray *data, uint64_t len)
+{
+#ifndef CONFIG_USER_ONLY
+    if (len == 0) {
+        return false;
+    }
+
+    g_byte_array_set_size(data, len);
+    cpu_physical_memory_rw(addr, (void *)data->data, len, 0);
+    return true;
+#else
+    return false;
+#endif
+}
+
  int qemu_plugin_read_register(struct qemu_plugin_register *reg, GByteArray *buf)
  {
      g_assert(current_cpu);
@@ -534,6 +550,7 @@ int qemu_plugin_read_register(struct qemu_plugin_register *reg, GByteArray *buf)
      return gdb_read_register(current_cpu, buf, GPOINTER_TO_INT(reg) - 1);
  }
 
+
  struct qemu_plugin_scoreboard *qemu_plugin_scoreboard_new(size_t element_size)
  {
      return plugin_scoreboard_new(element_size);
plugins/qemu-plugins.symbols +0 -0
index ca773d8d9f..5d9cfd71bb 100644

plugins/qemu-plugins.symbols line 24

Quoted 5 lines
plugins/qemu-plugins.symbols +2 -0
@@ -20,6 +20,8 @@
    qemu_plugin_num_vcpus;
    qemu_plugin_outs;
    qemu_plugin_path_to_binary;
+  qemu_plugin_read_cpu_memory_hwaddr;
+  qemu_plugin_read_io_memory_hwaddr;

As you mentioned, you can remove the second one for v2.

Quoted 3 lines
plugins/qemu-plugins.symbols +0 -0
    qemu_plugin_read_register;
    qemu_plugin_register_atexit_cb;
    qemu_plugin_register_flush_cb;
#4 lore raw

And by the way, feel free to integrate this with your other series (as it's a very similar topic) in a v3, so we can review both at the same time.

Thanks,
Pierrick

Quoted 104 lines · Pierrick Bouvier

Hi Rowan,

thanks for this good complement on the virt address read function.

However, to be able to merge a new plugins API function, we must have a concrete usage of it, through one of the existing plugin. What could be a good demonstration of value brought by being able to read a physical address?

Thanks,
Pierrick

Quoted 80 lines · Rowan Hart
---
   include/qemu/qemu-plugin.h   | 22 ++++++++++++++++++++++
   plugins/api.c                | 17 +++++++++++++++++
   plugins/qemu-plugins.symbols |  2 ++
   3 files changed, 41 insertions(+)
include/qemu/qemu-plugin.h +22 -0
index c71c705b69..25f39c0960 100644
@@ -868,6 +868,28 @@ QEMU_PLUGIN_API
   int qemu_plugin_read_register(struct qemu_plugin_register *handle,
                                 GByteArray *buf);
 
+/**
+ * qemu_plugin_read_cpu_memory_hwaddr() - read CPU memory from hwaddr
+ *
+ * @addr: A virtual address to read from
+ * @data: A byte array to store data into
+ * @len: The number of bytes to read, starting from @addr
+ *
+ * @len bytes of data is read starting at @addr and stored into @data. If @data
+ * is not large enough to hold @len bytes, it will be expanded to the necessary
+ * size, reallocating if necessary. @len must be greater than 0.
+ *
+ * This function does not ensure writes are flushed prior to reading, so
+ * callers should take care when calling this function in plugin callbacks to
+ * avoid attempting to read data which may not yet be written and should use
+ * the memory callback API instead.
+ *
+ * Returns true on success and false on failure.
+ */
+QEMU_PLUGIN_API
+bool qemu_plugin_read_cpu_memory_hwaddr(uint64_t addr,
+                                          GByteArray *data, size_t len);
+
   /**
    * qemu_plugin_scoreboard_new() - alloc a new scoreboard
    *
plugins/api.c +17 -0
index 2ff13d09de..c87bed6641 100644
@@ -527,6 +527,22 @@ GArray *qemu_plugin_get_registers(void)
       return create_register_handles(regs);
   }
 
+bool qemu_plugin_read_cpu_memory_hwaddr(uint64_t addr,
+                                        GByteArray *data, uint64_t len)
+{
+#ifndef CONFIG_USER_ONLY
+    if (len == 0) {
+        return false;
+    }
+
+    g_byte_array_set_size(data, len);
+    cpu_physical_memory_rw(addr, (void *)data->data, len, 0);
+    return true;
+#else
+    return false;
+#endif
+}
+
   int qemu_plugin_read_register(struct qemu_plugin_register *reg, GByteArray *buf)
   {
       g_assert(current_cpu);
@@ -534,6 +550,7 @@ int qemu_plugin_read_register(struct qemu_plugin_register *reg, GByteArray *buf)
       return gdb_read_register(current_cpu, buf, GPOINTER_TO_INT(reg) - 1);
   }
 
+
   struct qemu_plugin_scoreboard *qemu_plugin_scoreboard_new(size_t element_size)
   {
       return plugin_scoreboard_new(element_size);
plugins/qemu-plugins.symbols +0 -0
index ca773d8d9f..5d9cfd71bb 100644

plugins/qemu-plugins.symbols line 24

Quoted 5 lines
plugins/qemu-plugins.symbols +2 -0
@@ -20,6 +20,8 @@
     qemu_plugin_num_vcpus;
     qemu_plugin_outs;
     qemu_plugin_path_to_binary;
+  qemu_plugin_read_cpu_memory_hwaddr;
+  qemu_plugin_read_io_memory_hwaddr;

As you mentioned, you can remove the second one for v2.

Quoted 3 lines
plugins/qemu-plugins.symbols +0 -0
     qemu_plugin_read_register;
     qemu_plugin_register_atexit_cb;
     qemu_plugin_register_flush_cb;
#5 lore raw
In reply to Rowan Hart

Rowan Hart <[email protected]> writes:

Apologies for the delay, I realise this has been sitting on my review queue too long.

Quoted 12 lines
---
 include/qemu/qemu-plugin.h   | 22 ++++++++++++++++++++++
 plugins/api.c                | 17 +++++++++++++++++
 plugins/qemu-plugins.symbols |  2 ++
 3 files changed, 41 insertions(+)
include/qemu/qemu-plugin.h +0 -0
index c71c705b69..25f39c0960 100644

include/qemu/qemu-plugin.h QEMU_PLUGIN_API · line 874

Quoted 7 lines
include/qemu/qemu-plugin.h +4 -0
@@ -868,6 +868,28 @@ QEMU_PLUGIN_API
 int qemu_plugin_read_register(struct qemu_plugin_register *handle,
                               GByteArray *buf);
 
+/**
+ * qemu_plugin_read_cpu_memory_hwaddr() - read CPU memory from hwaddr
+ *
+ * @addr: A virtual address to read from

A physical address

include/qemu/qemu-plugin.h QEMU_PLUGIN_API · line 885

Quoted 11 lines
include/qemu/qemu-plugin.h +11 -0
+ * @data: A byte array to store data into
+ * @len: The number of bytes to read, starting from @addr
+ *
+ * @len bytes of data is read starting at @addr and stored into @data. If @data
+ * is not large enough to hold @len bytes, it will be expanded to the necessary
+ * size, reallocating if necessary. @len must be greater than 0.
+ *
+ * This function does not ensure writes are flushed prior to reading, so
+ * callers should take care when calling this function in plugin callbacks to
+ * avoid attempting to read data which may not yet be written and should use
+ * the memory callback API instead.

Maybe we should be clear this can only be called from a vCPU context? See bellow.

Quoted 18 lines
include/qemu/qemu-plugin.h +7 -0
+ *
+ * Returns true on success and false on failure.
+ */
+QEMU_PLUGIN_API
+bool qemu_plugin_read_cpu_memory_hwaddr(uint64_t addr,
+                                          GByteArray *data, size_t len);
+
 /**
  * qemu_plugin_scoreboard_new() - alloc a new scoreboard
  *
plugins/api.c +0 -0
index 2ff13d09de..c87bed6641 100644
@@ -527,6 +527,22 @@ GArray *qemu_plugin_get_registers(void)
     return create_register_handles(regs);
 }
 

plugins/api.c GArray *qemu_plugin_get_registers(void) · line 539

Quoted 10 lines
plugins/api.c +10 -0
+bool qemu_plugin_read_cpu_memory_hwaddr(uint64_t addr,
+                                        GByteArray *data, uint64_t len)
+{
+#ifndef CONFIG_USER_ONLY
+    if (len == 0) {
+        return false;
+    }
+
+    g_byte_array_set_size(data, len);
+    cpu_physical_memory_rw(addr, (void *)data->data, len, 0);

One concern here is that cpu_physical_memory_* swallows any error conditions so the user might not get what they are expecting even if we return true here.

It would be safer I think to use address_space_read_full with &address_space_memory and MEMTXATTRS_UNSPECIFIED and check the result so we can signal to the user when it fails.

However that does gloss over some details because you can have multiple address spaces. As each vCPU can potentially have its own maybe we want:

  g_assert(current_cpu);
  res = address_space_read_full(current_cpu->as, addr, attrs, buf, len);
  return res == MEMTX_OK ? true : false;

However even that elides a complexity because a CPU can have multiple AddressSpaces depending on what mode it is in.

For example Arm can have normal, secure and potentially normal and secure versions of tag memory. Currently we have no way of exposing that information to plugins (and maybe we don't want to, currently arm_addressspace() is only used for internal page table walking and some stack stuff).

x86 has two potential address spaces with the extra one being used for I think System Management Mode.

Quoted 30 lines
plugins/api.c +7 -0
+    return true;
+#else
+    return false;
+#endif
+}
+
 int qemu_plugin_read_register(struct qemu_plugin_register *reg, GByteArray *buf)
 {
     g_assert(current_cpu);
@@ -534,6 +550,7 @@ int qemu_plugin_read_register(struct qemu_plugin_register *reg, GByteArray *buf)
     return gdb_read_register(current_cpu, buf, GPOINTER_TO_INT(reg) - 1);
 }
 
+
 struct qemu_plugin_scoreboard *qemu_plugin_scoreboard_new(size_t element_size)
 {
     return plugin_scoreboard_new(element_size);
plugins/qemu-plugins.symbols +2 -0
index ca773d8d9f..5d9cfd71bb 100644
@@ -20,6 +20,8 @@
   qemu_plugin_num_vcpus;
   qemu_plugin_outs;
   qemu_plugin_path_to_binary;
+  qemu_plugin_read_cpu_memory_hwaddr;
+  qemu_plugin_read_io_memory_hwaddr;
   qemu_plugin_read_register;
   qemu_plugin_register_atexit_cb;
   qemu_plugin_register_flush_cb;
Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#6 lore raw
In reply to Rowan Hart
Quoted 6 lines · Rowan Hart

This patch adds a single API function which allows reading from a guest CPU physical address.

I don't know of a good way to add a self-contained test for this feature to tests/tcg/plugins, but I did come up with a small test case to demonstrate the functionality using peiyuanix/riscv-os:

We have bare metal system tests (hello and memory) for i386, alpha, loongarch64, aarch64 and arm. If you fancy having a go at implementing a boot.S for riscv64 that would be super helpful for the check-tcg tests as a whole.

See:

tests/tcg/i386/system/boot.S
tests/tcg/alpha/system/boot.S
tests/tcg/loongarch64/system/boot.S
tests/tcg/aarch64/system/boot.S
tests/tcg/x86_64/system/boot.S
tests/tcg/arm/system/boot.S

for what is needed (basically a MMU-enabled flat memory map and some sort of emit char helper, probably using semihosting in this case)

Quoted 15 lines

First, grab and build the firmware code:

curl -o firmware.S https://raw.githubusercontent.com/peiyuanix/riscv-os/main/03-Bare-Metal-Hello-RISC-V/firmware.s
curl -o firmware.x https://raw.githubusercontent.com/peiyuanix/riscv-os/main/03-Bare-Metal-Hello-RISC-V/firmware.ld
riscv64-linux-gnu-as firmware.S -o firmware.o
riscv64-linux-gnu-ld -T firmware.x -o firmare firmware.o
riscv64-linux-gnu-objcopy -O binary -S firmware firmware.bin

Next, grab and build the plugin (just dumps from phys address on first instruction executed):

curl -o dump-riscv-firmware.c https://gist.githubusercontent.com/novafacing/5abc08052fab671a0fb26547810b4c55/raw/33772d614d6e36eae30e3405af34f149d7cc608b/dump-riscv-firmware.c
gcc -rdynamic -shared -fPIC -Iinclude/qemu $(pkg-config --cflags --libs glib-2.0) -o libdump-riscv-firmware.so dump-riscv-firmware.c

<snip>

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
author
#7 lore raw
Quoted 10 lines

See:

tests/tcg/i386/system/boot.S
tests/tcg/alpha/system/boot.S
tests/tcg/loongarch64/system/boot.S
tests/tcg/aarch64/system/boot.S
tests/tcg/x86_64/system/boot.S
tests/tcg/arm/system/boot.S

Quoted 2 lines

for what is needed (basically a MMU-enabled flat memory map and some sort of emit char helper, probably using semihosting in this case)

Sounds good! Sorry for the long pause, had some stuff going on :)

Investigating this now! I've never messed with semihosting, should be fun.