Skip to contentSkip to conversation

dlcall: correct the syscall number claim and the guest requirements

PATCH In review Ziyang Zhang 2 participants 8 messages Jul 19 – Jul 21

lore
Share & tools
lore
#0 lore raw

cover dlcall: correct the syscall number claim and the guest requirements

Hi,

Three small follow-ups to the dlcall plugin merged for 11.1. There is no functional change: all of them are comment and documentation fixes.

The first one matters most. The plugin picks 4096 as its magic syscall number, and the comment justified that by saying every Linux ABI keeps its syscall numbers well below it. That is not true for MIPS O32, which bases its numbering at 4000, so 4000 + 96 is a real getpriority. Loading the plugin on such a guest would let the filter consume a legitimate getpriority() and read its arguments as a dlcall request, where the first one selects an operation and the second is taken as a string pointer.

Raising the number does not rescue that case, which is the part I had wrong at first. O32 answers numbers its table does not define with ENOSYS in cpu_loop(), before do_syscall() and therefore before the filter, so on O32 there is no number that is both free and reachable. Its N32 and N64 ABIs base at 6000 and 5000 and have no such gate, so they are unaffected.

I checked this under qemu-mips rather than only reading cpu_loop(), using a plugin that logs what its syscall filter receives. A plain getpriority() from the guest arrives at the filter as num=4096 a1=0, which is what dlcall would read as an operation ID with a NULL key, while syscall(8192, ...) returns ENOSYS and never reaches the filter at all.

There is an upper bound elsewhere too. arm32 answers anything past ARM_NR_BASE (0xf0000) with ENOSYS or SIGILL before do_syscall(), as the comment in tests/tcg/multiarch/test-plugin-syscall-filter.c already notes, so a large number is not a safe default either. 4096 sits below that and above the ordinary syscall numbers, which is why it works on the targets people use this with.

That test carried the same wrong reason, so the second patch corrects it. It picked 4096 believing no ISA uses the number, when what actually keeps it safe is that the filter matches on the first argument as well: a real syscall carrying this number falls through untouched. tests/tcg/plugins/setpc.c matches on the number alone, which works for the single test program it is loaded with, but the collision applies there too, so neither test shows that 4096 is free.

I have deliberately not changed the default, nor restricted which guests may load the plugin. The number is already configurable with syscall_num=N, and dlcall is generic infrastructure rather than something tied to one userspace project, so it should not decide which guest architectures people may experiment with. What was wrong here was the reasoning in the comments, so they now describe what a target actually does with the number.

The last patch extends the guest_base == 0 warning, since the guest must match the host's pointer width and endianness as well, and tidies two things in the
prose: a library is not turned into thunks but has thunks produced for it, and argument marshalling, callbacks and variadic functions belong with Lorelei rather than in the plugin's description, since they are precisely what it does not do.

Also in the first patch, two out pointers were dereferenced without the assert the others carry.

Thanks,
Ziyang Zhang

Ziyang Zhang (3):
contrib/plugins/dlcall: correct the syscall number claim, note the
data model
tests/tcg: correct why the magic syscall number is safe here docs/about/emulation: sharpen the dlcall boundary and its guest
requirements

 contrib/plugins/dlcall.c                      | 32 +++++++++++++------
 docs/about/emulation.rst                      | 31 ++++++++++++------
 .../multiarch/test-plugin-syscall-filter.c    |  9 ++++--
 3 files changed, 50 insertions(+), 22 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/3 contrib/plugins/dlcall: correct the syscall number claim, note the data model

The comment claimed every Linux ABI keeps its syscall numbers well below the minimum, and that the minimum is all N has to satisfy. Neither holds.

MIPS O32 bases its numbering at 4000, so the default 4096 is its getpriority. Raising N does not rescue it either, because O32 answers numbers its table does not define with ENOSYS before the filter runs, which leaves no number that is both free and reachable on that ABI. arm32 bounds N from above too, with ENOSYS or SIGILL past ARM_NR_BASE. Say all of this, so the number can be chosen with the target in mind.

guest_base == 0 is not the only requirement either. Host pointers are written back through the caller's out pointers, so the guest must match the host's pointer width and endianness. Fold that into the existing warning.

Also assert the two out pointers that lacked it, and point at Lorelei for argument marshalling, callbacks and variadic functions.

---
 contrib/plugins/dlcall.c | 32 +++++++++++++++++++++++---------
 1 file changed, 23 insertions(+), 9 deletions(-)
contrib/plugins/dlcall.c +23 -9
index 9d2230b2d1..b624735e62 100644
@@ -10,7 +10,8 @@
  * nothing about how a library is thunked. Any toolchain can implement the
  * userspace side. Lorelei is one end-to-end implementation (guest/host
  * runtimes plus a thunk compiler that generates thunks from a library's
- * headers):
+ * headers), and how it handles argument marshalling, callbacks and variadic
+ * functions can serve as a reference:
  * https://github.com/rover2024/lorelei
  *
  * See docs/about/emulation.rst|Dynamic Linking Call for details and examples.
@@ -20,12 +21,13 @@
  * execution in the QEMU host process. It is NOT a sandbox and provides no
  * isolation; only load it for guests you fully trust.
  *
- * WARNING: requires guest_base == 0, which is qemu-user's default. Pointer
- * operands are dereferenced as host addresses directly, and the invoked host
- * functions dereference guest pointers with no address translation, so guest
- * and host must share a single address space. A non-zero guest_base (e.g. set
- * via -B/-R) would make every pointer off by guest_base and hit unrelated
- * host memory.
+ * WARNING: requires guest_base == 0, which is qemu-user's default, and a
+ * guest whose pointer width and endianness match the host's. Pointer operands
+ * are dereferenced as host addresses directly, and the invoked host functions
+ * dereference guest pointers with no address translation, so guest and host
+ * must share a single address space and agree on how a pointer is stored. A
+ * non-zero guest_base (e.g. set via -B/-R) would make every pointer off by
+ * guest_base and hit unrelated host memory.
  *
  * SPDX-License-Identifier: GPL-2.0-or-later
  */
@@ -46,8 +48,18 @@ QEMU_PLUGIN_EXPORT int qemu_plugin_version = QEMU_PLUGIN_VERSION;
  *
  * It defaults to DLCALL_SYSCALL_DEFAULT and can be overridden at load time
  * with the "syscall_num=N" argument. To avoid hijacking a real syscall the
- * guest might issue, N must be at least DLCALL_SYSCALL_MIN: every Linux ABI
- * keeps its syscall numbers well below this; numbers from here up are free.
+ * guest might issue, N must be at least DLCALL_SYSCALL_MIN, which most Linux
+ * ABIs keep their syscall numbers well below.
+ *
+ * N also has to reach the filter at all, which bounds it from above in a
+ * target specific way: arm32 answers anything past ARM_NR_BASE (0xf0000) with
+ * ENOSYS or SIGILL before do_syscall() runs, while aarch64 has no such bound.
+ *
+ * MIPS O32 bases its numbering at 4000, so the default is a real syscall there
+ * (getpriority). Raising N does not help either, because O32 rejects numbers
+ * its table does not define, again before the filter runs, which leaves no
+ * number that is both free and reachable on that ABI. Its N32 and N64 ABIs
+ * base at 6000 and 5000 and have no such gate, so they are unaffected.
  */
 enum {
     DLCALL_SYSCALL_DEFAULT = 4096,
@@ -168,6 +180,7 @@ static bool vcpu_syscall_filter(unsigned int vcpu_index,
         case DLCALL_ID_FREE_LIBRARY: {
             void *handle = (void *) a2;
             int *ret_ptr = (int *) a3;
+            assert(ret_ptr);
             *ret_ptr = dlclose(handle);
             *sysret = 0;
             break;
@@ -176,6 +189,7 @@ static bool vcpu_syscall_filter(unsigned int vcpu_index,
         /* Get the last error message for a library event. */
         case DLCALL_ID_GET_LIBRARY_ERROR: {
             const char **error_ptr = (const char **) a2;
+            assert(error_ptr);
             *error_ptr = dlerror();
             *sysret = 0;
             break;
Signature · 1 line
2.34.1
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] contrib/plugins/dlcall: correct the syscall number claim, note the data model

Quoted 17 lines · Ziyang Zhang

The comment claimed every Linux ABI keeps its syscall numbers well below the minimum, and that the minimum is all N has to satisfy. Neither holds.

MIPS O32 bases its numbering at 4000, so the default 4096 is its getpriority. Raising N does not rescue it either, because O32 answers numbers its table does not define with ENOSYS before the filter runs, which leaves no number that is both free and reachable on that ABI. arm32 bounds N from above too, with ENOSYS or SIGILL past ARM_NR_BASE. Say all of this, so the number can be chosen with the target in mind.

guest_base == 0 is not the only requirement either. Host pointers are written back through the caller's out pointers, so the guest must match the host's pointer width and endianness. Fold that into the existing warning.

Also assert the two out pointers that lacked it, and point at Lorelei for argument marshalling, callbacks and variadic functions.

Quoted 7 lines
---
 contrib/plugins/dlcall.c | 32 +++++++++++++++++++++++---------
 1 file changed, 23 insertions(+), 9 deletions(-)
#3 lore raw

PATCH 2/3 tests/tcg: correct why the magic syscall number is safe here

In reply to Ziyang Zhang

The comment said 4096 was picked because no ISA in Linux uses it. The same comment already notes that mips 32 bits numbers from 4000, which makes 4096 its getpriority.

What actually keeps this test safe is the filter, which matches on the first argument as well, so a real syscall carrying this number is left alone. Say that instead.

---
 tests/tcg/multiarch/test-plugin-syscall-filter.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
tests/tcg/multiarch/test-plugin-syscall-filter.c +6 -3
index 951e338a7c..089fb2a110 100644
@@ -23,9 +23,12 @@ int main(int argc, char *argv[])
      * "linux-user/arm/cpu_loop.c:cpu_loop".
      * As well, some arch expect a minimum, like 4000 for mips 32 bits.
      *
-     * Therefore, we pick 4096 because, as of now, no ISA in Linux uses this
-     * number. This is just a test case; replace this number as needed in the
-     * future.
+     * Therefore, we pick 4096, which sits between those bounds. It is not
+     * unused everywhere though: mips 32 bits numbers from 4000, so 4096 is its
+     * getpriority. This test is unaffected because the filter also requires
+     * the first argument to be 0x66CCFF, so a real syscall carrying this
+     * number falls through untouched. This is just a test case, so replace
+     * this number as needed in the future.
      *
      * The corresponding syscall filter is implemented in
      * "tests/tcg/plugins/syscall.c".
Signature · 1 line
2.34.1
Reviewed-by on v1 2/3
#4 lore raw

Re: [PATCH 2/3] tests/tcg: correct why the magic syscall number is safe here

Quoted 8 lines · Ziyang Zhang

The comment said 4096 was picked because no ISA in Linux uses it. The same comment already notes that mips 32 bits numbers from 4000, which makes 4096 its getpriority.

What actually keeps this test safe is the filter, which matches on the first argument as well, so a real syscall carrying this number is left alone. Say that instead.

Quoted 7 lines
---
 tests/tcg/multiarch/test-plugin-syscall-filter.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
#5 lore raw

PATCH 3/3 docs/about/emulation: sharpen the dlcall boundary and its guest requirements

In reply to Ziyang Zhang

Record the same data model requirement as the plugin: guest_base == 0 is necessary but not sufficient.

Describe the magic syscall number the way the plugin now does. It has to be a number the guest ABI does not use and does not reject before the plugin sees it, rather than merely a high one, so show syscall_num= being used as well.

A library is not turned into thunks, it is left alone and the thunks are produced for it, so say that instead. Argument marshalling, callbacks and variadic functions are also what the plugin does not do, and listing them in its description blurs the boundary it draws. Move them to Lorelei, where they are pointed at as a reference.

---
 docs/about/emulation.rst | 31 +++++++++++++++++++++----------
 1 file changed, 21 insertions(+), 10 deletions(-)
docs/about/emulation.rst +21 -10
index b861501e85..c58149086c 100644
@@ -1072,14 +1072,13 @@ syscall, so the real kernel never sees it.
    Trusted guests only. The guest can load arbitrary host libraries and run
    arbitrary code in the QEMU host process. The plugin is not a sandbox and
    provides no isolation. It also requires ``guest_base == 0`` (qemu-user's
-   default), as guest pointers are dereferenced as host addresses with no
-   translation.
+   default) and a guest whose pointer width and endianness match the host's, as
+   guest pointers are dereferenced as host addresses with no translation.
 
 The plugin intentionally keeps the QEMU side lightweight and knows nothing
-about any particular library or its calling convention. Turning a real library
-into working thunks, including argument marshalling, callbacks and variadic
-functions, is done entirely in userspace, and any toolchain can implement the
-interface.
+about any particular library or its calling convention. Producing the thunks
+for a real library is done entirely in userspace, and any toolchain can
+implement the interface.
 
 Loading the plugin is all that is required from QEMU's side:
 
@@ -1087,11 +1086,21 @@ Loading the plugin is all that is required from QEMU's side:
 
    qemu-x86_64 -plugin contrib/plugins/libdlcall.so <guest-program> ...
 
+If the default number does not suit the guest ABI, pick another one, and build
+the userspace side to issue the same one:
+
+.. code-block:: shell
+
+   qemu-x86_64 -plugin contrib/plugins/libdlcall.so,syscall_num=8192 \
+       <guest-program> ...
+
 `Lorelei <https://github.com/rover2024/lorelei>`_ is one end-to-end userspace
 implementation of this: it provides the guest and host runtimes and an
 automated toolchain that generates the thunks from a library's headers, so guest
-library calls run on the host's native libraries. It supports an x86_64 guest
-running on an x86_64, aarch64 or riscv64 host.
+library calls run on the host's native libraries. How it handles the parts the
+plugin leaves out, including argument marshalling, callbacks and variadic
+functions, can serve as a reference. It supports an x86_64 guest running on an
+x86_64, aarch64 or riscv64 host.
 
 A minimal end-to-end example uses a one-function library, ``libhello.so``, built
 two ways: the guest build tags its output ``(from the guest)`` and the host
@@ -1201,8 +1210,10 @@ which prints::
   * - Option
     - Description
   * - syscall_num=N
-    - The magic syscall number the guest issues (default 4096). Must be high
-      enough not to clash with a real syscall.
+    - The magic syscall number the guest issues (default 4096). It must be a
+      number the guest ABI does not use for a real syscall, and does not
+      reject before the plugin sees it, which bounds the choice from both
+      sides.
 
 Other emulation features
 ------------------------
Signature · 1 line
2.34.1
Reviewed-by on v1 3/3
#6 lore raw

Re: [PATCH 3/3] docs/about/emulation: sharpen the dlcall boundary and its guest requirements

Quoted 13 lines · Ziyang Zhang

Record the same data model requirement as the plugin: guest_base == 0 is necessary but not sufficient.

Describe the magic syscall number the way the plugin now does. It has to be a number the guest ABI does not use and does not reject before the plugin sees it, rather than merely a high one, so show syscall_num= being used as well.

A library is not turned into thunks, it is left alone and the thunks are produced for it, so say that instead. Argument marshalling, callbacks and variadic functions are also what the plugin does not do, and listing them in its description blurs the boundary it draws. Move them to Lorelei, where they are pointed at as a reference.

Quoted 7 lines
---
 docs/about/emulation.rst | 31 +++++++++++++++++++++----------
 1 file changed, 21 insertions(+), 10 deletions(-)
#7 lore raw
In reply to Ziyang Zhang
Quoted 66 lines · Ziyang Zhang

Hi,

Three small follow-ups to the dlcall plugin merged for 11.1. There is no functional change: all of them are comment and documentation fixes.

The first one matters most. The plugin picks 4096 as its magic syscall number, and the comment justified that by saying every Linux ABI keeps its syscall numbers well below it. That is not true for MIPS O32, which bases its numbering at 4000, so 4000 + 96 is a real getpriority. Loading the plugin on such a guest would let the filter consume a legitimate getpriority() and read its arguments as a dlcall request, where the first one selects an operation and the second is taken as a string pointer.

Raising the number does not rescue that case, which is the part I had wrong at first. O32 answers numbers its table does not define with ENOSYS in cpu_loop(), before do_syscall() and therefore before the filter, so on O32 there is no number that is both free and reachable. Its N32 and N64 ABIs base at 6000 and 5000 and have no such gate, so they are unaffected.

I checked this under qemu-mips rather than only reading cpu_loop(), using a plugin that logs what its syscall filter receives. A plain getpriority() from the guest arrives at the filter as num=4096 a1=0, which is what dlcall would read as an operation ID with a NULL key, while syscall(8192, ...) returns ENOSYS and never reaches the filter at all.

There is an upper bound elsewhere too. arm32 answers anything past ARM_NR_BASE (0xf0000) with ENOSYS or SIGILL before do_syscall(), as the comment in tests/tcg/multiarch/test-plugin-syscall-filter.c already notes, so a large number is not a safe default either. 4096 sits below that and above the ordinary syscall numbers, which is why it works on the targets people use this with.

That test carried the same wrong reason, so the second patch corrects it. It picked 4096 believing no ISA uses the number, when what actually keeps it safe is that the filter matches on the first argument as well: a real syscall carrying this number falls through untouched. tests/tcg/plugins/setpc.c matches on the number alone, which works for the single test program it is loaded with, but the collision applies there too, so neither test shows that 4096 is free.

I have deliberately not changed the default, nor restricted which guests may load the plugin. The number is already configurable with syscall_num=N, and dlcall is generic infrastructure rather than something tied to one userspace project, so it should not decide which guest architectures people may experiment with. What was wrong here was the reasoning in the comments, so they now describe what a target actually does with the number.

The last patch extends the guest_base == 0 warning, since the guest must match the host's pointer width and endianness as well, and tidies two things in the
prose: a library is not turned into thunks but has thunks produced for it, and argument marshalling, callbacks and variadic functions belong with Lorelei rather than in the plugin's description, since they are precisely what it does not do.

Also in the first patch, two out pointers were dereferenced without the assert the others carry.

Thanks,
Ziyang Zhang

Ziyang Zhang (3):
contrib/plugins/dlcall: correct the syscall number claim, note the
data model
tests/tcg: correct why the magic syscall number is safe here docs/about/emulation: sharpen the dlcall boundary and its guest
requirements

Quoted 5 lines
 contrib/plugins/dlcall.c                      | 32 +++++++++++++------
 docs/about/emulation.rst                      | 31 ++++++++++++------
 .../multiarch/test-plugin-syscall-filter.c    |  9 ++++--
 3 files changed, 50 insertions(+), 22 deletions(-)

This was merged into master (83e8f88135cba892934b1d4d04f4c063e213700a). Thank you for your contribution!

Regards,
Pierrick