Skip to contentSkip to conversation

contrib/plugins: add dlcall to call host functions from a guest

PATCH Superseded Ziyang Zhang 2 participants 14 messages Jul 10 – Jul 16 Arm/Amlogic Meson SoC support

v11 of 12 Superseded Latest is v12 →

lore
Share & tools
lore
#0 lore raw

v11 cover contrib/plugins: add dlcall to call host functions from a guest

Hi,

This patch adds a single plugin, contrib/plugins/dlcall.c (~250 lines, no changes to QEMU core), that lets a linux-user guest call functions in the host's native shared libraries instead of emulating them.

It is the natural next step on top of the vCPU syscall-filter callback that I contributed and that was merged earlier:

https://lore.kernel.org/qemu-devel/[email protected]/

Why bother? Because it turns slow, instruction-by-instruction emulation of a library into a native host call. Some results, all on completely unmodified guest binaries:

* minizip (the stock zlib utility) compresses several times faster, because the actual deflate runs natively on the host instead of being translated.
* Real OpenGL/Vulkan games run under qemu-user: SuperTuxKart and Hollow Knight are playable, with their graphics calls going straight to the host GPU.

How it works
============

The guest makes a system call with a reserved number (4096 by default) that no real Linux ABI uses. Its first argument selects a pass-through operation, and the rest carry operands:

syscall(4096, op, arg1, arg2, ...)

          |     |    \............ operands (pointers / values)
          |     \................. which pass-through operation
          \....................... the reserved "magic" number

The plugin registers a vCPU syscall filter: before QEMU forwards a syscall to the host kernel, the filter runs, sees 4096, performs the operation on the host, writes the result back, and tells QEMU the syscall is consumed, so the real kernel never sees it.

The whole interface is just a handful of primitives:

* query a host attribute
* dlopen / dlclose a host shared library
* dlsym a symbol, and read the last dlerror
* invoke a resolved host function with a void(void *, void *) signature

That is all the plugin does. It knows nothing about zlib, X11 or OpenGL, or about any library's calling convention.

The same machinery also runs in reverse: when a host function needs to call back into the guest (a qsort comparator, an allocator, a GUI or game callback), control re-enters the guest to run the callback and then resumes the suspended host call. This reentry is what lets stateful, callback-driven APIs work, not just leaf functions.

Why the plugin belongs in QEMU, and the rest does not
=====================================================

Only the plugin lives in the tree. Everything else is ordinary userspace:

--- userspace (out of tree, not tied to any DBT) -------------

      guest: unmodified program  ->  guest runtime + thunk libs
  --------------------------------------------------------------
                 |  syscall(4096, op, args)   (only crossing point)
                 v
  === inside QEMU: THIS PATCH, ~250 lines ======================
      dlcall plugin:  dlopen / dlsym / invoke a host fn
  ==============================================================
                 |
                 v
  --- userspace (out of tree) ----------------------------------
      host: host runtime + thunk libs  ->  real libz / libGL ...

--------------------------------------------------------------

The split is deliberate, and it is why only this one file is proposed for the tree:

* This plugin defines the most general interaction interface for native
pass-through: the magic-syscall ABI between an emulated guest and its emulator. That contract is what every pass-through implementation builds on, so it belongs in a stable, shared place.
* It is also the only piece that is inherently QEMU-specific: it plugs into QEMU's syscall-filter hook and runs inside the QEMU process. The argument marshalling, calling conventions, callbacks/reentry and per-library coverage are not tied to any particular DBT and behave as ordinary userspace, so they should stay out of tree rather than couple QEMU to them.

Background: we presented this approach at KVM Forum 2025, "Lorelei: Enable QEMU to Leverage Native Shared Libraries":

https://www.youtube.com/watch?v=_jioQFm7wyU

The userspace side
==================

A fair point on v3 was that, on its own, the plugin is only half of an
interface: useful only if the other half (the guest/host runtimes and the per-library thunks) is public and specified, rather than a private demo. That half is now available as a standalone, documented, CI-tested project, Lorelei:

https://github.com/rover2024/lorelei

Lorelei provides the guest and host runtimes and a Thunk Library Compiler (TLC, built on Clang LibTooling) that reads a library's headers and generates the guest and host thunks automatically, including the awkward cases of function-pointer callbacks and variadic functions. It has CI for x86_64, arm64 and riscv64 hosts, and the devkit is exercised end to end on Ubuntu, Debian, Fedora and Arch. Lorelei and its thunk libraries are MIT-licensed.

Adding a thunk is deliberately little work, and needs nothing but a release. Lorelei's releases ship a prebuilt toolchain (a "devkit"); after unpacking one, a single command reads a library and its headers and generates both the guest and host thunks, with no per-function code, no manifest and no build system to drive:

  devkit/bin/LoreMakeThunk.py --name z --lib libz.so.1 --header zlib.h -o thunks

Two runnable examples come with Lorelei, each with a Makefile that does exactly this and runs the example under the plugin in one step:

* hello, a minimal one-function library:
https://github.com/rover2024/lorelei/tree/main/examples/hello
* demo, variadic functions and a callback that reenters the guest:
https://github.com/rover2024/lorelei/tree/main/examples/demo

If you do not have qemu or an x86_64 rootfs on hand, the project also ships a small Docker setup that builds qemu and runs the examples for you, so a single command reproduces the whole flow.

Because the plugin is not upstream yet, Lorelei currently builds and tests against the QEMU fork that carries it. Its releases ship prebuilt guest and host thunk trees for x86_64, aarch64 and riscv64 hosts, so trying it out does not require building from source:

https://github.com/rover2024/lorelei/releases

The plugin stays deliberately minimal and prescribes nothing about how thunking is done. Lorelei is one reference implementation of the userspace side. Any toolchain, or another instrumentation framework, can implement the same dlcall interface.

A from-scratch walkthrough of the bare mechanism, with the minizip and OpenGL/X11 examples above, is also available here:

https://github.com/rover2024/qemu-passthrough-test

It is fully opt-in (loaded with -plugin) and targets linux-user, where the guest and host already share a trust domain. The test cases use x86_64 guests and run on x86_64, arm64 and riscv64 Linux hosts.

Feedback on the plugin and on the pass-through approach is welcome.

Changes since v10:

Documentation only, no change to the plugin. Updated the docs/about/emulation.rst walkthrough to the current Lorelei userspace:

* The example now runs the x86_64 guest on an aarch64 host, showing the

    cross-architecture case rather than x86_64-on-x86_64.

* The thunk run no longer sets a LORELEI_THUNK_PATH variable: a guest thunk now records where its host thunk is, so discovery needs only LD_LIBRARY_PATH and the guest thunk directory is a flat thunks/<arch>.
* The devkit download resolves the release asset for the host arch, whose name carries the version.
* The hello library prints a plain sentence rather than a terse debug line.

Changes since v9:

Documentation only, no change to the plugin. Polished the docs/about/emulation.rst walkthrough for clarity: tightened the LD_LIBRARY_PATH explanation, named the directory-tree listing, and trimmed a couple of redundant phrasings.

Changes since v8:

Following Pierrick Bouvier's v8 request, the guest program in the docs/about/emulation.rst example no longer links with an rpath. It carries no rpath at all, so which libhello.so it loads is chosen entirely by LD_LIBRARY_PATH at run time: the baseline run puts the guest build there, and the thunk run puts the generated guest thunk there in its place. The plugin code is unchanged.

Changes since v7:

Reworked the docs/about/emulation.rst example into a complete, self-contained walkthrough, following Pierrick Bouvier's v7 review:

* Included the full source files and the exact compiler commands for the two libraries and the guest program, runnable end to end.
* The example now shows the thunk as a drop-in: the same unmodified guest program prints "from guest" with its own library, and "from host" once the generated guest thunk is placed ahead of it on LD_LIBRARY_PATH.
The plugin code is unchanged.

Changes since v6:

Addressing Pierrick Bouvier's v6 request that trying the userspace side need only the release, with no second repository to clone:

* The thunk generator now ships inside the devkit that Lorelei's releases provide, as bin/LoreMakeThunk.py. After unpacking a devkit, a single command reads a library and its headers and generates both the guest and host thunks, producing all the boilerplate itself. It takes plain command-line arguments, with no manifest or config file and no cmake/make/git behind it, so the whole flow is: download a devkit, run one command, run the guest under the plugin.
* docs/about/emulation.rst: rewrote the hello walkthrough to that flow (download a devkit, one LoreMakeThunk command, run under the plugin) and pointed the example links at the lorelei repository, where the examples now live. The plugin code is unchanged.

Changes since v5:

Following a discussion with Pierrick Bouvier on making the userspace side easy to try, this version focuses on that.

* docs/about/emulation.rst: expanded the example into a runnable hello walkthrough (the wrapped one-function library and its thunk manifest) and added links to the two example directories, hello and demo. Each example comes with a Makefile that builds the thunk and runs it under the plugin in one command, plus a small Docker setup for hosts without qemu or an x86_64 rootfs. Also reframed the section as a dynamic linking function call interception mechanism (tracing/auditing as further uses, acceleration as one use) and renamed it to "Dynamic Linking Call".
* The plugin header now names it "dlcall (Dynamic Linking Call)" and refers to Lorelei as one end-to-end userspace implementation. The plugin code is otherwise unchanged.

Changes since v4:

* Added a documentation patch for docs/about/emulation.rst (Pierrick Bouvier): what the plugin does, its trusted-guests and guest_base == 0 constraints, how to load it, and a minimal example, with a pointer to Lorelei.
* The plugin's header comment now points to Lorelei as one userspace implementation. The plugin code is otherwise unchanged.

Changes since v3:

* Lorelei, the userspace toolchain that implements this interface, is now a public, documented, CI-tested project, with a Thunk Library Compiler that generates the guest/host thunks automatically. This addresses the v3 feedback that the interface needs a public implementation behind it. The plugin code itself is unchanged.

Changes since v2:

* Dropped the RFC tag. The approach was positively received on v2.
* Rebased on master and adjusted the syscall-filter callback to its updated signature (int64_t sysret, added userdata, dropped the plugin id argument).

Changes since v1:

* Renamed the plugin from "passthrough" to "dlcall" (Pierrick Bouvier). The old name was too generic. The name "dlcall" reflects what the plugin actually does (dlopen/dlsym a host symbol and call it) and avoids confusion with QEMU's existing plugin hostcall concept (QEMU_PLUGIN_*_HOSTCALL).
* Made the magic syscall number configurable at load time via the "syscall_num=N" argument, defaulting to 4096 and rejecting values low enough to clash with a real syscall (Pierrick Bouvier).

v1: https://lore.kernel.org/qemu-devel/[email protected]/
v2: https://lore.kernel.org/qemu-devel/[email protected]/
v3: https://lore.kernel.org/qemu-devel/[email protected]/
v4: https://lore.kernel.org/qemu-devel/[email protected]/
v5: https://lore.kernel.org/qemu-devel/[email protected]/
v6: https://lore.kernel.org/qemu-devel/[email protected]/
v7: https://lore.kernel.org/qemu-devel/[email protected]/
v8: https://lore.kernel.org/qemu-devel/[email protected]/
v9: https://lore.kernel.org/qemu-devel/[email protected]/
v10: https://lore.kernel.org/qemu-devel/[email protected]/

Thanks,
Ziyang Zhang

Ziyang Zhang (2):
contrib/plugins: add a minimal dlcall plugin docs/about/emulation: document the dlcall plugin

 contrib/plugins/dlcall.c    | 248 ++++++++++++++++++++++++++++++++++++
 contrib/plugins/meson.build |   5 +
 docs/about/emulation.rst    | 156 +++++++++++++++++++++++
 3 files changed, 409 insertions(+)
 create mode 100644 contrib/plugins/dlcall.c
Signature · 1 line
2.34.1
#1 lore raw

PATCH v11 1/2 contrib/plugins: add a minimal dlcall plugin

Add a minimal dlcall plugin that lets the guest invoke host functions through magic system calls. The plugin registers a vCPU syscall filter callback that intercepts a reserved syscall number and dispatches a set of pass-through operations: querying host attributes, loading and freeing shared libraries, resolving symbols, retrieving the last library error, and invoking a host function through a common interface.

The magic syscall number defaults to 4096 and can be overridden at load time with the "syscall_num=N" argument; values low enough to clash with a real syscall are rejected.

---
 contrib/plugins/dlcall.c    | 248 ++++++++++++++++++++++++++++++++++++
 contrib/plugins/meson.build |   5 +
 2 files changed, 253 insertions(+)
 create mode 100644 contrib/plugins/dlcall.c
contrib/plugins/dlcall.c new file +248 -0
new file mode 100644
index 0000000000..9d2230b2d1
@@ -0,0 +1,248 @@
+/*
+ * Copyright (C) 2026, Ziyang Zhang <[email protected]>
+ *
+ * dlcall (Dynamic Linking Call) plugin: lets a linux-user guest invoke host
+ * functions by issuing a magic system call. The guest can ask QEMU to dlopen()
+ * a host shared library, dlsym() a symbol, and call it with guest-supplied
+ * arguments.
+ *
+ * The plugin intentionally keeps the QEMU side lightweight and prescribes
+ * nothing about how a library is thunked. Any toolchain can implement the
+ * userspace side. Lorelei is one end-to-end implementation (guest/host
+ * runtimes plus a thunk compiler that generates thunks from a library's
+ * headers):
+ * https://github.com/rover2024/lorelei
+ *
+ * See docs/about/emulation.rst|Dynamic Linking Call for details and examples.
+ *
+ * WARNING: trusted guests only. The guest can load arbitrary host libraries
+ * and execute arbitrary host code with arbitrary arguments, i.e. full code
+ * execution in the QEMU host process. It is NOT a sandbox and provides no
+ * isolation; only load it for guests you fully trust.
+ *
+ * WARNING: requires guest_base == 0, which is qemu-user's default. Pointer
+ * operands are dereferenced as host addresses directly, and the invoked host
+ * functions dereference guest pointers with no address translation, so guest
+ * and host must share a single address space. A non-zero guest_base (e.g. set
+ * via -B/-R) would make every pointer off by guest_base and hit unrelated
+ * host memory.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include <assert.h>
+#include <errno.h>
+#include <string.h>
+#include <stdio.h>
+#include <glib.h>
+#include <dlfcn.h>
+
+#include <qemu-plugin.h>
+
+QEMU_PLUGIN_EXPORT int qemu_plugin_version = QEMU_PLUGIN_VERSION;
+
+/*
+ * The magic system call number for dlcall.
+ *
+ * It defaults to DLCALL_SYSCALL_DEFAULT and can be overridden at load time
+ * with the "syscall_num=N" argument. To avoid hijacking a real syscall the
+ * guest might issue, N must be at least DLCALL_SYSCALL_MIN: every Linux ABI
+ * keeps its syscall numbers well below this; numbers from here up are free.
+ */
+enum {
+    DLCALL_SYSCALL_DEFAULT = 4096,
+    DLCALL_SYSCALL_MIN = 4096,
+};
+
+static int64_t dlcall_syscall_num = DLCALL_SYSCALL_DEFAULT;
+
+/*
+ * dlcall calling convention.
+ *
+ * The guest issues the magic system call (dlcall_syscall_num). The first
+ * argument (a1) is one of the call IDs below; the remaining arguments (a2, a3,
+ * a4, ...) are that ID's operands. All pointer operands are guest virtual
+ * addresses that the plugin dereferences as host addresses directly (see the
+ * guest_base requirement above). Results are written back through
+ * caller-provided "out" pointers rather than returned in the syscall value.
+ *
+ * The syscall return value (*sysret) only reports dispatch status: 0 on a
+ * recognised ID, -EINVAL for an unknown one. The actual success/failure of an
+ * operation (e.g. a NULL handle from dlopen) is delivered through its out
+ * pointer, exactly like the underlying libdl call.
+ *
+ * Operands per ID:
+ *
+ *   DLCALL_ID_GET_HOST_ATTRIBUTE
+ *     a2  const char *key        in:  attribute name to query
+ *     a3  const char **attr_ptr  out: matching value, or NULL if unknown
+ *
+ *   DLCALL_ID_LOAD_LIBRARY                            (wraps dlopen)
+ *     a2  const char *path       in:  library path
+ *     a3  int flags              in:  dlopen() flags (e.g. RTLD_NOW)
+ *     a4  void **handle_ptr      out: library handle, or NULL on failure
+ *
+ *   DLCALL_ID_GET_PROC_ADDRESS                        (wraps dlsym)
+ *     a2  void *handle           in:  library handle
+ *     a3  const char *name       in:  symbol name
+ *     a4  void **entry_ptr       out: symbol address, or NULL if not found
+ *
+ *   DLCALL_ID_FREE_LIBRARY                            (wraps dlclose)
+ *     a2  void *handle           in:  library handle
+ *     a3  int *ret_ptr           out: dlclose() return value (0 on success)
+ *
+ *   DLCALL_ID_GET_LIBRARY_ERROR                       (wraps dlerror)
+ *     a2  const char **error_ptr out: last libdl error string, or NULL
+ *
+ *   DLCALL_ID_INVOKE_PROC                             (calls the symbol)
+ *     a2  void *proc             in:  function pointer, signature
+ *                                     void (*)(void *arg1, void *arg2)
+ *     a3  void *arg1             in:  first argument forwarded to proc
+ *     a4  void *arg2             in:  second argument forwarded to proc
+ */
+enum DlcallID {
+    DLCALL_ID_GET_HOST_ATTRIBUTE,
+    DLCALL_ID_LOAD_LIBRARY,
+    DLCALL_ID_GET_PROC_ADDRESS,
+    DLCALL_ID_FREE_LIBRARY,
+    DLCALL_ID_GET_LIBRARY_ERROR,
+    DLCALL_ID_INVOKE_PROC,
+};
+
+static inline const char *query_host_attribute(const char *key)
+{
+    if (strcmp(key, "emu") == 0) {
+        return "qemu";
+    }
+    return NULL;
+}
+
+static inline void invoke_proc(void *proc, void *arg1, void *arg2)
+{
+    typedef void (*Func)(void * /*arg1*/, void * /*arg2*/);
+    Func func = (Func) proc;
+    func(arg1, arg2);
+}
+
+static bool vcpu_syscall_filter(unsigned int vcpu_index,
+                                int64_t num, uint64_t a1, uint64_t a2,
+                                uint64_t a3, uint64_t a4, uint64_t a5,
+                                uint64_t a6, uint64_t a7, uint64_t a8,
+                                int64_t *sysret, void *userdata)
+{
+    if (num == dlcall_syscall_num) {
+        switch (a1) {
+        /* Query host attribute by a reserved key. */
+        case DLCALL_ID_GET_HOST_ATTRIBUTE: {
+            const char *key = (const char *) a2;
+            const char **attr_ptr = (const char **) a3;
+            assert(attr_ptr);
+            *attr_ptr = query_host_attribute(key);
+            *sysret = 0;
+            break;
+        }
+
+        /* Load a shared library. */
+        case DLCALL_ID_LOAD_LIBRARY: {
+            const char *path = (const char *) a2;
+            int flags = (int) a3;
+            void **handle_ptr = (void **) a4;
+            assert(handle_ptr);
+            *handle_ptr = dlopen(path, flags);
+            *sysret = 0;
+            break;
+        }
+
+        /* Get the address of a function in a shared library. */
+        case DLCALL_ID_GET_PROC_ADDRESS: {
+            void *handle = (void *) a2;
+            const char *name = (const char *) a3;
+            void **entry_ptr = (void **) a4;
+            assert(entry_ptr);
+            *entry_ptr = dlsym(handle, name);
+            *sysret = 0;
+            break;
+        }
+
+        /* Free a shared library. */
+        case DLCALL_ID_FREE_LIBRARY: {
+            void *handle = (void *) a2;
+            int *ret_ptr = (int *) a3;
+            *ret_ptr = dlclose(handle);
+            *sysret = 0;
+            break;
+        }
+
+        /* Get the last error message for a library event. */
+        case DLCALL_ID_GET_LIBRARY_ERROR: {
+            const char **error_ptr = (const char **) a2;
+            *error_ptr = dlerror();
+            *sysret = 0;
+            break;
+        }
+
+        /* Invoke a function of a common interface. */
+        case DLCALL_ID_INVOKE_PROC: {
+            void *proc = (void *) a2;
+            void *arg1 = (void *) a3;
+            void *arg2 = (void *) a4;
+            assert(proc);
+            invoke_proc(proc, arg1, arg2);
+            *sysret = 0;
+            break;
+        }
+
+        default:
+            *sysret = -EINVAL;
+            break;
+        }
+        return true;
+    }
+    return false;
+}
+
+QEMU_PLUGIN_EXPORT int qemu_plugin_install(qemu_plugin_id_t id,
+                                           const qemu_info_t *info,
+                                           int argc, char **argv)
+{
+    if (info->system_emulation) {
+        fprintf(stderr, "plugin dlcall: only useful for user emulation\n");
+        return -1;
+    }
+
+    for (int i = 0; i < argc; i++) {
+        char *opt = argv[i];
+        g_auto(GStrv) tokens = g_strsplit(opt, "=", 2);
+        if (g_strcmp0(tokens[0], "syscall_num") == 0) {
+            const char *val = tokens[1];
+            char *endptr = NULL;
+            guint64 num;
+            if (!val || *val == '\0') {
+                fprintf(stderr,
+                        "plugin dlcall: missing value for syscall_num\n");
+                return -1;
+            }
+            num = g_ascii_strtoull(val, &endptr, 0);
+            if (*endptr != '\0' || g_strrstr(val, "-") != NULL) {
+                fprintf(stderr,
+                        "plugin dlcall: invalid syscall_num '%s'\n", val);
+                return -1;
+            }
+            if (num < DLCALL_SYSCALL_MIN || num > G_MAXINT64) {
+                fprintf(stderr,
+                        "plugin dlcall: syscall_num %s is out of range; "
+                        "it must be >= %d to avoid clashing with a real "
+                        "syscall\n", val, DLCALL_SYSCALL_MIN);
+                return -1;
+            }
+            dlcall_syscall_num = (int64_t) num;
+        } else {
+            fprintf(stderr, "plugin dlcall: unknown option '%s'\n", opt);
+            return -1;
+        }
+    }
+
+    qemu_plugin_register_vcpu_syscall_filter_cb(id, vcpu_syscall_filter, NULL);
+
+    return 0;
+}
contrib/plugins/meson.build +5 -0
index 099319e7a1..e7fc4d6d8f 100644
@@ -19,6 +19,11 @@ if host_os != 'windows'
   contrib_plugins += 'lockstep.c'
 endif
 
+if host_os == 'linux'
+  # dlcall passes guest calls through to host libraries; linux-user only
+  contrib_plugins += 'dlcall.c'
+endif
+
 if 'cpp' in all_languages
   contrib_plugins += 'cpp.cpp'
 endif
Signature · 1 line
2.34.1
#2 lore raw

PATCH v11 2/2 docs/about/emulation: document the dlcall plugin

In reply to Ziyang Zhang

Document the dlcall plugin under Example Plugins: what it does, the trusted- guests and guest_base == 0 constraints, how to load it, and a pointer to Lorelei, one end-to-end userspace implementation, for the toolchain and a runnable example.

---
 docs/about/emulation.rst | 156 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 156 insertions(+)
docs/about/emulation.rst +156 -0
index 3b4c365933..4f5ad1b3c6 100644
@@ -1046,6 +1046,162 @@ Count traps
 This plugin counts the number of interrupts (asynchronous events), exceptions
 (synchronous events) and host calls (e.g. semihosting) per cpu.
 
+Dynamic Linking Call
+....................
+
+``contrib/plugins/dlcall.c``
+
+This plugin provides a dynamic linking function call interception mechanism
+for linux-user guests: the guest hands a call off to the host, where the plugin
+runs native code in its place instead of the guest emulating it. Interception
+alone enables several uses, for instance tracing or auditing guest calls.
+One use is acceleration by leveraging the host's native shared libraries. For
+example, a thunk layer can run the stock zlib ``minizip`` utility under
+emulation while forwarding its ``deflate`` calls to the host's native zlib
+library (libz). This avoids emulating those selected library calls instruction
+by instruction.
+
+The guest issues a reserved "magic" system call (4096 by default, configurable
+with ``syscall_num=N``) whose first argument selects a pass-through operation:
+dlopen/dlclose a host library, dlsym a symbol, and invoke a resolved host
+function. The plugin performs the operation on the host and consumes the
+syscall, so the real kernel never sees it.
+
+.. warning::
+
+   Trusted guests only. The guest can load arbitrary host libraries and run
+   arbitrary code in the QEMU host process. The plugin is not a sandbox and
+   provides no isolation. It also requires ``guest_base == 0`` (qemu-user's
+   default), as guest pointers are dereferenced as host addresses with no
+   translation.
+
+The plugin intentionally keeps the QEMU side lightweight and knows nothing
+about any particular library or its calling convention. Turning a real library
+into working thunks, including argument marshalling, callbacks and variadic
+functions, is done entirely in userspace, and any toolchain can implement the
+interface.
+
+Loading the plugin is all that is required from QEMU's side:
+
+.. code-block:: shell
+
+   qemu-x86_64 -plugin contrib/plugins/libdlcall.so <guest-program> ...
+
+`Lorelei <https://github.com/rover2024/lorelei>`_ is one end-to-end userspace
+implementation of this: it provides the guest and host runtimes and an
+automated toolchain that generates the thunks from a library's headers, so guest
+library calls run on the host's native libraries. It supports an x86_64 guest
+running on an x86_64, aarch64 or riscv64 host.
+
+A minimal end-to-end example uses a one-function library, ``libhello.so``, built
+two ways: the guest build tags its output ``(from the guest)`` and the host
+build ``(from the host)``. An unmodified guest program ``main`` calls
+``hello("World", 7)``, and the thunk makes that same binary reach the host build
+in place of its own. The sources live under ``src/``:
+
+.. code-block:: c
+
+   /* src/hello.h */
+   void hello(const char *name, int lucky);
+
+.. code-block:: c
+
+   /* src/hello_guest.c */
+   #include "hello.h"
+   #include <stdio.h>
+
+   void hello(const char *name, int lucky)
+   {
+       printf("Hello, %s! Your lucky number is %d. (from the guest)\n", name, lucky);
+   }
+
+.. code-block:: c
+
+   /* src/hello_host.c */
+   #include "hello.h"
+   #include <stdio.h>
+
+   void hello(const char *name, int lucky)
+   {
+       printf("Hello, %s! Your lucky number is %d. (from the host)\n", name, lucky);
+   }
+
+.. code-block:: c
+
+   /* src/main.c */
+   #include "hello.h"
+
+   int main(void)
+   {
+       hello("World", 7);
+       return 0;
+   }
+
+Lorelei ships a prebuilt toolchain (a "devkit") in its releases. Download the
+one for your host and unpack it:
+
+.. code-block:: shell
+
+   # for an aarch64 host; see https://github.com/rover2024/lorelei/releases
+   ARCH=aarch64
+   TAG=$(curl -fsSL -o /dev/null -w '%{url_effective}' \
+         https://github.com/rover2024/lorelei/releases/latest | sed 's|.*/tag/||')
+   wget "https://github.com/rover2024/lorelei/releases/download/$TAG/lorelei-devkit-$ARCH-${TAG#v}.tar.xz"
+   tar -xf lorelei-devkit-$ARCH-*.tar.xz
+   DEVKIT=lorelei-devkit-$ARCH
+
+Build the guest ``libhello.so`` (x86_64) and the host ``libhello.so`` (this
+host's own architecture), then the guest program:
+
+.. code-block:: shell
+
+   mkdir -p build/guest build/host
+   $DEVKIT/bin/x86_64-linux-gnu-clang -shared -fPIC src/hello_guest.c -o build/guest/libhello.so
+   cc -shared -fPIC src/hello_host.c -o build/host/libhello.so
+   $DEVKIT/bin/x86_64-linux-gnu-clang src/main.c -Isrc -Lbuild/guest -lhello -o build/guest/main
+
+Run it under qemu:
+
+.. code-block:: shell
+
+   qemu-x86_64 -E LD_LIBRARY_PATH=build/guest build/guest/main
+
+which prints::
+
+   Hello, World! Your lucky number is 7. (from the guest)
+
+Now generate the thunk from the host ``libhello.so``. This produces a guest-side
+``libhello.so`` that stands in for the guest build, and a host-side thunk library
+that dispatches to the host build:
+
+.. code-block:: shell
+
+   $DEVKIT/bin/LoreMakeThunk.py --name hello --lib build/host/libhello.so \
+       --header hello.h -o thunks -- -Isrc
+
+Run the same ``main`` under the plugin. The call reaches the host build now:
+
+.. code-block:: shell
+
+   LD_LIBRARY_PATH=$DEVKIT/lib:build/host \
+       qemu-x86_64 -plugin contrib/plugins/libdlcall.so \
+       -E LD_LIBRARY_PATH=$DEVKIT/x86_64/lib:thunks/x86_64 \
+       build/guest/main
+
+which prints::
+
+   Hello, World! Your lucky number is 7. (from the host)
+
+.. list-table:: Dynamic Linking Call arguments
+  :widths: 20 80
+  :header-rows: 1
+
+  * - Option
+    - Description
+  * - syscall_num=N
+    - The magic syscall number the guest issues (default 4096). Must be high
+      enough not to clash with a real syscall.
+
 Other emulation features
 ------------------------
 
Signature · 1 line
2.34.1
#3 lore raw

Re: [PATCH v11 2/2] docs/about/emulation: document the dlcall plugin

Quoted 169 lines · Ziyang Zhang

Document the dlcall plugin under Example Plugins: what it does, the trusted- guests and guest_base == 0 constraints, how to load it, and a pointer to Lorelei, one end-to-end userspace implementation, for the toolchain and a runnable example.

---
 docs/about/emulation.rst | 156 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 156 insertions(+)
docs/about/emulation.rst +149 -0
index 3b4c365933..4f5ad1b3c6 100644
@@ -1046,6 +1046,162 @@ Count traps
 This plugin counts the number of interrupts (asynchronous events), exceptions
 (synchronous events) and host calls (e.g. semihosting) per cpu.
 
+Dynamic Linking Call
+....................
+
+``contrib/plugins/dlcall.c``
+
+This plugin provides a dynamic linking function call interception mechanism
+for linux-user guests: the guest hands a call off to the host, where the plugin
+runs native code in its place instead of the guest emulating it. Interception
+alone enables several uses, for instance tracing or auditing guest calls.
+One use is acceleration by leveraging the host's native shared libraries. For
+example, a thunk layer can run the stock zlib ``minizip`` utility under
+emulation while forwarding its ``deflate`` calls to the host's native zlib
+library (libz). This avoids emulating those selected library calls instruction
+by instruction.
+
+The guest issues a reserved "magic" system call (4096 by default, configurable
+with ``syscall_num=N``) whose first argument selects a pass-through operation:
+dlopen/dlclose a host library, dlsym a symbol, and invoke a resolved host
+function. The plugin performs the operation on the host and consumes the
+syscall, so the real kernel never sees it.
+
+.. warning::
+
+   Trusted guests only. The guest can load arbitrary host libraries and run
+   arbitrary code in the QEMU host process. The plugin is not a sandbox and
+   provides no isolation. It also requires ``guest_base == 0`` (qemu-user's
+   default), as guest pointers are dereferenced as host addresses with no
+   translation.
+
+The plugin intentionally keeps the QEMU side lightweight and knows nothing
+about any particular library or its calling convention. Turning a real library
+into working thunks, including argument marshalling, callbacks and variadic
+functions, is done entirely in userspace, and any toolchain can implement the
+interface.
+
+Loading the plugin is all that is required from QEMU's side:
+
+.. code-block:: shell
+
+   qemu-x86_64 -plugin contrib/plugins/libdlcall.so <guest-program> ...
+
+`Lorelei <https://github.com/rover2024/lorelei>`_ is one end-to-end userspace
+implementation of this: it provides the guest and host runtimes and an
+automated toolchain that generates the thunks from a library's headers, so guest
+library calls run on the host's native libraries. It supports an x86_64 guest
+running on an x86_64, aarch64 or riscv64 host.
+
+A minimal end-to-end example uses a one-function library, ``libhello.so``, built
+two ways: the guest build tags its output ``(from the guest)`` and the host
+build ``(from the host)``. An unmodified guest program ``main`` calls
+``hello("World", 7)``, and the thunk makes that same binary reach the host build
+in place of its own. The sources live under ``src/``:
+
+.. code-block:: c
+
+   /* src/hello.h */
+   void hello(const char *name, int lucky);
+
+.. code-block:: c
+
+   /* src/hello_guest.c */
+   #include "hello.h"
+   #include <stdio.h>
+
+   void hello(const char *name, int lucky)
+   {
+       printf("Hello, %s! Your lucky number is %d. (from the guest)\n", name, lucky);
+   }
+
+.. code-block:: c
+
+   /* src/hello_host.c */
+   #include "hello.h"
+   #include <stdio.h>
+
+   void hello(const char *name, int lucky)
+   {
+       printf("Hello, %s! Your lucky number is %d. (from the host)\n", name, lucky);
+   }
+
+.. code-block:: c
+
+   /* src/main.c */
+   #include "hello.h"
+
+   int main(void)
+   {
+       hello("World", 7);
+       return 0;
+   }
+
+Lorelei ships a prebuilt toolchain (a "devkit") in its releases. Download the
+one for your host and unpack it:
+
+.. code-block:: shell
+
+   # for an aarch64 host; see https://github.com/rover2024/lorelei/releases
+   ARCH=aarch64
+   TAG=$(curl -fsSL -o /dev/null -w '%{url_effective}' \
+         https://github.com/rover2024/lorelei/releases/latest | sed 's|.*/tag/||')
+   wget "https://github.com/rover2024/lorelei/releases/download/$TAG/lorelei-devkit-$ARCH-${TAG#v}.tar.xz"
+   tar -xf lorelei-devkit-$ARCH-*.tar.xz
+   DEVKIT=lorelei-devkit-$ARCH
+
+Build the guest ``libhello.so`` (x86_64) and the host ``libhello.so`` (this
+host's own architecture), then the guest program:
+
+.. code-block:: shell
+
+   mkdir -p build/guest build/host
+   $DEVKIT/bin/x86_64-linux-gnu-clang -shared -fPIC src/hello_guest.c -o build/guest/libhello.so
+   cc -shared -fPIC src/hello_host.c -o build/host/libhello.so
+   $DEVKIT/bin/x86_64-linux-gnu-clang src/main.c -Isrc -Lbuild/guest -lhello -o build/guest/main
+
+Run it under qemu:
+
+.. code-block:: shell
+
+   qemu-x86_64 -E LD_LIBRARY_PATH=build/guest build/guest/main
+
+which prints::
+
+   Hello, World! Your lucky number is 7. (from the guest)
+
+Now generate the thunk from the host ``libhello.so``. This produces a guest-side
+``libhello.so`` that stands in for the guest build, and a host-side thunk library
+that dispatches to the host build:
+
+.. code-block:: shell
+
+   $DEVKIT/bin/LoreMakeThunk.py --name hello --lib build/host/libhello.so \
+       --header hello.h -o thunks -- -Isrc
+
+Run the same ``main`` under the plugin. The call reaches the host build now:
+
+.. code-block:: shell
+
+   LD_LIBRARY_PATH=$DEVKIT/lib:build/host \
+       qemu-x86_64 -plugin contrib/plugins/libdlcall.so \
+       -E LD_LIBRARY_PATH=$DEVKIT/x86_64/lib:thunks/x86_64 \
+       build/guest/main
+
+which prints::
+
+   Hello, World! Your lucky number is 7. (from the host)
+
+.. list-table:: Dynamic Linking Call arguments
+  :widths: 20 80
+  :header-rows: 1

docs/about/emulation.rst Count traps · line 1207

Quoted 10 lines
docs/about/emulation.rst +7 -0
+
+  * - Option
+    - Description
+  * - syscall_num=N
+    - The magic syscall number the guest issues (default 4096). Must be high
+      enough not to clash with a real syscall.
+
 Other emulation features
 ------------------------
 

Great work, the workflow is now ready!

A few small issues are left with lib dependencies.

libstdc++
---------

LD_LIBRARY_PATH=$DEVKIT/lib:build/host \
./build/qemu-x86_64 -plugin ./build/contrib/plugins/libdlcall.so \
-E LD_LIBRARY_PATH=$DEVKIT/x86_64/lib:thunks/x86_64 \
-L /usr/x86_64-linux-gnu/ \
build/guest/main
build/guest/main: error while loading shared libraries: libstdc++.so.6:
cannot open shared object file: No such file or directory

Indeed, libstc++ is not available in machine's x86_64 sysroot by default. It works with -L $DEVKIT/x86_64/sysroot but as mentioned previously, we don't want to use another sysroot than the one that user has, since it can contain extra libraries we are not aware about (or Lorelei one can bring extra dependencies that we don't want).

So the last thing needed is to link statically libstdc++ for guest thunk and host thunk. In addition, you can do it for lorelei .so (I mean .so included in devkit), so thunking is fully self contained and does not require extra dependencies from the system where it runs. Same for libgcc.
Link parameters you need are -static-libstdc++ -static-libgcc. Once added, all thunks and lorelei devkit should be portable between different machines 👍.

The only base dependency we can assume will be present is libc (which also includes libm.so).
We could argue if c++ should be considered as a base dependency also. For debian (+ubuntu), fedora and archlinux, it's not.
See what is included in debian libc package. For instance, in our case:
https://packages.debian.org/trixie/all/libc6-amd64-cross/filelist

doc
---

While at it, please add -L /usr/x86_64-linux-gnu/ to command line so it matches the default people will use.

rpath
-----

It seems that .so in thunks have rpath set to $ORIGIN:$ORIGIN/../../lib, which should not be needed. LD_LIBRARY_PATH will specify where to find the correct libraries, please remove it.

You can use this command to double check this:
readelf -d thunks/libhello_HTL.so thunks/x86_64/libhello.so No RUNPATH should be present.

thunks/libhello_HTL.so has an explicit dependency to build/host/libhello.so. It should not include build/host and just link to libhello.so. Remove rpath also.

thunks/x86_64/libhello.so, remove rpath.

To clarify, lorelei-devkit-*/*/lib/* can use rpath since devkit itself is self contained. Thunks can not.

-----

Once solved, we'll have a clean and minimal workflow, which ensure thunks do not require any extra dependency.

With this, any user will be able to grab a lorelei devkit + a thunk tarball and use it as a drag and drop replacement for their system libraries, as long as it matches their version of course.

Keep up the good work!
Pierrick

#4 lore raw

Re: [PATCH v11 2/2] docs/about/emulation: document the dlcall plugin

Hi,

Quoted 68 lines · Pierrick Bouvier

Great work, the workflow is now ready!

A few small issues are left with lib dependencies.

libstdc++
---------

LD_LIBRARY_PATH=$DEVKIT/lib:build/host \
./build/qemu-x86_64 -plugin ./build/contrib/plugins/libdlcall.so \

(quoted diff) +0 -2
-E LD_LIBRARY_PATH=$DEVKIT/x86_64/lib:thunks/x86_64 \
-L /usr/x86_64-linux-gnu/ \

build/guest/main
build/guest/main: error while loading shared libraries: libstdc++.so.6:
cannot open shared object file: No such file or directory

Indeed, libstc++ is not available in machine's x86_64 sysroot by default. It works with -L $DEVKIT/x86_64/sysroot but as mentioned previously, we don't want to use another sysroot than the one that user has, since it can contain extra libraries we are not aware about (or Lorelei one can bring extra dependencies that we don't want).

So the last thing needed is to link statically libstdc++ for guest thunk and host thunk. In addition, you can do it for lorelei .so (I mean .so included in devkit), so thunking is fully self contained and does not require extra dependencies from the system where it runs. Same for libgcc.
Link parameters you need are -static-libstdc++ -static-libgcc. Once added, all thunks and lorelei devkit should be portable between different machines 👍.

The only base dependency we can assume will be present is libc (which also includes libm.so).
We could argue if c++ should be considered as a base dependency also. For debian (+ubuntu), fedora and archlinux, it's not.
See what is included in debian libc package. For instance, in our case:
https://packages.debian.org/trixie/all/libc6-amd64-cross/filelist

doc
---

While at it, please add -L /usr/x86_64-linux-gnu/ to command line so it matches the default people will use.

rpath
-----

It seems that .so in thunks have rpath set to $ORIGIN:$ORIGIN/../../lib, which should not be needed. LD_LIBRARY_PATH will specify where to find the correct libraries, please remove it.

You can use this command to double check this:
readelf -d thunks/libhello_HTL.so thunks/x86_64/libhello.so No RUNPATH should be present.

thunks/libhello_HTL.so has an explicit dependency to build/host/libhello.so. It should not include build/host and just link to libhello.so. Remove rpath also.

thunks/x86_64/libhello.so, remove rpath.

To clarify, lorelei-devkit-*/*/lib/* can use rpath since devkit itself is self contained. Thunks can not.

-----

Once solved, we'll have a clean and minimal workflow, which ensure thunks do not require any extra dependency.

Quoted 6 lines

With this, any user will be able to grab a lorelei devkit + a thunk tarball and use it as a drag and drop replacement for their system libraries, as long as it matches their version of course.

Keep up the good work!
Pierrick

Thanks for the careful review.

On libstdcxx / libgcc: rather than static-linking them into every thunk, I'd prefer to bundle the shared libraries in the devkit and let the thunks link against them dynamically. The reasoning:

The devkit already has to carry libstdcxx for LoreTLC itself, which is a large C++ program. Shipping it once as a shared library and having the guest and host thunks link that same copy avoids baking a static libstdc++ into every thunk, and keeps LoreTLC and the thunks on one runtime.

The portability you're after is preserved. libstdc++.so.6 and libgcc_s.so.1 sit beside the lorelei runtimes (the devkit's host lib/ and guest x86_64/lib/), so they are found on the same LD_LIBRARY_PATH the run already sets. A clean Fedora or Arch container with only libc installed runs a devkit + thunk pack directly, nothing else needed. The one base dependency we assume is still libc. The devkit brings its own libstdcxx and libgcc.

So static vs dynamic doesn't change what the user must have installed. I went with dynamic because it's the smaller and simpler of the two here.

For the rest: I'll drop the RPATH from the thunk libraries, make the host thunk record its real library's SONAME (libfoo.so) instead of the build path, and add -L /usr/x86_64-linux-gnu/ to the documented command.

#5 lore raw

Re: [PATCH v11 2/2] docs/about/emulation: document the dlcall plugin

Quoted 100 lines · Ziyang Zhang

Hi,

Quoted 68 lines · Pierrick Bouvier

Great work, the workflow is now ready!

A few small issues are left with lib dependencies.

libstdc++
---------

LD_LIBRARY_PATH=$DEVKIT/lib:build/host \
./build/qemu-x86_64 -plugin ./build/contrib/plugins/libdlcall.so \

(quoted diff) +0 -2
-E LD_LIBRARY_PATH=$DEVKIT/x86_64/lib:thunks/x86_64 \
-L /usr/x86_64-linux-gnu/ \

build/guest/main
build/guest/main: error while loading shared libraries: libstdc++.so.6:
cannot open shared object file: No such file or directory

Indeed, libstc++ is not available in machine's x86_64 sysroot by default. It works with -L $DEVKIT/x86_64/sysroot but as mentioned previously, we don't want to use another sysroot than the one that user has, since it can contain extra libraries we are not aware about (or Lorelei one can bring extra dependencies that we don't want).

So the last thing needed is to link statically libstdc++ for guest thunk and host thunk. In addition, you can do it for lorelei .so (I mean .so included in devkit), so thunking is fully self contained and does not require extra dependencies from the system where it runs. Same for libgcc.
Link parameters you need are -static-libstdc++ -static-libgcc. Once added, all thunks and lorelei devkit should be portable between different machines 👍.

The only base dependency we can assume will be present is libc (which also includes libm.so).
We could argue if c++ should be considered as a base dependency also. For debian (+ubuntu), fedora and archlinux, it's not.
See what is included in debian libc package. For instance, in our case:
https://packages.debian.org/trixie/all/libc6-amd64-cross/filelist

doc
---

While at it, please add -L /usr/x86_64-linux-gnu/ to command line so it matches the default people will use.

rpath
-----

It seems that .so in thunks have rpath set to $ORIGIN:$ORIGIN/../../lib, which should not be needed. LD_LIBRARY_PATH will specify where to find the correct libraries, please remove it.

You can use this command to double check this:
readelf -d thunks/libhello_HTL.so thunks/x86_64/libhello.so No RUNPATH should be present.

thunks/libhello_HTL.so has an explicit dependency to build/host/libhello.so. It should not include build/host and just link to libhello.so. Remove rpath also.

thunks/x86_64/libhello.so, remove rpath.

To clarify, lorelei-devkit-*/*/lib/* can use rpath since devkit itself is self contained. Thunks can not.

-----

Once solved, we'll have a clean and minimal workflow, which ensure thunks do not require any extra dependency.

Quoted 6 lines

With this, any user will be able to grab a lorelei devkit + a thunk tarball and use it as a drag and drop replacement for their system libraries, as long as it matches their version of course.

Keep up the good work!
Pierrick

Thanks for the careful review.

On libstdcxx / libgcc: rather than static-linking them into every thunk, I'd
prefer to bundle the shared libraries in the devkit and let the thunks link against them dynamically. The reasoning:

The devkit already has to carry libstdcxx for LoreTLC itself, which is a large
C++ program. Shipping it once as a shared library and having the guest and host thunks link that same copy avoids baking a static libstdc++ into every thunk, and keeps LoreTLC and the thunks on one runtime.

The portability you're after is preserved. libstdc++.so.6 and libgcc_s.so.1 sit beside the lorelei runtimes (the devkit's host lib/ and guest x86_64/lib/),
so they are found on the same LD_LIBRARY_PATH the run already sets. A clean Fedora or Arch container with only libc installed runs a devkit + thunk pack
directly, nothing else needed. The one base dependency we assume is still libc. The devkit brings its own libstdcxx and libgcc.

Quoted 4 lines

So static vs dynamic doesn't change what the user must have installed. I went
with dynamic because it's the smaller and simpler of the two here.

The problem is that if that the guest program, or the host library (host libhello) may depend on libstdc++. If they do and if they depend on a (newer) symbol not present in lorelei libstdc++, we break ABI. Same goes for libgcc.

By not doing this, you'll be forced to provide a lorelei devkit everytime a new libstdc++ is published, or a distribution updates its gcc.

The only solution is to embed libstd++ and libgcc statically, or rely on host libraries. If you choose the second option, you know have the opposite problem. So I think the only way to solve it is really static linking, even if it will slightly increase binaries size.

Quoted 5 lines

For the rest: I'll drop the RPATH from the thunk libraries, make the host thunk record its real library's SONAME (libfoo.so) instead of the build path,
and add -L /usr/x86_64-linux-gnu/ to the documented command.

Regards,
Pierrick

#6 lore raw

Re: [PATCH v11 2/2] docs/about/emulation: document the dlcall plugin

Quoted 5 lines · Pierrick Bouvier

The problem is that if that the guest program, or the host library (host libhello) may depend on libstdc++. If they do and if they depend on a (newer) symbol not present in lorelei libstdc++, we break ABI. Same goes for libgcc.

Quoted 8 lines

By not doing this, you'll be forced to provide a lorelei devkit everytime a new libstdc++ is published, or a distribution updates its gcc.

The only solution is to embed libstd++ and libgcc statically, or rely on host libraries. If you choose the second option, you know have the opposite problem. So I think the only way to solve it is really static linking, even if it will slightly increase binaries size.

The devkit already ships libstdc++ and libgcc anyway: its own clang/LoreTLC link them dynamically, and we bundle them so the toolchain even starts. So bundling libstdc++ for the thunk flow adds no new dependency, it just reuses one the devkit must carry. Static-linking the thunks wouldn't make the devkit any smaller or more self-contained either, since clang still needs the dynamic libstdc++; we'd only be adding static copies on top of it.

On the ABI risk itself: libstdc++ keeps a backward-compatible ABI, and the copy we bundle is recent, so a thunked host library only breaks if it was built against a strictly newer libstdc++ and references a new symbol. Even then it needs no new devkit, because the resolution order is the integrator's to choose. On a host newer than our bundle, putting the system libstdc++ ahead of the devkit's on LD_LIBRARY_PATH lets the host library bind the newest one, while clang and the Lorelei runtime keep working against it by backward compatibility. So a gcc update on the host doesn't force us to reship.

There's also an active reason not to static-link. Both the Lorelei runtime and the generated thunk libraries depend on libstdc++. If we statically link libstdc++ into a thunk while the host library it dispatches to links the system libstdc++ dynamically, the process ends up with two libstdc++ copies on the host side, right across the boundary where C++ objects and exceptions pass between the thunk and the real library. That can break:

- Exceptions: one thrown in the host library may not be caught in the thunk. The two copies have different std::type_info for the same type, so the match fails and it hits std::terminate.
- dynamic_cast / typeid across the boundary: they assume a single type_info per type, so they can start returning null or misidentifying types.
- Duplicated global state: two sets of the iostream objects, the locale/facets, and the exception emergency buffer.

Based on the above reasons, I prefer to do that than statically link and grow the big artifact. Does that hold for you?

Thanks,
Ziyang Zhang

#7 lore raw

Re: [PATCH v11 2/2] docs/about/emulation: document the dlcall plugin

Quoted 16 lines · Ziyang Zhang
Quoted 5 lines · Pierrick Bouvier

The problem is that if that the guest program, or the host library (host libhello) may depend on libstdc++. If they do and if they depend on a (newer) symbol not present in lorelei libstdc++, we break ABI. Same goes for libgcc.

Quoted 9 lines

By not doing this, you'll be forced to provide a lorelei devkit everytime a new libstdc++ is published, or a distribution updates its gcc.

The only solution is to embed libstd++ and libgcc statically, or rely on host libraries. If you choose the second option, you know have the opposite problem. So I think the only way to solve it is really static linking, even if it will slightly increase binaries size.

Quoted 8 lines

The devkit already ships libstdc++ and libgcc anyway: its own clang/ LoreTLC link them dynamically, and we bundle them so the toolchain even starts. So bundling libstdc++ for the thunk flow adds no new dependency, it just reuses one the devkit must carry. Static-linking the thunks wouldn't make the devkit any smaller or more self-contained either, since clang still needs the dynamic libstdc++; we'd only be adding static copies on top of it.

There is a difference between the toolchain dependencies, and thunks dependency. For the first one, it's perfectly find to provide libstdc++, for the second, it can create conflicts at runtime that may be very hard to diagnose and solve.

Quoted 10 lines

On the ABI risk itself: libstdc++ keeps a backward-compatible ABI, and the copy we bundle is recent, so a thunked host library only breaks if it was built against a strictly newer libstdc++ and references a new symbol. Even then it needs no new devkit, because the resolution order is the integrator's to choose. On a host newer than our bundle, putting the system libstdc++ ahead of the devkit's on LD_LIBRARY_PATH lets the host library bind the newest one, while clang and the Lorelei runtime keep working against it by backward compatibility. So a gcc update on the host doesn't force us to reship.

Sure can choose indeed. However, he has to find where such libraries are located on his system, and prepend that to LD_LIBRARY_PATH. Not the most user-friendly thing to guess and debug IMHO.

Quoted 8 lines

There's also an active reason not to static-link. Both the Lorelei runtime and the generated thunk libraries depend on libstdc++. If we statically link libstdc++ into a thunk while the host library it dispatches to links the system libstdc++ dynamically, the process ends up with two libstdc++ copies on the host side, right across the boundary where C++ objects and exceptions pass between the thunk and the real library. That can break:

Quoted 9 lines

- Exceptions: one thrown in the host library may not be caught in the thunk. The two copies have different std::type_info for the same type, so the match fails and it hits std::terminate.
- dynamic_cast / typeid across the boundary: they assume a single type_info per type, so they can start returning null or misidentifying types.
- Duplicated global state: two sets of the iostream objects, the locale/ facets, and the exception emergency buffer.

That's a good point, but does lorelei even provide a solution for that? I guess that exception raised from an host library and sent back to guest binary with target specific types will not result in what is expected at the moment.

Quoted 3 lines

Based on the above reasons, I prefer to do that than statically link and grow the big artifact. Does that hold for you?

Ok.

Quoted 3 lines

Thanks,
Ziyang Zhang

Regards,
Pierrick

#8 lore raw

Re: [PATCH v11 2/2] docs/about/emulation: document the dlcall plugin

Quoted 73 lines · Pierrick Bouvier
Quoted 16 lines · Ziyang Zhang
Quoted 5 lines · Pierrick Bouvier

The problem is that if that the guest program, or the host library (host libhello) may depend on libstdc++. If they do and if they depend on a (newer) symbol not present in lorelei libstdc++, we break ABI. Same goes for libgcc.

Quoted 9 lines

By not doing this, you'll be forced to provide a lorelei devkit everytime a new libstdc++ is published, or a distribution updates its gcc.

The only solution is to embed libstd++ and libgcc statically, or rely on host libraries. If you choose the second option, you know have the opposite problem. So I think the only way to solve it is really static linking, even if it will slightly increase binaries size.

Quoted 8 lines

The devkit already ships libstdc++ and libgcc anyway: its own clang/ LoreTLC link them dynamically, and we bundle them so the toolchain even starts. So bundling libstdc++ for the thunk flow adds no new dependency, it just reuses one the devkit must carry. Static-linking the thunks wouldn't make the devkit any smaller or more self-contained either, since clang still needs the dynamic libstdc++; we'd only be adding static copies on top of it.

There is a difference between the toolchain dependencies, and thunks dependency. For the first one, it's perfectly find to provide libstdc++, for the second, it can create conflicts at runtime that may be very hard to diagnose and solve.

Quoted 10 lines

On the ABI risk itself: libstdc++ keeps a backward-compatible ABI, and the copy we bundle is recent, so a thunked host library only breaks if it was built against a strictly newer libstdc++ and references a new symbol. Even then it needs no new devkit, because the resolution order is the integrator's to choose. On a host newer than our bundle, putting the system libstdc++ ahead of the devkit's on LD_LIBRARY_PATH lets the host library bind the newest one, while clang and the Lorelei runtime keep working against it by backward compatibility. So a gcc update on the host doesn't force us to reship.

Sure can choose indeed. However, he has to find where such libraries are located on his system, and prepend that to LD_LIBRARY_PATH. Not the most user-friendly thing to guess and debug IMHO.

Quoted 8 lines

There's also an active reason not to static-link. Both the Lorelei runtime and the generated thunk libraries depend on libstdc++. If we statically link libstdc++ into a thunk while the host library it dispatches to links the system libstdc++ dynamically, the process ends up with two libstdc++ copies on the host side, right across the boundary where C++ objects and exceptions pass between the thunk and the real library. That can break:

Quoted 9 lines

- Exceptions: one thrown in the host library may not be caught in the thunk. The two copies have different std::type_info for the same type, so the match fails and it hits std::terminate.
- dynamic_cast / typeid across the boundary: they assume a single type_info per type, so they can start returning null or misidentifying types.
- Duplicated global state: two sets of the iostream objects, the locale/ facets, and the exception emergency buffer.

That's a good point, but does lorelei even provide a solution for that? I guess that exception raised from an host library and sent back to guest binary with target specific types will not result in what is expected at the moment.

Quoted 3 lines

Based on the above reasons, I prefer to do that than statically link and grow the big artifact. Does that hold for you?

Quoted 8 lines

Ok.

Quoted 3 lines

Thanks,
Ziyang Zhang

Regards,
Pierrick

Wait a second. I missed something.

Sorry, last time I answered without actually checking the libstdc++ situation on the four distros. You were right to push on it, so I looked properly.

You were right that the x86_64 sysroot is the crux: libc6-amd64-cross gives /usr/x86_64-linux-gnu/ a libc but no libstdc++, so the guest side has no system libstdc++ to fall back on. That's the side we should keep providing ourselves, and the devkit already ships the x86_64 libstdc++ next to the guest runtime for exactly that. It's only a default, though: if a guest executable needs a newer libstdc++, the user can put a newer x86_64 libstdc++.so ahead of ours on the guest LD_LIBRARY_PATH (the -E one) and it takes precedence, while our runtime keeps working against it by backward compatibility. So we are not locking anyone to our version.

The host side is different, there libstdc++ is reliably present. I checked each distro's stock minimal base image, with nothing installed on top:

Debian 12: libstdc++6 12.2.0-14+deb12u1
Ubuntu 24.04: libstdc++6 14.2.0-4ubuntu2~24.04.1
Fedora 40: libstdc++-14.2.1-3.fc40
Arch (2024-08 snapshot): libstdc++.so.6 from gcc-libs 14.2.1
It ships out of the box on all four, and the C compiler pulls it in on Debian, Ubuntu and Arch.

So the split I'd propose: provide libstdc++ only on the guest side (the x86_64 sysroot has none), and on the host side rely on the system's, so the host thunk and the real host library both bind the host's own libstdc++ rather than a copy we ship. That directly addresses the ABI point you raised: nothing we ship shadows the host's libstdc++, so a thunk never forces an older libstdc++ onto a host library built against a newer one.

Does that split sound right to you?

#9 lore raw

Re: [PATCH v11 2/2] docs/about/emulation: document the dlcall plugin

Quoted 122 lines · Ziyang Zhang
Quoted 76 lines · Pierrick Bouvier
Quoted 19 lines · Ziyang Zhang
Quoted 11 lines · Pierrick Bouvier

The problem is that if that the guest program, or the host library (host
libhello) may depend on libstdc++. If they do and if they depend on a (newer) symbol not present in lorelei libstdc++, we break ABI. Same goes
for libgcc.

By not doing this, you'll be forced to provide a lorelei devkit everytime a new libstdc++ is published, or a distribution updates its gcc.

Quoted 6 lines

The only solution is to embed libstd++ and libgcc statically, or rely on
host libraries. If you choose the second option, you know have the opposite problem. So I think the only way to solve it is really static linking, even if it will slightly increase binaries size.

Quoted 8 lines

The devkit already ships libstdc++ and libgcc anyway: its own clang/ LoreTLC link them dynamically, and we bundle them so the toolchain even starts. So bundling libstdc++ for the thunk flow adds no new dependency, it just reuses one the devkit must carry. Static-linking the thunks wouldn't make the devkit any smaller or more self-contained either, since clang still needs the dynamic libstdc++; we'd only be adding static copies on top of it.

There is a difference between the toolchain dependencies, and thunks dependency. For the first one, it's perfectly find to provide libstdc++, for the second, it can create conflicts at runtime that may be very hard to diagnose and solve.

Quoted 10 lines

On the ABI risk itself: libstdc++ keeps a backward-compatible ABI, and the copy we bundle is recent, so a thunked host library only breaks if it was built against a strictly newer libstdc++ and references a new symbol. Even then it needs no new devkit, because the resolution order is the integrator's to choose. On a host newer than our bundle, putting the system libstdc++ ahead of the devkit's on LD_LIBRARY_PATH lets the host library bind the newest one, while clang and the Lorelei runtime keep working against it by backward compatibility. So a gcc update on the host doesn't force us to reship.

Sure can choose indeed. However, he has to find where such libraries are located on his system, and prepend that to LD_LIBRARY_PATH. Not the most user-friendly thing to guess and debug IMHO.

Quoted 8 lines

There's also an active reason not to static-link. Both the Lorelei runtime and the generated thunk libraries depend on libstdc++. If we statically link libstdc++ into a thunk while the host library it dispatches to links the system libstdc++ dynamically, the process ends up with two libstdc++ copies on the host side, right across the boundary where C++ objects and exceptions pass between the thunk and the real library. That can break:

Quoted 9 lines

- Exceptions: one thrown in the host library may not be caught in the thunk. The two copies have different std::type_info for the same type, so the match fails and it hits std::terminate.
- dynamic_cast / typeid across the boundary: they assume a single type_info per type, so they can start returning null or misidentifying types.
- Duplicated global state: two sets of the iostream objects, the locale/ facets, and the exception emergency buffer.

That's a good point, but does lorelei even provide a solution for that? I guess that exception raised from an host library and sent back to guest binary with target specific types will not result in what is expected at the moment.

Quoted 3 lines

Based on the above reasons, I prefer to do that than statically link and grow the big artifact. Does that hold for you?

Quoted 8 lines

Ok.

Quoted 3 lines

Thanks,
Ziyang Zhang

Regards,
Pierrick

Wait a second. I missed something.

Sorry, last time I answered without actually checking the libstdc++ situation on the four distros. You were right to push on it, so I looked properly.

You were right that the x86_64 sysroot is the crux: libc6-amd64-cross gives /usr/x86_64-linux-gnu/ a libc but no libstdc++, so the guest side has no system libstdc++ to fall back on. That's the side we should keep providing ourselves, and the devkit already ships the x86_64 libstdc++ next to the guest runtime for exactly that. It's only a default, though:
if a guest executable needs a newer libstdc++, the user can put a newer x86_64 libstdc++.so ahead of ours on the guest LD_LIBRARY_PATH (the -E one) and it takes precedence, while our runtime keeps working against it by backward compatibility. So we are not locking anyone to our version.

The host side is different, there libstdc++ is reliably present. I checked each distro's stock minimal base image, with nothing installed on top:

Debian 12: libstdc++6 12.2.0-14+deb12u1
Ubuntu 24.04: libstdc++6 14.2.0-4ubuntu2~24.04.1
Fedora 40: libstdc++-14.2.1-3.fc40
Arch (2024-08 snapshot): libstdc++.so.6 from gcc-libs 14.2.1 It ships out of the box on all four, and the C compiler pulls it in on Debian, Ubuntu and Arch.

So the split I'd propose: provide libstdc++ only on the guest side (the x86_64 sysroot has none), and on the host side rely on the system's, so the host thunk and the real host library both bind the host's own libstdc++ rather than a copy we ship. That directly addresses the ABI point you raised: nothing we ship shadows the host's libstdc++, so a thunk never forces an older libstdc++ onto a host library built against a newer one.

Quoted 2 lines

Does that split sound right to you?

That's a reasonable default, and Lorelei can be updated for this without having to update anything on QEMU side (or its documentation).

If I had to choose, I would still do static linking for guest and host thunks, because I don't think we can handle exceptions correctly anyway. But the most important is that current example already works out of the box, which makes this series ready for upstream.

Regards,
Pierrick

#10 lore raw

Re: [PATCH v11 2/2] docs/about/emulation: document the dlcall plugin

Quoted 129 lines · Pierrick Bouvier
Quoted 122 lines · Ziyang Zhang
Quoted 76 lines · Pierrick Bouvier
Quoted 19 lines · Ziyang Zhang
Quoted 11 lines · Pierrick Bouvier

The problem is that if that the guest program, or the host library (host
libhello) may depend on libstdc++. If they do and if they depend on a (newer) symbol not present in lorelei libstdc++, we break ABI. Same goes
for libgcc.

By not doing this, you'll be forced to provide a lorelei devkit everytime a new libstdc++ is published, or a distribution updates its gcc.

Quoted 6 lines

The only solution is to embed libstd++ and libgcc statically, or rely on
host libraries. If you choose the second option, you know have the opposite problem. So I think the only way to solve it is really static linking, even if it will slightly increase binaries size.

Quoted 8 lines

The devkit already ships libstdc++ and libgcc anyway: its own clang/ LoreTLC link them dynamically, and we bundle them so the toolchain even starts. So bundling libstdc++ for the thunk flow adds no new dependency, it just reuses one the devkit must carry. Static-linking the thunks wouldn't make the devkit any smaller or more self-contained either, since clang still needs the dynamic libstdc++; we'd only be adding static copies on top of it.

There is a difference between the toolchain dependencies, and thunks dependency. For the first one, it's perfectly find to provide libstdc++, for the second, it can create conflicts at runtime that may be very hard to diagnose and solve.

Quoted 10 lines

On the ABI risk itself: libstdc++ keeps a backward-compatible ABI, and the copy we bundle is recent, so a thunked host library only breaks if it was built against a strictly newer libstdc++ and references a new symbol. Even then it needs no new devkit, because the resolution order is the integrator's to choose. On a host newer than our bundle, putting the system libstdc++ ahead of the devkit's on LD_LIBRARY_PATH lets the host library bind the newest one, while clang and the Lorelei runtime keep working against it by backward compatibility. So a gcc update on the host doesn't force us to reship.

Sure can choose indeed. However, he has to find where such libraries are located on his system, and prepend that to LD_LIBRARY_PATH. Not the most user-friendly thing to guess and debug IMHO.

Quoted 8 lines

There's also an active reason not to static-link. Both the Lorelei runtime and the generated thunk libraries depend on libstdc++. If we statically link libstdc++ into a thunk while the host library it dispatches to links the system libstdc++ dynamically, the process ends up with two libstdc++ copies on the host side, right across the boundary where C++ objects and exceptions pass between the thunk and the real library. That can break:

Quoted 9 lines

- Exceptions: one thrown in the host library may not be caught in the thunk. The two copies have different std::type_info for the same type, so the match fails and it hits std::terminate.
- dynamic_cast / typeid across the boundary: they assume a single type_info per type, so they can start returning null or misidentifying types.
- Duplicated global state: two sets of the iostream objects, the locale/ facets, and the exception emergency buffer.

That's a good point, but does lorelei even provide a solution for that? I guess that exception raised from an host library and sent back to guest binary with target specific types will not result in what is expected at the moment.

Quoted 3 lines

Based on the above reasons, I prefer to do that than statically link and grow the big artifact. Does that hold for you?

Quoted 8 lines

Ok.

Quoted 3 lines

Thanks,
Ziyang Zhang

Regards,
Pierrick

Wait a second. I missed something.

Sorry, last time I answered without actually checking the libstdc++ situation on the four distros. You were right to push on it, so I looked properly.

You were right that the x86_64 sysroot is the crux: libc6-amd64-cross gives /usr/x86_64-linux-gnu/ a libc but no libstdc++, so the guest side has no system libstdc++ to fall back on. That's the side we should keep providing ourselves, and the devkit already ships the x86_64 libstdc++ next to the guest runtime for exactly that. It's only a default, though:
if a guest executable needs a newer libstdc++, the user can put a newer x86_64 libstdc++.so ahead of ours on the guest LD_LIBRARY_PATH (the -E one) and it takes precedence, while our runtime keeps working against it by backward compatibility. So we are not locking anyone to our version.

The host side is different, there libstdc++ is reliably present. I checked each distro's stock minimal base image, with nothing installed on top:

Debian 12: libstdc++6 12.2.0-14+deb12u1
Ubuntu 24.04: libstdc++6 14.2.0-4ubuntu2~24.04.1
Fedora 40: libstdc++-14.2.1-3.fc40
Arch (2024-08 snapshot): libstdc++.so.6 from gcc-libs 14.2.1 It ships out of the box on all four, and the C compiler pulls it in on Debian, Ubuntu and Arch.

So the split I'd propose: provide libstdc++ only on the guest side (the x86_64 sysroot has none), and on the host side rely on the system's, so the host thunk and the real host library both bind the host's own libstdc++ rather than a copy we ship. That directly addresses the ABI point you raised: nothing we ship shadows the host's libstdc++, so a thunk never forces an older libstdc++ onto a host library built against a newer one.

Quoted 2 lines

Does that split sound right to you?

That's a reasonable default, and Lorelei can be updated for this without having to update anything on QEMU side (or its documentation).

Quoted 7 lines

If I had to choose, I would still do static linking for guest and host thunks, because I don't think we can handle exceptions correctly anyway. But the most important is that current example already works out of the box, which makes this series ready for upstream.

Regards,
Pierrick

Thank you, and thanks for the thorough review across all the revisions. Your feedback made both the plugin and the walkthrough much better than where they started.

The libstdc++ discussion was valuable too. I'll keep static linking for the thunks in mind, and the exception-across-the-boundary case is a fair point I want to revisit properly. Since all of that lives on the Lorelei side, we can iterate on it without touching QEMU.

Thanks,
Ziyang Zhang

#11 lore raw

Re: [PATCH v11 2/2] docs/about/emulation: document the dlcall plugin

Quoted 146 lines · Ziyang Zhang
Quoted 145 lines · Pierrick Bouvier
Quoted 133 lines · Ziyang Zhang
Quoted 87 lines · Pierrick Bouvier
Quoted 20 lines · Ziyang Zhang
Quoted 11 lines · Pierrick Bouvier

The problem is that if that the guest program, or the host library (host
libhello) may depend on libstdc++. If they do and if they depend on a (newer) symbol not present in lorelei libstdc++, we break ABI. Same goes
for libgcc.

By not doing this, you'll be forced to provide a lorelei devkit everytime a new libstdc++ is published, or a distribution updates its gcc.

Quoted 7 lines

The only solution is to embed libstd++ and libgcc statically, or rely on
host libraries. If you choose the second option, you know have the opposite problem. So I think the only way to solve it is really static
linking, even if it will slightly increase binaries size.

Quoted 10 lines

The devkit already ships libstdc++ and libgcc anyway: its own clang/ LoreTLC link them dynamically, and we bundle them so the toolchain even
starts. So bundling libstdc++ for the thunk flow adds no new dependency,
it just reuses one the devkit must carry. Static-linking the thunks wouldn't make the devkit any smaller or more self-contained either, since clang still needs the dynamic libstdc++; we'd only be adding static copies on top of it.

There is a difference between the toolchain dependencies, and thunks dependency. For the first one, it's perfectly find to provide libstdc++,
for the second, it can create conflicts at runtime that may be very hard
to diagnose and solve.

Quoted 11 lines

On the ABI risk itself: libstdc++ keeps a backward-compatible ABI, and the copy we bundle is recent, so a thunked host library only breaks if it was built against a strictly newer libstdc++ and references a new symbol. Even then it needs no new devkit, because the resolution order is the integrator's to choose. On a host newer than our bundle, putting
the system libstdc++ ahead of the devkit's on LD_LIBRARY_PATH lets the host library bind the newest one, while clang and the Lorelei runtime keep working against it by backward compatibility. So a gcc update on the host doesn't force us to reship.

Sure can choose indeed. However, he has to find where such libraries are
located on his system, and prepend that to LD_LIBRARY_PATH. Not the most
user-friendly thing to guess and debug IMHO.

Quoted 9 lines

There's also an active reason not to static-link. Both the Lorelei runtime and the generated thunk libraries depend on libstdc++. If we statically link libstdc++ into a thunk while the host library it dispatches to links the system libstdc++ dynamically, the process ends up with two libstdc++ copies on the host side, right across the boundary
where C++ objects and exceptions pass between the thunk and the real library. That can break:

Quoted 10 lines

- Exceptions: one thrown in the host library may not be caught in the thunk. The two copies have different std::type_info for the same type, so the match fails and it hits std::terminate.
- dynamic_cast / typeid across the boundary: they assume a single type_info per type, so they can start returning null or misidentifying types.
- Duplicated global state: two sets of the iostream objects, the locale/
facets, and the exception emergency buffer.

That's a good point, but does lorelei even provide a solution for that? I guess that exception raised from an host library and sent back to guest binary with target specific types will not result in what is expected at the moment.

Quoted 4 lines

Based on the above reasons, I prefer to do that than statically link and
grow the big artifact. Does that hold for you?

Quoted 8 lines

Ok.

Quoted 3 lines

Thanks,
Ziyang Zhang

Regards,
Pierrick

Wait a second. I missed something.

Sorry, last time I answered without actually checking the libstdc++ situation on the four distros. You were right to push on it, so I looked properly.

You were right that the x86_64 sysroot is the crux: libc6-amd64-cross gives /usr/x86_64-linux-gnu/ a libc but no libstdc++, so the guest side has no system libstdc++ to fall back on. That's the side we should keep providing ourselves, and the devkit already ships the x86_64 libstdc++ next to the guest runtime for exactly that. It's only a default, though:
if a guest executable needs a newer libstdc++, the user can put a newer x86_64 libstdc++.so ahead of ours on the guest LD_LIBRARY_PATH (the -E one) and it takes precedence, while our runtime keeps working against it by backward compatibility. So we are not locking anyone to our version.

The host side is different, there libstdc++ is reliably present. I checked each distro's stock minimal base image, with nothing installed on top:

Debian 12: libstdc++6 12.2.0-14+deb12u1
Ubuntu 24.04: libstdc++6 14.2.0-4ubuntu2~24.04.1
Fedora 40: libstdc++-14.2.1-3.fc40
Arch (2024-08 snapshot): libstdc++.so.6 from gcc-libs 14.2.1 It ships out of the box on all four, and the C compiler pulls it in on Debian, Ubuntu and Arch.

So the split I'd propose: provide libstdc++ only on the guest side (the x86_64 sysroot has none), and on the host side rely on the system's, so the host thunk and the real host library both bind the host's own libstdc++ rather than a copy we ship. That directly addresses the ABI point you raised: nothing we ship shadows the host's libstdc++, so a thunk never forces an older libstdc++ onto a host library built against a newer one.

Quoted 2 lines

Does that split sound right to you?

That's a reasonable default, and Lorelei can be updated for this without having to update anything on QEMU side (or its documentation).

If I had to choose, I would still do static linking for guest and host thunks, because I don't think we can handle exceptions correctly anyway. But the most important is that current example already works out of the box, which makes this series ready for upstream.

Quoted 7 lines
Quoted 2 lines

Regards,
Pierrick

Thank you, and thanks for the thorough review across all the revisions. Your feedback made both the plugin and the walkthrough much better than where they started.

I know it can be frustrating to address all the picky comments, but in this case, I really think it makes Lorelei better and easier to use, and not just to accommodate my personal taste. Hopefully it will help to make it more popular and integrated with other emulators also!

Quoted 5 lines

The libstdc++ discussion was valuable too. I'll keep static linking for the thunks in mind, and the exception-across-the-boundary case is a fair point I want to revisit properly. Since all of that lives on the Lorelei side, we can iterate on it without touching QEMU.

Yes, the important was to have the simplest example possible, without requiring any external dependency. I hope you can use it on Lorelei side as a minimal CI test to keep things working as they are.

Quoted 3 lines

Thanks,
Ziyang Zhang

Congrats again for your work on Lorelei!
You're welcome to share your paper here once it will be published :).

Cheers,
Pierrick

#12 lore raw

Re: [PATCH v11 2/2] docs/about/emulation: document the dlcall plugin

Quoted 4 lines · Pierrick Bouvier

I know it can be frustrating to address all the picky comments, but in this case, I really think it makes Lorelei better and easier to use, and not just to accommodate my personal taste. Hopefully it will help to make it more popular and integrated with other emulators also!

Not frustrating at all. Thank you for getting it into QEMU upstream, which makes it accessible to many more people, and I really appreciate your recognition of Lorelei's potential.

Quoted 3 lines

Yes, the important was to have the simplest example possible, without requiring any external dependency. I hope you can use it on Lorelei side as a minimal CI test to keep things working as they are.

That is a good idea, and I will keep the example as a standing check on the Lorelei side. Thank you for your meticulous guidance over these weeks, it has improved Lorelei's usability a great deal.

Quoted 2 lines

Congrats again for your work on Lorelei!
You're welcome to share your paper here once it will be published :).

Thank you, I will definitely share it once it is out.

We will keep exploring what native pass-through can do on top of the dlcall plugin. For instance, we already have pass-through working for C++ libraries.

On the case you mentioned earlier, running Arm or RISC-V programs on an x86_64 host, that is a real need for some companies, and Lorelei should be able to support it without much trouble, since it is ISA-neutral.

Thanks,
Ziyang Zhang

#13 lore raw

Re: [PATCH v11 2/2] docs/about/emulation: document the dlcall plugin

In reply to Pierrick Bouvier
Quoted 18 lines · Pierrick Bouvier
Quoted 8 lines

There's also an active reason not to static-link. Both the Lorelei runtime and the generated thunk libraries depend on libstdc++. If we statically link libstdc++ into a thunk while the host library it dispatches to links the system libstdc++ dynamically, the process ends up with two libstdc++ copies on the host side, right across the boundary where C++ objects and exceptions pass between the thunk and the real library. That can break:

Quoted 9 lines

- Exceptions: one thrown in the host library may not be caught in the thunk. The two copies have different std::type_info for the same type, so the match fails and it hits std::terminate.
- dynamic_cast / typeid across the boundary: they assume a single type_info per type, so they can start returning null or misidentifying types.
- Duplicated global state: two sets of the iostream objects, the locale/ facets, and the exception emergency buffer.

Quoted 4 lines

That's a good point, but does lorelei even provide a solution for that? I guess that exception raised from an host library and sent back to guest binary with target specific types will not result in what is expected at the moment.

Currently, the Lorelei host runtime disables exceptions. However, it is possible that the host thunk library enables exceptions, and these exceptions should be caught by the host library.

Additionally, in the Lorelei guest/host runtime, there is a logging module. Therefore, if the stdout buffer of the host library is not consistent, the output will become disordered.