contrib/plugins: add dlcall to call host functions from a guest
v9 cover contrib/plugins: add dlcall to call host functions from a guest
Hi,
This patch adds a single plugin, contrib/plugins/dlcall.c (~250 lines, no changes to QEMU core), that lets a linux-user guest call functions in the host's native shared libraries instead of emulating them.
It is the natural next step on top of the vCPU syscall-filter callback that I contributed and that was merged earlier:
https://lore.kernel.org/qemu-devel/[email protected]/
Why bother? Because it turns slow, instruction-by-instruction emulation of a library into a native host call. Some results, all on completely unmodified guest binaries:
* minizip (the stock zlib utility) compresses several times faster, because the actual deflate runs natively on the host instead of being translated.
* Real OpenGL/Vulkan games run under qemu-user: SuperTuxKart and Hollow Knight are playable, with their graphics calls going straight to the host GPU.
How it works
============
The guest makes a system call with a reserved number (4096 by default) that no real Linux ABI uses. Its first argument selects a pass-through operation, and the rest carry operands:
syscall(4096, op, arg1, arg2, ...)
| | \............ operands (pointers / values)
| \................. which pass-through operation
\....................... the reserved "magic" numberThe plugin registers a vCPU syscall filter: before QEMU forwards a syscall to the host kernel, the filter runs, sees 4096, performs the operation on the host, writes the result back, and tells QEMU the syscall is consumed, so the real kernel never sees it.
The whole interface is just a handful of primitives:
* query a host attribute
* dlopen / dlclose a host shared library
* dlsym a symbol, and read the last dlerror
* invoke a resolved host function with a void(void *, void *) signature
That is all the plugin does. It knows nothing about zlib, X11 or OpenGL, or about any library's calling convention.
The same machinery also runs in reverse: when a host function needs to call back into the guest (a qsort comparator, an allocator, a GUI or game callback), control re-enters the guest to run the callback and then resumes the suspended host call. This reentry is what lets stateful, callback-driven APIs work, not just leaf functions.
Why the plugin belongs in QEMU, and the rest does not
=====================================================
Only the plugin lives in the tree. Everything else is ordinary userspace:
--- userspace (out of tree, not tied to any DBT) -------------
guest: unmodified program -> guest runtime + thunk libs
--------------------------------------------------------------
| syscall(4096, op, args) (only crossing point)
v
=== inside QEMU: THIS PATCH, ~250 lines ======================
dlcall plugin: dlopen / dlsym / invoke a host fn
==============================================================
|
v
--- userspace (out of tree) ----------------------------------
host: host runtime + thunk libs -> real libz / libGL ...--------------------------------------------------------------
The split is deliberate, and it is why only this one file is proposed for the tree:
* This plugin defines the most general interaction interface for native
pass-through: the magic-syscall ABI between an emulated guest and its emulator. That contract is what every pass-through implementation builds on, so it belongs in a stable, shared place.
* It is also the only piece that is inherently QEMU-specific: it plugs into QEMU's syscall-filter hook and runs inside the QEMU process. The argument marshalling, calling conventions, callbacks/reentry and per-library coverage are not tied to any particular DBT and behave as ordinary userspace, so they should stay out of tree rather than couple QEMU to them.
Background: we presented this approach at KVM Forum 2025, "Lorelei: Enable QEMU to Leverage Native Shared Libraries":
https://www.youtube.com/watch?v=_jioQFm7wyU
The userspace side
==================
A fair point on v3 was that, on its own, the plugin is only half of an
interface: useful only if the other half (the guest/host runtimes and the per-library thunks) is public and specified, rather than a private demo. That half is now available as a standalone, documented, CI-tested project, Lorelei:
https://github.com/rover2024/lorelei
Lorelei provides the guest and host runtimes and a Thunk Library Compiler (TLC, built on Clang LibTooling) that reads a library's headers and generates the guest and host thunks automatically, including the awkward cases of function-pointer callbacks and variadic functions. It has CI for x86_64, arm64 and riscv64 hosts. Lorelei and its thunk libraries are MIT-licensed.
Adding a thunk is deliberately little work, and needs nothing but a release. Lorelei's releases ship a prebuilt toolchain (a "devkit"); after unpacking one, a single command reads a library and its headers and generates both the guest and host thunks, with no per-function code, no manifest and no build system to drive:
devkit/bin/LoreMakeThunk.py --name z --lib libz.so.1 --header zlib.h -o thunks
Two runnable examples come with Lorelei, each with a Makefile that does exactly this and runs the example under the plugin in one step:
* hello, a minimal one-function library:
https://github.com/rover2024/lorelei/tree/main/examples/hello
* demo, variadic functions and a callback that reenters the guest:
https://github.com/rover2024/lorelei/tree/main/examples/demo
If you do not have qemu or an x86_64 rootfs on hand, the project also ships a small Docker setup that builds qemu and runs the examples for you, so a single command reproduces the whole flow.
Because the plugin is not upstream yet, Lorelei currently builds and tests against the QEMU fork that carries it. Its releases ship prebuilt guest and host thunk trees for x86_64, aarch64 and riscv64 hosts, so trying it out does not require building from source:
https://github.com/rover2024/lorelei/releases
The plugin stays deliberately minimal and prescribes nothing about how thunking is done. Lorelei is one reference implementation of the userspace side. Any toolchain, or another instrumentation framework, can implement the same dlcall interface.
A from-scratch walkthrough of the bare mechanism, with the minizip and OpenGL/X11 examples above, is also available here:
https://github.com/rover2024/qemu-passthrough-test
It is fully opt-in (loaded with -plugin) and targets linux-user, where the guest and host already share a trust domain. The test cases use x86_64 guests and run on x86_64, arm64 and riscv64 Linux hosts.
Feedback on the plugin and on the pass-through approach is welcome.
Changes since v8:
Following Pierrick Bouvier's v8 request, the guest program in the docs/about/emulation.rst example no longer links with an rpath. It carries no rpath at all, so which libhello.so it loads is chosen entirely by LD_LIBRARY_PATH at run time: the baseline run puts the guest build there, and the thunk run puts the generated guest thunk there in its place. The plugin code is unchanged.
Changes since v7:
Reworked the docs/about/emulation.rst example into a complete, self-contained walkthrough, following Pierrick Bouvier's v7 review:
* Included the full source files and the exact compiler commands for the two libraries and the guest program, runnable end to end.
* The example now shows the thunk as a drop-in: the same unmodified guest program prints "from guest" with its own library, and "from host" once the generated guest thunk is placed ahead of it on LD_LIBRARY_PATH.
The plugin code is unchanged.
Changes since v6:
Addressing Pierrick Bouvier's v6 request that trying the userspace side need only the release, with no second repository to clone:
* The thunk generator now ships inside the devkit that Lorelei's releases provide, as bin/LoreMakeThunk.py. After unpacking a devkit, a single command reads a library and its headers and generates both the guest and host thunks, producing all the boilerplate itself. It takes plain command-line arguments, with no manifest or config file and no cmake/make/git behind it, so the whole flow is: download a devkit, run one command, run the guest under the plugin.
* docs/about/emulation.rst: rewrote the hello walkthrough to that flow (download a devkit, one LoreMakeThunk command, run under the plugin) and pointed the example links at the lorelei repository, where the examples now live. The plugin code is unchanged.
Changes since v5:
Following a discussion with Pierrick Bouvier on making the userspace side easy to try, this version focuses on that.
* docs/about/emulation.rst: expanded the example into a runnable hello walkthrough (the wrapped one-function library and its thunk manifest) and added links to the two example directories, hello and demo. Each example comes with a Makefile that builds the thunk and runs it under the plugin in one command, plus a small Docker setup for hosts without qemu or an x86_64 rootfs. Also reframed the section as a dynamic linking function call interception mechanism (tracing/auditing as further uses, acceleration as one use) and renamed it to "Dynamic Linking Call".
* The plugin header now names it "dlcall (Dynamic Linking Call)" and refers to Lorelei as one end-to-end userspace implementation. The plugin code is otherwise unchanged.
Changes since v4:
* Added a documentation patch for docs/about/emulation.rst (Pierrick Bouvier): what the plugin does, its trusted-guests and guest_base == 0 constraints, how to load it, and a minimal example, with a pointer to Lorelei.
* The plugin's header comment now points to Lorelei as one userspace implementation. The plugin code is otherwise unchanged.
Changes since v3:
* Lorelei, the userspace toolchain that implements this interface, is now a public, documented, CI-tested project, with a Thunk Library Compiler that generates the guest/host thunks automatically. This addresses the v3 feedback that the interface needs a public implementation behind it. The plugin code itself is unchanged.
Changes since v2:
* Dropped the RFC tag. The approach was positively received on v2.
* Rebased on master and adjusted the syscall-filter callback to its updated signature (int64_t sysret, added userdata, dropped the plugin id argument).
Changes since v1:
* Renamed the plugin from "passthrough" to "dlcall" (Pierrick Bouvier). The old name was too generic. The name "dlcall" reflects what the plugin actually does (dlopen/dlsym a host symbol and call it) and avoids confusion with QEMU's existing plugin hostcall concept (QEMU_PLUGIN_*_HOSTCALL).
* Made the magic syscall number configurable at load time via the "syscall_num=N" argument, defaulting to 4096 and rejecting values low enough to clash with a real syscall (Pierrick Bouvier).
v1: https://lore.kernel.org/qemu-devel/[email protected]/
v2: https://lore.kernel.org/qemu-devel/[email protected]/
v3: https://lore.kernel.org/qemu-devel/[email protected]/
v4: https://lore.kernel.org/qemu-devel/[email protected]/
v5: https://lore.kernel.org/qemu-devel/[email protected]/
v6: https://lore.kernel.org/qemu-devel/[email protected]/
v7: https://lore.kernel.org/qemu-devel/[email protected]/
v8: https://lore.kernel.org/qemu-devel/[email protected]/
Thanks,
Ziyang Zhang
Ziyang Zhang (2):
contrib/plugins: add a minimal dlcall plugin docs/about/emulation: document the dlcall plugin
contrib/plugins/dlcall.c | 248 ++++++++++++++++++++++++++++++++++++ contrib/plugins/meson.build | 5 + docs/about/emulation.rst | 202 ++++++++++++++++++++++++++++++++++++ 3 files changed, 455 insertions(+) create mode 100644 contrib/plugins/dlcall.c
Signature · 1 line
2.34.1
PATCH v9 1/2 contrib/plugins: add a minimal dlcall plugin
Add a minimal dlcall plugin that lets the guest invoke host functions through magic system calls. The plugin registers a vCPU syscall filter callback that intercepts a reserved syscall number and dispatches a set of pass-through operations: querying host attributes, loading and freeing shared libraries, resolving symbols, retrieving the last library error, and invoking a host function through a common interface.
The magic syscall number defaults to 4096 and can be overridden at load time with the "syscall_num=N" argument; values low enough to clash with a real syscall are rejected.
--- contrib/plugins/dlcall.c | 248 ++++++++++++++++++++++++++++++++++++ contrib/plugins/meson.build | 5 + 2 files changed, 253 insertions(+) create mode 100644 contrib/plugins/dlcall.c
contrib/plugins/dlcall.c new file +248 -0
@@ -0,0 +1,248 @@ +/* +* Copyright (C) 2026, Ziyang Zhang <[email protected]> +* +* dlcall (Dynamic Linking Call) plugin: lets a linux-user guest invoke host +* functions by issuing a magic system call. The guest can ask QEMU to dlopen() +* a host shared library, dlsym() a symbol, and call it with guest-supplied +* arguments. +* +* The plugin intentionally keeps the QEMU side lightweight and prescribes +* nothing about how a library is thunked. Any toolchain can implement the +* userspace side. Lorelei is one end-to-end implementation (guest/host +* runtimes plus a thunk compiler that generates thunks from a library's +* headers): +* https://github.com/rover2024/lorelei +* +* See docs/about/emulation.rst|Dynamic Linking Call for details and examples. +* +* WARNING: trusted guests only. The guest can load arbitrary host libraries +* and execute arbitrary host code with arbitrary arguments, i.e. full code +* execution in the QEMU host process. It is NOT a sandbox and provides no +* isolation; only load it for guests you fully trust. +* +* WARNING: requires guest_base == 0, which is qemu-user's default. Pointer +* operands are dereferenced as host addresses directly, and the invoked host +* functions dereference guest pointers with no address translation, so guest +* and host must share a single address space. A non-zero guest_base (e.g. set +* via -B/-R) would make every pointer off by guest_base and hit unrelated +* host memory. +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#include <assert.h> +#include <errno.h> +#include <string.h> +#include <stdio.h> +#include <glib.h> +#include <dlfcn.h> + +#include <qemu-plugin.h> + +QEMU_PLUGIN_EXPORT int qemu_plugin_version = QEMU_PLUGIN_VERSION; + +/* +* The magic system call number for dlcall. +* +* It defaults to DLCALL_SYSCALL_DEFAULT and can be overridden at load time +* with the "syscall_num=N" argument. To avoid hijacking a real syscall the +* guest might issue, N must be at least DLCALL_SYSCALL_MIN: every Linux ABI +* keeps its syscall numbers well below this; numbers from here up are free. +*/ +enum { +DLCALL_SYSCALL_DEFAULT =4096 , +DLCALL_SYSCALL_MIN =4096 , +}; + +static int64_t dlcall_syscall_num =DLCALL_SYSCALL_DEFAULT ; + +/* +* dlcall calling convention. +* +* The guest issues the magic system call (dlcall_syscall_num). The first +* argument (a1) is one of the call IDs below; the remaining arguments (a2, a3, +* a4, ...) are that ID's operands. All pointer operands are guest virtual +* addresses that the plugin dereferences as host addresses directly (see the +* guest_base requirement above). Results are written back through +* caller-provided "out" pointers rather than returned in the syscall value. +* +* The syscall return value (*sysret) only reports dispatch status: 0 on a +* recognised ID, -EINVAL for an unknown one. The actual success/failure of an +* operation (e.g. a NULL handle from dlopen) is delivered through its out +* pointer, exactly like the underlying libdl call. +* +* Operands per ID: +* +* DLCALL_ID_GET_HOST_ATTRIBUTE +* a2 const char *key in: attribute name to query +* a3 const char **attr_ptr out: matching value, or NULL if unknown +* +* DLCALL_ID_LOAD_LIBRARY (wraps dlopen) +* a2 const char *path in: library path +* a3 int flags in: dlopen() flags (e.g. RTLD_NOW) +* a4 void **handle_ptr out: library handle, or NULL on failure +* +* DLCALL_ID_GET_PROC_ADDRESS (wraps dlsym) +* a2 void *handle in: library handle +* a3 const char *name in: symbol name +* a4 void **entry_ptr out: symbol address, or NULL if not found +* +* DLCALL_ID_FREE_LIBRARY (wraps dlclose) +* a2 void *handle in: library handle +* a3 int *ret_ptr out: dlclose() return value (0 on success) +* +* DLCALL_ID_GET_LIBRARY_ERROR (wraps dlerror) +* a2 const char **error_ptr out: last libdl error string, or NULL +* +* DLCALL_ID_INVOKE_PROC (calls the symbol) +* a2 void *proc in: function pointer, signature +* void (*)(void *arg1, void *arg2) +* a3 void *arg1 in: first argument forwarded to proc +* a4 void *arg2 in: second argument forwarded to proc +*/ +enum DlcallID { +DLCALL_ID_GET_HOST_ATTRIBUTE , +DLCALL_ID_LOAD_LIBRARY , +DLCALL_ID_GET_PROC_ADDRESS , +DLCALL_ID_FREE_LIBRARY , +DLCALL_ID_GET_LIBRARY_ERROR , +DLCALL_ID_INVOKE_PROC , +}; + +static inline const char *query_host_attribute (const char *key) +{ +if (strcmp (key,"emu" ) ==0 ) { +return "qemu" ; + } +return NULL ; +} + +static inline void invoke_proc (void *proc,void *arg1,void *arg2) +{ +typedef void (*Func )(void */*arg1*/ ,void */*arg2*/ ); +Func func = (Func ) proc; +func (arg1, arg2); +} + +static bool vcpu_syscall_filter (unsigned int vcpu_index, +int64_t num,uint64_t a1,uint64_t a2, +uint64_t a3,uint64_t a4,uint64_t a5, +uint64_t a6,uint64_t a7,uint64_t a8, +int64_t *sysret,void *userdata) +{ +if (num == dlcall_syscall_num) { +switch (a1) { +/* Query host attribute by a reserved key. */ +case DLCALL_ID_GET_HOST_ATTRIBUTE : { +const char *key = (const char *) a2; +const char **attr_ptr = (const char **) a3; +assert (attr_ptr); + *attr_ptr =query_host_attribute (key); + *sysret =0 ; +break ; + } + +/* Load a shared library. */ +case DLCALL_ID_LOAD_LIBRARY : { +const char *path = (const char *) a2; +int flags = (int ) a3; +void **handle_ptr = (void **) a4; +assert (handle_ptr); + *handle_ptr =dlopen (path, flags); + *sysret =0 ; +break ; + } + +/* Get the address of a function in a shared library. */ +case DLCALL_ID_GET_PROC_ADDRESS : { +void *handle = (void *) a2; +const char *name = (const char *) a3; +void **entry_ptr = (void **) a4; +assert (entry_ptr); + *entry_ptr =dlsym (handle, name); + *sysret =0 ; +break ; + } + +/* Free a shared library. */ +case DLCALL_ID_FREE_LIBRARY : { +void *handle = (void *) a2; +int *ret_ptr = (int *) a3; + *ret_ptr =dlclose (handle); + *sysret =0 ; +break ; + } + +/* Get the last error message for a library event. */ +case DLCALL_ID_GET_LIBRARY_ERROR : { +const char **error_ptr = (const char **) a2; + *error_ptr =dlerror (); + *sysret =0 ; +break ; + } + +/* Invoke a function of a common interface. */ +case DLCALL_ID_INVOKE_PROC : { +void *proc = (void *) a2; +void *arg1 = (void *) a3; +void *arg2 = (void *) a4; +assert (proc); +invoke_proc (proc, arg1, arg2); + *sysret =0 ; +break ; + } + +default : + *sysret = -EINVAL ; +break ; + } +return true; + } +return false; +} + +QEMU_PLUGIN_EXPORT int qemu_plugin_install(qemu_plugin_id_t id, +const qemu_info_t *info, +int argc,char **argv) +{ +if (info->system_emulation) { +fprintf (stderr,"plugin dlcall: only useful for user emulation\n" ); +return -1 ; + } + +for (int i =0 ; i < argc; i++) { +char *opt = argv[i]; +g_auto (GStrv) tokens =g_strsplit (opt,"=" ,2 ); +if (g_strcmp0 (tokens[0 ],"syscall_num" ) ==0 ) { +const char *val = tokens[1 ]; +char *endptr =NULL ; +guint64 num; +if (!val || *val =='\0' ) { +fprintf (stderr, +"plugin dlcall: missing value for syscall_num\n" ); +return -1 ; + } + num =g_ascii_strtoull (val, &endptr,0 ); +if (*endptr !='\0' ||g_strrstr (val,"-" ) !=NULL ) { +fprintf (stderr, +"plugin dlcall: invalid syscall_num '%s'\n" , val); +return -1 ; + } +if (num <DLCALL_SYSCALL_MIN || num >G_MAXINT64 ) { +fprintf (stderr, +"plugin dlcall: syscall_num %s is out of range; " +"it must be >= %d to avoid clashing with a real " +"syscall\n" , val,DLCALL_SYSCALL_MIN ); +return -1 ; + } + dlcall_syscall_num = (int64_t ) num; + }else { +fprintf (stderr,"plugin dlcall: unknown option '%s'\n" , opt); +return -1 ; + } + } + +qemu_plugin_register_vcpu_syscall_filter_cb (id, vcpu_syscall_filter,NULL ); + +return 0 ; +}
contrib/plugins/meson.build +5 -0
@@ -19,6 +19,11 @@ if host_os != 'windows' contrib_plugins += 'lockstep.c' endif +if host_os == 'linux' + # dlcall passes guest calls through to host libraries; linux-user only + contrib_plugins += 'dlcall.c' +endif + if 'cpp' in all_languages contrib_plugins += 'cpp.cpp' endif
Signature · 1 line
2.34.1
PATCH v9 2/2 docs/about/emulation: document the dlcall plugin
In reply to Ziyang ZhangDocument the dlcall plugin under Example Plugins: what it does, the trusted- guests and guest_base == 0 constraints, how to load it, and a pointer to Lorelei, one end-to-end userspace implementation, for the toolchain and a runnable example.
--- docs/about/emulation.rst | 202 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 202 insertions(+)
docs/about/emulation.rst +202 -0
@@ -1046,6 +1046,208 @@ Count traps This plugin counts the number of interrupts (asynchronous events), exceptions (synchronous events) and host calls (e.g. semihosting) per cpu. +Dynamic Linking Call +.................... + +``contrib/plugins/dlcall.c`` + +This plugin provides a dynamic linking function call interception mechanism +for linux-user guests: the guest hands a call off to the host, where the plugin +runs native code in its place instead of the guest emulating it. Interception +alone enables several uses, for instance tracing or auditing guest calls. +One use is acceleration by leveraging the host's native shared libraries. For +example, a thunk layer can run the stock zlib ``minizip`` utility under +emulation while forwarding its ``deflate`` calls to the host's native zlib +library (libz). This avoids emulating those selected library calls instruction +by instruction. + +The guest issues a reserved "magic" system call (4096 by default, configurable +with ``syscall_num=N``) whose first argument selects a pass-through operation: +dlopen/dlclose a host library, dlsym a symbol, and invoke a resolved host +function. The plugin performs the operation on the host and consumes the +syscall, so the real kernel never sees it. + +.. warning:: + + Trusted guests only. The guest can load arbitrary host libraries and run + arbitrary code in the QEMU host process. The plugin is not a sandbox and + provides no isolation. It also requires ``guest_base == 0`` (qemu-user's + default), as guest pointers are dereferenced as host addresses with no + translation. + +The plugin intentionally keeps the QEMU side lightweight and knows nothing +about any particular library or its calling convention. Turning a real library +into working thunks, including argument marshalling, callbacks and variadic +functions, is done entirely in userspace, and any toolchain can implement the +interface. + +Loading the plugin is all that is required from QEMU's side: + +.. code-block:: shell + + qemu-x86_64 -plugin contrib/plugins/libdlcall.so <guest-program> ... + +`Lorelei <https://github.com/rover2024/lorelei>`_ is one end-to-end userspace +implementation of this: it provides the guest and host runtimes and an +automated toolchain that generates the thunks from a library's headers, so guest +library calls run on the host's native libraries. It supports an x86_64 guest +running on an x86_64, aarch64 or riscv64 host. + +A minimal end-to-end example uses a one-function library, ``libhello.so``, built +two ways: the guest build tags its output ``from guest`` and the host build tags +it ``from host``. An unmodified guest program ``main`` calls ``hello("world", +7)``, and the thunk makes that same binary reach the host build in place of its +own. The sources live under ``src/``: + +.. code-block:: c + + /* src/hello.h */ + #ifdef __cplusplus + extern "C" { + #endif + void hello(const char *name, int lucky); + #ifdef __cplusplus + } + #endif + +.. code-block:: c + + /* src/hello_guest.c */ + #include "hello.h" + #include <stdio.h> + + void hello(const char *name, int lucky) + { + printf("hello from guest: %s, lucky %d\n", name, lucky); + fflush(stdout); + } + +.. code-block:: c + + /* src/hello_host.c */ + #include "hello.h" + #include <stdio.h> + + void hello(const char *name, int lucky) + { + printf("hello from host: %s, lucky %d\n", name, lucky); + fflush(stdout); + } + +.. code-block:: c + + /* src/main.c */ + #include "hello.h" + + int main(void) + { + hello("world", 7); + return 0; + } + +Lorelei ships a prebuilt toolchain (a "devkit") in its releases. Download the +one for your host and unpack it: + +.. code-block:: shell + + # <arch> is your host architecture: x86_64, aarch64 or riscv64 + wget https://github.com/rover2024/lorelei/releases/download/v<version>/lorelei-devkit-<arch>-<version>.tar.xz + tar -xf lorelei-devkit-<arch>-<version>.tar.xz + DEVKIT=lorelei-devkit-<arch> + +Build the guest ``libhello.so`` (x86_64) and the host ``libhello.so`` (this +host's own architecture), then the guest program: + +.. code-block:: shell + + mkdir -p build/guest build/host + $DEVKIT/bin/x86_64-linux-gnu-clang -shared -fPIC src/hello_guest.c -o build/guest/libhello.so + cc -shared -fPIC src/hello_host.c -o build/host/libhello.so + $DEVKIT/bin/x86_64-linux-gnu-clang src/main.c -Isrc -Lbuild/guest -lhello -o build/guest/main + +Run it under qemu: + +.. code-block:: shell + + qemu-x86_64 -E LD_LIBRARY_PATH=build/guest build/guest/main + +which prints:: + + hello from guest: world, lucky 7 + +Now generate the thunk from the host ``libhello.so``. This produces a guest-side +``libhello.so`` that stands in for the guest build, and a host-side thunk library +that dispatches to the host build: + +.. code-block:: shell + + $DEVKIT/bin/LoreMakeThunk.py --name hello --lib build/host/libhello.so \ + --header hello.h -o thunks -- -Isrc + +Run the same ``main`` under the plugin, with the generated guest ``libhello.so`` +on ``LD_LIBRARY_PATH`` in place of its own. The call reaches the host build now, +from the same unmodified binary: + +.. code-block:: shell + + LORELEI_THUNK_PATH=thunks \ + LD_LIBRARY_PATH=$DEVKIT/lib:build/host \ + qemu-x86_64 -plugin contrib/plugins/libdlcall.so \ + -E LD_LIBRARY_PATH=thunks/x86_64/lib/x86_64-LoreGTL:$DEVKIT/x86_64/lib \ + build/guest/main + +which prints:: + + hello from host: world, lucky 7 + +The guest ``LD_LIBRARY_PATH``, passed with ``-E``, is where the emulated program +looks for its libraries. Because ``main`` has no rpath, whatever ``libhello.so`` +this path resolves is the one it loads. The thunk run lists the generated thunk +here, not the guest build, so the thunk stands in for it: + +* ``thunks/x86_64/lib/x86_64-LoreGTL`` holds the generated guest ``libhello.so``. +* ``$DEVKIT/x86_64/lib`` holds the guest runtime support shipped with the devkit. + +The host ``LD_LIBRARY_PATH`` is QEMU's own search path: + +* ``$DEVKIT/lib`` holds the host runtime support shipped with the devkit. +* ``build/host`` holds the host ``libhello.so`` the host thunk dispatches to. + +The finished tree:: + + src/ + hello.h + hello_guest.c + hello_host.c + main.c + build/ + guest/ + libhello.so (guest build of the library) + main (the guest program) + host/ + libhello.so (host build of the library) + thunks/ + lib/x86_64-LoreHTL/libhello_HTL.so (host thunk) + x86_64/lib/x86_64-LoreGTL/libhello.so (guest thunk, the drop-in) + +See the runnable +`hello <https://github.com/rover2024/lorelei/tree/main/examples/hello>`_ +(minimal) and +`demo <https://github.com/rover2024/lorelei/tree/main/examples/demo>`_ +(variadic functions and a callback that reenters the guest) examples, and +`Lorelei <https://github.com/rover2024/lorelei>`_ for the toolchain and the +runtime environment they expect. + +.. list-table:: Dynamic Linking Call arguments + :widths: 20 80 + :header-rows: 1 + + * - Option + - Description + * - syscall_num=N + - The magic syscall number the guest issues (default 4096). Must be high + enough not to clash with a real syscall. + Other emulation features ------------------------
Signature · 1 line
2.34.1
Re: [PATCH v9 2/2] docs/about/emulation: document the dlcall plugin
Thanks for the update.
Quoted 74 lines · Ziyang Zhang
Document the dlcall plugin under Example Plugins: what it does, the trusted- guests and
guest_base== 0 constraints, how to load it, and a pointer to Lorelei, one end-to-end userspace implementation, for the toolchain and a runnable example.Co-authored-by Kailiang Xu <[email protected]>Co-authored-by Mingyuan Xia <[email protected]>Signed-off-by Ziyang Zhang <[email protected]>--- docs/about/emulation.rst | 202 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 202 insertions(+)docs/about/emulation.rst +54
-0@@ -1046,6 +1046,208 @@ Count traps This plugin counts the number of interrupts (asynchronous events), exceptions (synchronous events) and host calls (e.g. semihosting) per cpu. +Dynamic Linking Call +.................... + +``contrib/plugins/dlcall.c`` + +This plugin provides a dynamic linking function call interception mechanism +for linux-user guests: the guest hands a call off to the host, where the plugin +runs native code in its place instead of the guest emulating it. Interception +alone enables several uses, for instance tracing or auditing guest calls. +One use is acceleration by leveraging the host's native shared libraries. For +example, a thunk layer can run the stock zlib ``minizip`` utility under +emulation while forwarding its ``deflate`` calls to the host's native zlib +library (libz). This avoids emulating those selected library calls instruction +by instruction. + +The guest issues a reserved "magic" system call (4096 by default, configurable +with ``syscall_num=N``) whose first argument selects a pass-through operation: +dlopen/dlclose a host library, dlsym a symbol, and invoke a resolved host +function. The plugin performs the operation on the host and consumes the +syscall, so the real kernel never sees it. + +.. warning:: + + Trusted guests only. The guest can load arbitrary host libraries and run + arbitrary code in the QEMU host process. The plugin is not a sandbox and + provides no isolation. It also requires ``guest_base == 0`` (qemu-user's + default), as guest pointers are dereferenced as host addresses with no + translation. + +The plugin intentionally keeps the QEMU side lightweight and knows nothing +about any particular library or its calling convention. Turning a real library +into working thunks, including argument marshalling, callbacks and variadic +functions, is done entirely in userspace, and any toolchain can implement the +interface. + +Loading the plugin is all that is required from QEMU's side: + +.. code-block:: shell + + qemu-x86_64 -plugin contrib/plugins/libdlcall.so <guest-program> ... + +`Lorelei <https://github.com/rover2024/lorelei>`_ is one end-to-end userspace +implementation of this: it provides the guest and host runtimes and an +automated toolchain that generates the thunks from a library's headers, so guest +library calls run on the host's native libraries. It supports an x86_64 guest +running on an x86_64, aarch64 or riscv64 host. + +A minimal end-to-end example uses a one-function library, ``libhello.so``, built +two ways: the guest build tags its output ``from guest`` and the host build tags +it ``from host``. An unmodified guest program ``main`` calls ``hello("world", +7)``, and the thunk makes that same binary reach the host build in place of its +own. The sources live under ``src/``: + +.. code-block:: c
docs/about/emulation.rst Count traps · line 1112
Quoted 10 lines
docs/about/emulation.rst +10
-0+ + /* src/hello.h */ + #ifdef __cplusplus + extern "C" { + #endif + void hello(const char *name, int lucky); + #ifdef __cplusplus + } + #endif +
Please remove the cplusplus part, we don't need it here.
docs/about/emulation.rst Count traps · line 1122
Quoted 10 lines
docs/about/emulation.rst +10
-0+.. code-block:: c + + /* src/hello_guest.c */ + #include "hello.h" + #include <stdio.h> + + void hello(const char *name, int lucky) + { + printf("hello from guest: %s, lucky %d\n", name, lucky); + fflush(stdout);
No need for fflush (same in hello_host)
Quoted 4 lines
docs/about/emulation.rst +4
-0+ } + +.. code-block:: c +
docs/about/emulation.rst Count traps · line 1136
Quoted 10 lines
docs/about/emulation.rst +10
-0+ /* src/hello_host.c */ + #include "hello.h" + #include <stdio.h> + + void hello(const char *name, int lucky) + { + printf("hello from host: %s, lucky %d\n", name, lucky); + fflush(stdout); + } +
Excellent, that's a nice way to see difference between guest and host.
Quoted 10 lines
docs/about/emulation.rst +10
-0+.. code-block:: c + + /* src/main.c */ + #include "hello.h" + + int main(void) + { + hello("world", 7); + return 0; + }
docs/about/emulation.rst Count traps · line 1156
Quoted 10 lines
docs/about/emulation.rst +10
-0+ +Lorelei ships a prebuilt toolchain (a "devkit") in its releases. Download the +one for your host and unpack it: + +.. code-block:: shell + + # <arch> is your host architecture: x86_64, aarch64 or riscv64 + wget https://github.com/rover2024/lorelei/releases/download/v<version>/lorelei-devkit-<arch>-<version>.tar.xz + tar -xf lorelei-devkit-<arch>-<version>.tar.xz + DEVKIT=lorelei-devkit-<arch>
Please replace this part with a proper url, choosing a specific version and arch. Ideally, you can publish under "latest" in addition to versioned releases on GitHub so url stays stable.
Also, please add a link to https://github.com/rover2024/lorelei/releases
in the comment so people can click this and see what is available.
docs/about/emulation.rst Count traps · line 1167
Quoted 11 lines
docs/about/emulation.rst +11
-0+ +Build the guest ``libhello.so`` (x86_64) and the host ``libhello.so`` (this +host's own architecture), then the guest program: + +.. code-block:: shell + + mkdir -p build/guest build/host + $DEVKIT/bin/x86_64-linux-gnu-clang -shared -fPIC src/hello_guest.c -o build/guest/libhello.so + cc -shared -fPIC src/hello_host.c -o build/host/libhello.so + $DEVKIT/bin/x86_64-linux-gnu-clang src/main.c -Isrc -Lbuild/guest -lhello -o build/guest/main +
Good.
Quoted 16 lines
docs/about/emulation.rst +16
-0+Run it under qemu: + +.. code-block:: shell + + qemu-x86_64 -E LD_LIBRARY_PATH=build/guest build/guest/main + +which prints:: + + hello from guest: world, lucky 7 + +Now generate the thunk from the host ``libhello.so``. This produces a guest-side +``libhello.so`` that stands in for the guest build, and a host-side thunk library +that dispatches to the host build: + +.. code-block:: shell +
docs/about/emulation.rst Count traps · line 1193
Quoted 10 lines
docs/about/emulation.rst +10
-0+ $DEVKIT/bin/LoreMakeThunk.py --name hello --lib build/host/libhello.so \ + --header hello.h -o thunks -- -Isrc + +Run the same ``main`` under the plugin, with the generated guest ``libhello.so`` +on ``LD_LIBRARY_PATH`` in place of its own. The call reaches the host build now, +from the same unmodified binary: + +.. code-block:: shell + + LORELEI_THUNK_PATH=thunks \
Why do we need this variable?
docs/about/emulation.rst Count traps · line 1198
Quoted 5 lines
docs/about/emulation.rst +5
-0+ LD_LIBRARY_PATH=$DEVKIT/lib:build/host \ + qemu-x86_64 -plugin contrib/plugins/libdlcall.so \ + -E LD_LIBRARY_PATH=thunks/x86_64/lib/x86_64-LoreGTL:$DEVKIT/x86_64/lib \ + build/guest/main +
This still fails on my side with following error message:
lorelei.guest-rt: failed to load host runtime: libLoreHostRT.so: cannot enable executable stack as shared object requires: Invalid argument
qemu: uncaught target signal 6 (Aborted) - core dumped Aborted
Seems like error message is incorrect in itself "invalid argument"?
I'm using debian trixie (stable) in case you want to reproduce this on a VM.
----
Please simplify thunks/ folder hierarchy. Make only one subfolder (thunks/x86_64) having both the thunks and lorelei lib, and remove the rest.
With this, we can simplify command to become:
LD_LIBRARY_PATH=$DEVKIT/lib/:build/host \
qemu-x86_64 -plugin contrib/plugins/libdlcall.so \
-E LD_LIBRARY_PATH=thunks/x86_64 build/guest/main
---
Also, in lorelei tarball, why do we have /lib and x86_64/lib that have same libraries, but with different sizes? What's the difference? It's confusing.
It seems like x86_64/lib folder is never used?
docs/about/emulation.rst Count traps · line 1202
Quoted 4 lines
docs/about/emulation.rst +4
-0+which prints:: + + hello from host: world, lucky 7 +
[
Quoted 28 lines
docs/about/emulation.rst +28
-0+The guest ``LD_LIBRARY_PATH``, passed with ``-E``, is where the emulated program +looks for its libraries. Because ``main`` has no rpath, whatever ``libhello.so`` +this path resolves is the one it loads. The thunk run lists the generated thunk +here, not the guest build, so the thunk stands in for it: + +* ``thunks/x86_64/lib/x86_64-LoreGTL`` holds the generated guest ``libhello.so``. +* ``$DEVKIT/x86_64/lib`` holds the guest runtime support shipped with the devkit. + +The host ``LD_LIBRARY_PATH`` is QEMU's own search path: + +* ``$DEVKIT/lib`` holds the host runtime support shipped with the devkit. +* ``build/host`` holds the host ``libhello.so`` the host thunk dispatches to. + +The finished tree:: + + src/ + hello.h + hello_guest.c + hello_host.c + main.c + build/ + guest/ + libhello.so (guest build of the library) + main (the guest program) + host/ + libhello.so (host build of the library) + thunks/ + lib/x86_64-LoreHTL/libhello_HTL.so (host thunk)
docs/about/emulation.rst Count traps · line 1240
Quoted 10 lines
docs/about/emulation.rst +10
-0+ x86_64/lib/x86_64-LoreGTL/libhello.so (guest thunk, the drop-in) + +See the runnable +`hello <https://github.com/rover2024/lorelei/tree/main/examples/hello>`_ +(minimal) and +`demo <https://github.com/rover2024/lorelei/tree/main/examples/demo>`_ +(variadic functions and a callback that reenters the guest) examples, and +`Lorelei <https://github.com/rover2024/lorelei>`_ for the toolchain and the +runtime environment they expect. +
]
Please remove the section above from the doc since it does not add useful information. Also, we already have a link to Lorelei above.
docs/about/emulation.rst Count traps · line 1253
Quoted 13 lines
docs/about/emulation.rst +10
-0+.. list-table:: Dynamic Linking Call arguments + :widths: 20 80 + :header-rows: 1 + + * - Option + - Description + * - syscall_num=N + - The magic syscall number the guest issues (default 4096). Must be high + enough not to clash with a real syscall. + Other emulation features ------------------------
We're getting closer, hang on.
We need to have a simple example that is functional before being able to accept the plugin.
Regards,
Pierrick
Re: [PATCH v9 2/2] docs/about/emulation: document the dlcall plugin
Thanks for the detailed review. Replies inline.
Quoted 1 line · Pierrick Bouvier
Please remove the cplusplus part, we don't need it here.
Will do, hello.h will be plain C. The guard was there because the thunk generator parses the header as C++, so without C linkage the generated guest library exports a mangled name and the drop-in fails to resolve hello(). I will fix that in the generator instead: it will wrap the header in extern "C" by default (treating it as a C library), with an opt-out for genuine C++ headers, so the example header stays clean C.
Quoted 1 line
No need for fflush (same in
hello_host)
I will remove both. One caveat on the host side though: dropping its fflush is not really recommended. "hello from host" runs in the host process, so it writes to the host libc's stdout buffer, and QEMU tears the process down on guest exit without running the host libc atexit handlers that would flush it. On a terminal it prints anyway (line-buffered, the newline flushes), but once stdout is redirected or piped it is fully buffered and the line is lost. So with both removed the example only shows the host line on a terminal.
Quoted 1 line
Excellent, that's a nice way to see difference between guest and host.
Thanks.
Quoted 7 lines
Please replace this part with a proper url, choosing a specific version and arch. Ideally, you can publish under "latest" in addition to versioned releases on GitHub so url stays stable.
Also, please add a link to
https://github.com/rover2024/lorelei/releases
in the comment so people can click this and see what is available.
Will do. I will pin the command to x86_64 and a specific release, add a comment linking to the releases page, and publish a stable "latest" download asset so the URL does not rot.
Quoted 1 line
Why do we need this variable?
It is the thunk-pack discovery path, and it does a different job from LD_LIBRARY_PATH. LD_LIBRARY_PATH is the loader's search path for shared objects. LORELEI_THUNK_PATH tells the host runtime where the thunk packs live, so it can find the host side of each thunk (the HTL) and the guest-to-host library mapping, and it can layer several packs. The runtime resolves these itself, they are not something the loader would find, so I would keep this variable rather than fold it into LD_LIBRARY_PATH. Flattening the layout (your next point) shortens the paths but does not remove the need for it.
Quoted 10 lines
This still fails on my side with following error message:
lorelei.guest-rt: failed to load host runtime: libLoreHostRT.so: cannot enable executable stack as shared object requires: Invalid argument
qemu: uncaught target signal 6 (Aborted) - core dumped AbortedSeems like error message is incorrect in itself "invalid argument"?
I'm using debian trixie (stable) in case you want to reproduce this on a VM.
Thank you, this is the real bug, and the new error string is exactly what let me find it. libLoreHostRT.so was linked with an executable
stack: one assembly file in the runtime (the coroutine context switch) was missing the ".note.GNU-stack" marker, so the linker marked the whole library's stack executable. A hardened loader (Debian trixie) then refuses to dlopen it.
The message is not wrong. "Invalid argument" is the kernel's errno for the refused stack mprotect, surfaced through dlerror. It read as a mystery only because older releases did not print the dlerror string at all.
The fix is to mark the runtime's stack non-executable (add the missing note, plus -z noexecstack at link time). I have confirmed the GNU_STACK flag flips from RWE to RW with that change. A new devkit and runtime release will carry it, and the example will load on trixie. I will follow up here once it is published so you can retry.
Quoted 8 lines
Please simplify thunks/ folder hierarchy. Make only one subfolder (thunks/x86_64) having both the thunks and lorelei lib, and remove the rest.
With this, we can simplify command to become:
LD_LIBRARY_PATH=$DEVKIT/lib/:build/host \
qemu-x86_64 -plugin contrib/plugins/libdlcall.so \
-E LD_LIBRARY_PATH=thunks/x86_64 build/guest/main
I would rather keep the current layout, and I think it is justified. Those subdirectories are not cosmetic: they are the contract the host runtime uses to discover thunks. It scans each LORELEI_THUNK_PATH prefix for <prefix>/x86_64/lib/x86_64-LoreGTL (guest thunks) and <prefix>/lib/<host-arch>-LoreHTL (host thunks), which is how it tells a guest thunk from a host thunk, and by architecture. The same guest-thunk directory is also what the guest loader reads through -E, so collapsing the -E path collapses the discovery path with it.
Merging the "lorelei lib" into that folder has a second cost: those are the runtime libraries, shipped once in the devkit, not per thunk. Thunk packs are deliberately standalone and do not each carry a copy of the runtime. Putting them together would either duplicate the runtime into every pack, or mix runtime libraries in with the thunks so the scan can no longer separate the two.
So the second -E entry is not redundant: one directory is this library's thunk, the other is the shared guest runtime. The extra path components are the price of that separation, and the existing thunk packs already follow this layout, so I would keep it. If the two directories read as redundant, I can add a one-line note saying which is which, but I would rather not collapse them.
Quoted 5 lines
Also, in lorelei tarball, why do we have /lib and x86_64/lib that have same libraries, but with different sizes? What's the difference? It's confusing.
It seems like x86_64/lib folder is never used?
They are two builds of the runtime. /lib is the host runtime, native to the host, loaded by the QEMU process (the plugin dlopens libLoreHostRT.so from there). x86_64/lib is the guest runtime, x86_64, loaded by the emulated guest (the generated guest thunk needs libLoreGuestRT.so from there). Same names, different targets, hence the different sizes. x86_64/lib is used, by the guest side. I agree the split is confusing and will make the layout and the docs clearer.
Quoted 2 lines
Please remove the section above from the doc since it does not add useful information. Also, we already have a link to Lorelei above.
Will remove it.
Quoted 3 lines
We're getting closer, hang on.
We need to have a simple example that is functional before being able to accept the plugin.
Understood, and the executable-stack fix above is what makes it functional. I will publish a new devkit and send a trimmed revision that addresses all of the above.
Thanks,
Ziyang Zhang
Re: [PATCH v9 2/2] docs/about/emulation: document the dlcall plugin
Quoted 12 lines · Ziyang Zhang
Thanks for the detailed review. Replies inline.
Quoted 1 line · Pierrick Bouvier
Please remove the cplusplus part, we don't need it here.
Will do,
hello.hwill be plain C. The guard was there because the thunk generator parses the header as C++, so without C linkage the generated guest library exports a mangled name and the drop-in fails to resolvehello(). I will fix that in the generator instead: it will wrap the header in extern "C" by default (treating it as a C library), with an opt-out for genuine C++ headers, so the example header stays clean C.
Yes, we can have a parameter --lang c|c++ for makethunks script to solve this.
Quoted 11 lines
Quoted 1 line
No need for fflush (same in
hello_host)I will remove both. One caveat on the host side though: dropping its fflush is not really recommended. "hello from host" runs in the host process, so it writes to the host libc's stdout buffer, and QEMU tears the process down on guest exit without running the host libc atexit handlers that would flush it. On a terminal it prints anyway (line-buffered, the newline flushes), but once stdout is redirected or piped it is fully buffered and the line is lost. So with both removed the example only shows the host line on a terminal.
Hum, I'm not sure to see why. QEMU or any native program uses libc and things are correctly flushed without needing an explicit fflush. What is special in Lorelei case?
Quoted 18 lines
Quoted 1 line
Excellent, that's a nice way to see difference between guest and host.
Thanks.
Quoted 7 lines
Please replace this part with a proper url, choosing a specific version and arch. Ideally, you can publish under "latest" in addition to versioned releases on GitHub so url stays stable.
Also, please add a link to
https://github.com/rover2024/lorelei/releases
in the comment so people can click this and see what is available.Will do. I will pin the command to
x86_64and a specific release, add a comment linking to the releases page, and publish a stable "latest" download asset so the URL does not rot.Quoted 1 line
Why do we need this variable?
Quoted 10 lines
It is the thunk-pack discovery path, and it does a different job from
LD_LIBRARY_PATH.LD_LIBRARY_PATHis the loader's search path for shared objects.LORELEI_THUNK_PATHtells the host runtime where the thunk packs live, so it can find the host side of each thunk (the HTL) and the guest-to-host library mapping, and it can layer several packs. The runtime resolves these itself, they are not something the loader would find, so I would keep this variable rather than fold it intoLD_LIBRARY_PATH. Flattening the layout (your next point) shortens the paths but does not remove the need for it.
Which .so exactly is covered by this?
Maybe it should be a parameter of plugin itself, that passes this down to Lorelei?
Quoted 23 lines
Quoted 10 lines
This still fails on my side with following error message:
lorelei.guest-rt: failed to load host runtime: libLoreHostRT.so: cannot enable executable stack as shared object requires: Invalid argument
qemu: uncaught target signal 6 (Aborted) - core dumped AbortedSeems like error message is incorrect in itself "invalid argument"?
I'm using debian trixie (stable) in case you want to reproduce this on a VM.
Thank you, this is the real bug, and the new error string is exactly what let me find it. libLoreHostRT.so was linked with an executable
stack: one assembly file in the runtime (the coroutine context switch) was missing the ".note.GNU-stack" marker, so the linker marked the whole library's stack executable. A hardened loader (Debian trixie) then refuses to dlopen it.The message is not wrong. "Invalid argument" is the kernel's errno for the refused stack mprotect, surfaced through dlerror. It read as a mystery only because older releases did not print the dlerror string at all.
Quoted 6 lines
The fix is to mark the runtime's stack non-executable (add the missing note, plus -z noexecstack at link time). I have confirmed the
GNU_STACKflag flips from RWE to RW with that change. A new devkit and runtime release will carry it, and the example will load on trixie. I will follow up here once it is published so you can retry.
👍
Quoted 9 lines
Quoted 8 lines
Please simplify thunks/ folder hierarchy. Make only one subfolder (thunks/x86_64) having both the thunks and lorelei lib, and remove the rest.
With this, we can simplify command to become:
LD_LIBRARY_PATH=$DEVKIT/lib/:build/host \
qemu-x86_64 -plugin contrib/plugins/libdlcall.so \
-E LD_LIBRARY_PATH=thunks/x86_64 build/guest/main
Quoted 8 lines
I would rather keep the current layout, and I think it is justified. Those subdirectories are not cosmetic: they are the contract the host runtime uses to discover thunks. It scans each
LORELEI_THUNK_PATHprefix for <prefix>/x86_64/lib/x86_64-LoreGTL (guest thunks) and <prefix>/lib/<host-arch>-LoreHTL (host thunks), which is how it tells a guest thunk from a host thunk, and by architecture. The same guest-thunk directory is also what the guest loader reads through -E, so collapsing the -E path collapses the discovery path with it.
The same thing can be achieved by looking into current LD_LIBRARY_PATH, without needing another env var for this. No guest binary will link to lorelei shared objects, so there is no conflict to be expected.
If any library name clashes between guest/host side, just rename things directly.
I would like to insist to see LORELEI_THUNK_PATH gone and that we rely on LD_LIBRARY_PATH only.
This way, it's clear what is consumed on host side, and what is consumed on guest side.
Having a magic variable forces to read Lorelei sources to understand what is going on, and it's not needed.
Quoted 20 lines
Merging the "lorelei lib" into that folder has a second cost: those are the runtime libraries, shipped once in the devkit, not per thunk. Thunk packs are deliberately standalone and do not each carry a copy of the runtime. Putting them together would either duplicate the runtime into every pack, or mix runtime libraries in with the thunks so the scan can no longer separate the two.
Quoted 1 line
So the second -E entry is not redundant: one directory is this library's
thunk, the other is the shared guest runtime. The extra path components are the price of that separation, and the existing thunk packs already follow this layout, so I would keep it. If the two directories read as redundant, I can add a one-line note saying which is which, but I would rather not collapse them.
Quoted 5 lines
Also, in lorelei tarball, why do we have /lib and x86_64/lib that have same libraries, but with different sizes? What's the difference? It's confusing.
It seems like x86_64/lib folder is never used?
Quoted 8 lines
They are two builds of the runtime. /lib is the host runtime, native to the host, loaded by the QEMU process (the plugin dlopens libLoreHostRT.so from there). x86_64/lib is the guest runtime,
x86_64, loaded by the emulated guest (the generated guest thunk needs libLoreGuestRT.so from there). Same names, different targets, hence the different sizes. x86_64/lib is used, by the guest side. I agree the split is confusing and will make the layout and the docs clearer.
libLoreHostRT.so is in both, so I'm confused about which is which. Thanks for clearing that also.
Quoted 5 lines
Quoted 2 lines
Please remove the section above from the doc since it does not add useful information. Also, we already have a link to Lorelei above.
Will remove it.
Quoted 8 lines
Quoted 3 lines
We're getting closer, hang on.
We need to have a simple example that is functional before being able to accept the plugin.Understood, and the executable-stack fix above is what makes it functional. I will publish a new devkit and send a trimmed revision that addresses all of the above.
We'll be close from the final version with this. Only thing left will be to clean the layouts, disambiguate files, and eliminate LORELEI_THUNK_PATH to rely on LD_LIBRARY_PATH only.
Quoted 2 lines
Thanks,
Ziyang Zhang
Thanks,
Pierrick
Quoted 11 lines · Ziyang Zhang
Feedback on the plugin and on the pass-through approach is welcome.
Changes since v8:
Following Pierrick Bouvier's v8 request, the guest program in the
docs/about/emulation.rstexample no longer links with an rpath. It carries no rpath at all, so which libhello.so it loads is chosen entirely byLD_LIBRARY_PATHat run time: the baseline run puts the guest build there, and the thunk run puts the generated guest thunk there in its place. The plugin code is unchanged.
This PATCH is deprecated, see v10:
https://lore.kernel.org/qemu-devel/[email protected]/