kexec-tools: Perform run-time linking of libxenctrl.so
Subsystems (2)
PATCH 1/1 kexec-tools: Perform run-time linking of libxenctrl.so
When kexec is utilized in a Xen environment, it has an explicit run-time dependency on libxenctrl.so. This dependency occurs during the configure stage and when building kexec-tools.
When kexec is utilized in a non-Xen environment (either bare metal or KVM), the configure and build of kexec-tools omits any reference to libxenctrl.so.
Thus today it is not currently possible to configure and build a *single* kexec that will work in *both* Xen and non-Xen environments, unless the libxenctrl.so is *always* present.
For example, a kexec configured for Xen in a Xen environment:
# ldd build/sbin/kexec
linux-vdso.so.1 => (0x00007ffdeba5c000)
libxenctrl.so.4.4 => /usr/lib64/libxenctrl.so.4.4 (0x00000038d8000000)
libz.so.1 => /lib64/libz.so.1 (0x00000038d6c00000)
libc.so.6 => /lib64/libc.so.6 (0x00000038d6000000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000038d6400000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000038d6800000)
/lib64/ld-linux-x86-64.so.2 (0x000055e9f8c6c000)
# build/sbin/kexec -v
kexec-tools 2.0.16However, the *same* kexec executable fails in a non-Xen environment:
# copy xen kexec to .
# ldd ./kexec
linux-vdso.so.1 => (0x00007fffa9da7000)
libxenctrl.so.4.4 => not found
liblzma.so.0 => /usr/lib64/liblzma.so.0 (0x0000003014e00000)
libz.so.1 => /lib64/libz.so.1 (0x000000300ea00000)
libc.so.6 => /lib64/libc.so.6 (0x000000300de00000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000000300e200000)
/lib64/ld-linux-x86-64.so.2 (0x0000558cc786c000)
# ./kexec -v
./kexec: error while loading shared libraries:
libxenctrl.so.4.4: cannot open shared object file: No such file or directoryAt Oracle we "workaround" this by having two kexec-tools packages, one for Xen and another for non-Xen environments. At Oracle, the desire is to offer a single kexec-tools package that works in either environment. To achieve this, kexec-tools would either have to ship with libxenctrl.so (which we have deemed as unacceptable), or we can make kexec perform run-time linking against libxenctrl.so.
This patch is one possible way to alleviate the explicit run-time dependency on libxenctrl.so. This implementation utilizes a set of macros to wrap calls into libxenctrl.so so that the library can instead be dlopen() and obtain the function via dlsym() and then make the call. The advantage of this implementation is that it requires few changes to the existing kexec-tools code. The dis- advantage is that it uses macros to remap libxenctrl functions and do work under the hood.
Another possible implementation worth considering is the approach taken by libvmi. Reference the following file:
https://github.com/libvmi/libvmi/blob/master/libvmi/driver/xen/libxc_wrapper.h
The libxc_wrapper_t structure definition that starts at line ~33 has members that are function pointers into libxenctrl.so. This structure is populated once and then later referenced/dereferenced by the callers of libxenctrl.so members. The advantage of this implementation is it is more explicit in managing the use of libxenctrl.so and its versions, but the disadvantage is it would require touching more of the kexec-tools code.
The following is a list libxenctrl members utilized by kexec:
Functions:xc_interface_openxc_kexec_get_rangexc_interface_closexc_kexec_get_rangexc_interface_openxc_get_max_cpusxc_kexec_get_rangexc_versionxc_kexec_execxc_kexec_statusxc_kexec_unloadxc_hypercall_buffer_array_createxc__hypercall_buffer_array_allocxc_hypercall_buffer_array_destroyxc_kexec_loadxc_get_machine_memory_map
Data:xc__hypercall_buffer_HYPERCALL_BUFFER_NULL
These were identified by configuring and building kexec-tools with Xen support, but omitting the -lxenctrl from the LDFLAGS in the Makefile for an x86_64 build.
The above libxenctrl members were referenced via these source files.
kexec/crashdump-xen.ckexec/kexec-xen.ckexec/arch/i386/kexec-x86-common.ckexec/arch/i386/crashdump-x86.c
This patch provides a wrapper around the calls to the above functions in libxenctrl.so. Every libxenctrl call must pass a xc_interface which it obtains from xc_interface_open(). So the existing code is already structured in a manner that facilitates graceful dlopen()'ing of the libxenctrl.so and the subsequent dlsym() of the required member.
The patch creates a wrapper function around xc_interface_open() and xc_interface_close() to perform the dlopen() and dlclose().
For the remaing xc_ functions, this patch defines a macro of the same name which performs the dlsym() and then invokes the function. See the _xc_call() macro for details.
There was one data item in libxenctrl.so that presented a unique problem, HYPERCALL_BUFFER_NULL. It was only utilized once, as
set_xen_guest_handle(xen_segs[s].buf.h, HYPERCALL_BUFFER_NULL);
I tried a variety of techniques but could not find a general macro-type solution without modifying xenctrl.h. So the solution was to declare a local HYPERCALL_BUFFER_NULL, and this appears to work. I admit I am not familiar with libxenctrl to state if this is a satisfactory workaround, so feedback here welcome. I can state that this allows kexec to load/unload/kexec on Xen and non-Xen environments that I've tested without issue.
With this patch applied, kexec-tools can be built with Xen support and yet there is no explicit run-time dependency on libxenctrl.so. Thus it can also be deployed in non-Xen environments where libxenctrl.so is not installed.
# ldd build/sbin/kexec
linux-vdso.so.1 => (0x00007fff7dbcd000)
liblzma.so.0 => /usr/lib64/liblzma.so.0 (0x00000038d9000000)
libz.so.1 => /lib64/libz.so.1 (0x00000038d6c00000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000038d6400000)
libc.so.6 => /lib64/libc.so.6 (0x00000038d6000000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000038d6800000)
/lib64/ld-linux-x86-64.so.2 (0x0000562dc0c14000)
# build/sbin/kexec -v
kexec-tools 2.0.16Currently this feature is enabled with the following:
./configure --with-xen-dl --with-xen=no
This is a bit clunky. I welcome feedback such as better names and/or usage of --with, as well as if we might make this feature the default.
--- configure.ac | 18 ++++++++++++++ kexec/Makefile | 1 + kexec/arch/i386/crashdump-x86.c | 4 +--- kexec/arch/i386/kexec-x86-common.c | 4 +--- kexec/crashdump-xen.c | 4 +--- kexec/kexec-xen.c | 43 +++++++++++++++++++++++++++++++++- kexec/kexec-xen.h | 48 ++++++++++++++++++++++++++++++++++++++ 7 files changed, 112 insertions(+), 10 deletions(-) create mode 100644 kexec/kexec-xen.h
configure.ac +18 -0
@@ -92,6 +92,9 @@ AC_ARG_WITH([lzma], AC_HELP_STRING([--without-lzma],[disable lzma support]), AC_ARG_WITH([xen], AC_HELP_STRING([--without-xen], [disable extended xen support]), [ with_xen="$withval"], [ with_xen=yes ] ) +AC_ARG_WITH([xen-dl], AC_HELP_STRING([--without-xen-dl], + [link libxenctrl.so at run-time rather than build-time]), [ with_xen_dl="$withval"], [ with_xen_dl=no ] ) + AC_ARG_WITH([booke], AC_HELP_STRING([--with-booke],[build for booke]), AC_DEFINE(CONFIG_BOOKE,1,
@@ -174,6 +177,21 @@ if test "$with_xen" = yes ; then AC_MSG_NOTICE([The kexec_status call is not available])) fi fi +if test "$with_xen_dl" = yes ; then + if test "$with_xen" = yes ; then + AC_MSG_ERROR([Options --with-xen and --with-xen-dl are mutually exclusive]) + fi + AC_DEFINE(CONFIG_LIBXENCTRL_DL, 1, [Define to 1 to link libxenctrl.so at run-time rather than build-time]) + AC_CHECK_HEADER(dlfcn.h, , AC_MSG_ERROR([Dynamic library linking not available])) + AC_CHECK_LIB(dl, dlopen, , AC_MSG_ERROR([Dynamic library linking not available])) + AC_CHECK_HEADER(xenctrl.h, + AC_DEFINE(HAVE_LIBXENCTRL, 1, [Define to 1 to enable run-time linking of libxenctrl.so]), + AC_MSG_ERROR([Xen support not available])) +dnl NOTE: Explicitly *NOT* performing AC_CHECK_LIB(xenctrl) as only need the header file to build + AC_CHECK_HEADER(xenctrl.h, + AC_DEFINE(HAVE_KEXEC_CMD_STATUS, 1, [Define to 1 so kexec_status call is available]), + AC_MSG_NOTICE([The kexec_status call is not available])) +fi dnl ---Sanity checks if test "$CC" = "no"; then AC_MSG_ERROR([cc not found]); fi
kexec/Makefile +1 -0
@@ -36,6 +36,7 @@ dist += kexec/Makefile \ kexec/kexec-elf-boot.h \ kexec/kexec-elf.h kexec/kexec-sha256.h \ kexec/kexec-zlib.h kexec/kexec-lzma.h \ + kexec/kexec-xen.h \ kexec/kexec-syscall.h kexec/kexec.h kexec/kexec.8 dist += kexec/proc_iomem.c
kexec/arch/i386/crashdump-x86.c +1 -3
@@ -44,9 +44,7 @@#include "kexec-x86.h" #include "crashdump-x86.h" -#ifdef HAVE_LIBXENCTRL -#include <xenctrl.h> -#endif /* HAVE_LIBXENCTRL */ +#include "../../kexec-xen.h" #include "x86-linux-setup.h"
kexec/arch/i386/kexec-x86-common.c +1 -3
@@ -40,9 +40,7 @@#include "../../crashdump.h" #include "kexec-x86.h" -#ifdef HAVE_LIBXENCTRL -#include <xenctrl.h> -#endif /* HAVE_LIBXENCTRL */ +#include "../../kexec-xen.h" /* Used below but not present in (older?) xenctrl.h */ #ifndef E820_PMEM
kexec/crashdump-xen.c +1 -3
@@ -18,9 +18,7 @@#include "config.h" -#ifdef HAVE_LIBXENCTRL -#include <xenctrl.h> -#endif +#include "kexec-xen.h" struct crash_note_info {unsigned long base;
kexec/kexec-xen.c +42 -1
52 lines not on this page · raw message
kexec/kexec-xen.h new file +48 -0
48 lines not on this page · raw message
Signature · 7 lines
2.7.4 _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
Quoted 11 lines · Eric DeVolder
When kexec is utilized in a Xen environment, it has an explicit run-time dependency on libxenctrl.so. This dependency occurs during the configure stage and when building kexec-tools.
When kexec is utilized in a non-Xen environment (either bare metal or KVM), the configure and build of kexec-tools omits any reference to libxenctrl.so.
Thus today it is not currently possible to configure and build a *single* kexec that will work in *both* Xen and non-Xen environments, unless the libxenctrl.so is *always* present.
[...]
I will take a look at this next week.
Thanks,
Daniel
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 61 lines · Eric DeVolder
When kexec is utilized in a Xen environment, it has an explicit run-time dependency on libxenctrl.so. This dependency occurs during the configure stage and when building kexec-tools.
When kexec is utilized in a non-Xen environment (either bare metal or KVM), the configure and build of kexec-tools omits any reference to libxenctrl.so.
Thus today it is not currently possible to configure and build a *single* kexec that will work in *both* Xen and non-Xen environments, unless the libxenctrl.so is *always* present.
For example, a kexec configured for Xen in a Xen environment:
# ldd build/sbin/kexec linux-vdso.so.1 => (0x00007ffdeba5c000) libxenctrl.so.4.4 => /usr/lib64/libxenctrl.so.4.4 (0x00000038d8000000) libz.so.1 => /lib64/libz.so.1 (0x00000038d6c00000) libc.so.6 => /lib64/libc.so.6 (0x00000038d6000000) libdl.so.2 => /lib64/libdl.so.2 (0x00000038d6400000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00000038d6800000) /lib64/ld-linux-x86-64.so.2 (0x000055e9f8c6c000) # build/sbin/kexec -v kexec-tools 2.0.16However, the *same* kexec executable fails in a non-Xen environment:
# copy xen kexec to . # ldd ./kexec linux-vdso.so.1 => (0x00007fffa9da7000) libxenctrl.so.4.4 => not found liblzma.so.0 => /usr/lib64/liblzma.so.0 (0x0000003014e00000) libz.so.1 => /lib64/libz.so.1 (0x000000300ea00000) libc.so.6 => /lib64/libc.so.6 (0x000000300de00000) libpthread.so.0 => /lib64/libpthread.so.0 (0x000000300e200000) /lib64/ld-linux-x86-64.so.2 (0x0000558cc786c000) # ./kexec -v ./kexec: error while loading shared libraries: libxenctrl.so.4.4: cannot open shared object file: No such file or directoryAt Oracle we "workaround" this by having two kexec-tools packages, one for Xen and another for non-Xen environments. At Oracle, the desire is to offer a single kexec-tools package that works in either environment. To achieve this, kexec-tools would either have to ship with libxenctrl.so (which we have deemed as unacceptable), or we can make kexec perform run-time linking against libxenctrl.so.
This patch is one possible way to alleviate the explicit run-time dependency on libxenctrl.so. This implementation utilizes a set of macros to wrap calls into libxenctrl.so so that the library can instead be
dlopen()and obtain the function viadlsym()and then make the call. The advantage of this implementation is that it requires few changes to the existing kexec-tools code. The dis- advantage is that it uses macros to remap libxenctrl functions and do work under the hood.Another possible implementation worth considering is the approach taken by libvmi. Reference the following file:
https://github.com/libvmi/libvmi/blob/master/libvmi/driver/xen/libxc_wrapper.h
Quoted 7 lines
The
libxc_wrapper_tstructure definition that starts at line ~33 has members that are function pointers into libxenctrl.so. This structure is populated once and then later referenced/dereferenced by the callers of libxenctrl.so members. The advantage of this implementation is it is more explicit in managing the use of libxenctrl.so and its versions, but the disadvantage is it would require touching more of the kexec-tools code.
After some thinking it seems to me that we can consider different approach. Could not we pack all basic Xen libraries into small package called, e.g. libxen? E.g. Debian provides libxen-4.8 which contains following files:
/usr/lib/x86_64-linux-gnu/libxencall-4.8.so.1 /usr/lib/x86_64-linux-gnu/libxencall-4.8.so.1.0 /usr/lib/x86_64-linux-gnu/libxenctrl-4.8.so /usr/lib/x86_64-linux-gnu/libxenevtchn-4.8.so.1 /usr/lib/x86_64-linux-gnu/libxenevtchn-4.8.so.1.0 /usr/lib/x86_64-linux-gnu/libxenforeignmemory-4.8.so.1 /usr/lib/x86_64-linux-gnu/libxenforeignmemory-4.8.so.1.0 /usr/lib/x86_64-linux-gnu/libxengnttab-4.8.so.1 /usr/lib/x86_64-linux-gnu/libxengnttab-4.8.so.1.1 /usr/lib/x86_64-linux-gnu/libxenguest-4.8.so /usr/lib/x86_64-linux-gnu/libxenlight-4.8.so /usr/lib/x86_64-linux-gnu/libxentoollog-4.8.so.1 /usr/lib/x86_64-linux-gnu/libxentoollog-4.8.so.1.0 /usr/lib/x86_64-linux-gnu/libxlutil-4.8.so /usr/share/bug/libxen-4.8/control /usr/share/doc/libxen-4.8/changelog.Debian.gz /usr/share/doc/libxen-4.8/changelog.gz /usr/share/doc/libxen-4.8/copyright
Then we do not have to install all other Xen stuff to make kexec-tools happy on non-Xen machines. And this should also work for other packages requiring above listed Xen libraries.
Daniel
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec