NodeJS, qemu wrapper and ELF interpreter fun
Hello,
I finally took some time to investigate the NodeJS build issue that occurs on x86-64:
http://autobuild.buildroot.net/results/4f2/4f29c5e663b5d01af20d671baf6504332b8c1f7b/build-end.log
The failure goes like this:
cd ../../tools/v8_gypfiles; python ../../deps/v8/tools/run.py ../../out/Release/v8-qemu-wrapper ../../out/Release/bytecode_builtins_list_generator ../../out/Release/obj.host/gen/generate-bytecode-output-root/builtins-generated/bytecodes-builtins-list.h ../../out/Release/bytecode_builtins_list_generator: /lib/x86_64-linux-gnu/libstdc++.so.6: version GLIBCXX_3.4.29' not found (required by ../../out/Release/bytecode_builtins_list_generator) ../../out/Release/bytecode_builtins_list_generator: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.33' not found (required by ../../out/Release/bytecode_builtins_list_generator) ../../out/Release/bytecode_builtins_list_generator: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.32' not found (required by ../../out/Release/bytecode_builtins_list_generator) ../../out/Release/bytecode_builtins_list_generator: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by ../../out/Release/bytecode_builtins_list_generator)
Return code is 1
This problem also has a bug report at
https://bugs.busybox.net/show_bug.cgi?id=14366.
As you probably know, v8-qemu-wrapper is a script that uses Qemu user-mode emulation to run some programs built for the target. We are then supposed to use the libraries built for the target, but here for some reason, the C library of the host gets used, and it fails badly because it is older than the C library that is available for the target.
So the question is: why the heck does Qemu use the host C library?
To answer this question, we first have to look at how the -L option of Qemu is implemented. This option is documented as such:
-L path QEMU_LD_PREFIX set the elf interpreter prefix to 'path'
And the v8-qemu-wrapper script makes this option point to $(STAGING_DIR), so that the ELF interpreter used is the one in $(STAGING_DIR).
However, contrary to what the option documentation says, this option does much more than setting the ELF interpreter prefix: it is going to affect how *all* system calls manipulating files (open, etc.) are going to work.
When this option is passed, the function init_paths() in
https://git.qemu.org/?p=qemu.git;a=blob;f=util/path.c is called at initialization time, and essentially its sets the global "base" variable to point to the directory passed as -L argument.
Then, for every single syscall that manipulates a path, this path will be passed through the path() function in the same file. This function will first attempt to resolve the path with "base" as a prefix, and if not, return the unprefixed path.
After adding some traces into this function, I was able to understand what happens:
(1) -L$(STAGING_DIR) is passed, causing "base" to point to $(STAGING_DIR)
(2) The target ELF interpreter from $(STAGING_DIR) is properly invoked
(3) When this ELF interpreter then resolves the libc.so.6 library, it first looks for /etc/ld.so.cache.
(4) Qemu first looks for /etc/ld.so.cache with the -L prefix, i.e $(STAGING_DIR)/etc/ld.so.cache, but it does not exist. So, the Qemu system call emulation falls back to /etc/ld.so.cache, which means the target ELF interpreter reads the /etc/ld.so.cache of the host system.
(5) This /etc/ld.so.cache of the host system says that libc.so.6 is in
/lib/x86_64-linux-gnu/
(6) The target ELF interpreter therefore tries to use /lib/x86_64-linux-gnu/libc.so.6. The Qemu system call emulation first tries $(STAGING_DIR)/lib/x86_64-linux-gnu/libc.so.6, but this library does not exist (it is in $(STAGING_DIR)/lib/libc.so.6), so the Qemu system call emulation falls back to /lib/x86_64-linux-gnu/libc.so.6 of the host system, which exist... but is too old compared to the target C library. Indeed, results from ld.so.cache take precedence over the simple resolution of library paths in /usr/lib and /lib.
We see 3 possible ideas to resolve this problem:
(A) Change the behavior of Qemu to not fallback to unprefixed paths:
when -L is passed, all path-related system calls should see the paths prefixed by the -L option.
Issue with this is that this change is unlikely to get accepted by Qemu upstream. And there might be some side effects we have not really identified.
(B) Create an empty $(STAGING_DIR)/etc/ld.so.cache. We have tested this solution and it works: it gets used instead of the host /etc/ld.so.cache. Because $(STAGING_DIR)/etc/ld.so.cache is empty, there's no libc.so.6 match, so the target ELF interpreter goes through its normal library location resolution logic, which falls back to trying in /usr/lib and /lib, which works as those paths ends up being prefixed with $(STAGING_DIR) by Qemu.
(C) Pass LD_LIBRARY_PATH pointing to $(STAGING_DIR)/lib and $(STAGING_DIR)/usr/lib in the Qemu wrapper. This works because LD_LIBRARY_PATH paths have precedence over paths given by ld.so.cache.
This is the solution already used by the GOI qemu wrapper in package/gobject-introspection/g-ir-scanner-qemuwrapper.in, and which was suggested in bug
https://bugs.busybox.net/show_bug.cgi?id=14366.
Any opinion? Or other ideas?
Thanks!
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
...
Quoted 11 lines
(C) Pass
LD_LIBRARY_PATHpointing to $(STAGING_DIR)/lib and $(STAGING_DIR)/usr/lib in the Qemu wrapper. This works becauseLD_LIBRARY_PATHpaths have precedence over paths given by ld.so.cache.This is the solution already used by the GOI qemu wrapper in package/gobject-introspection/g-ir-scanner-qemuwrapper.in, and which was suggested in bug
https://bugs.busybox.net/show_bug.cgi?id=14366.Any opinion? Or other ideas?
Using LD_LIBRARY_PATH like that is a bad idea. Just google for it and you'll find a load of stuff about it.
LD_LIBRARY_PATH should only really be used not substitute a non-standard library - eg for testing.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
David, All,
On 2022-11-01 08:46 +0000, David Laight spake thusly:
Quoted 5 lines
Quoted 5 lines
(C) Pass
LD_LIBRARY_PATHpointing to $(STAGING_DIR)/lib and $(STAGING_DIR)/usr/lib in the Qemu wrapper. This works becauseLD_LIBRARY_PATHpaths have precedence over paths given by ld.so.cache.
Quoted 8 lines
Quoted 6 lines
This is the solution already used by the GOI qemu wrapper in package/gobject-introspection/g-ir-scanner-qemuwrapper.in, and which was suggested in bug
https://bugs.busybox.net/show_bug.cgi?id=14366.Any opinion? Or other ideas?
Using
LD_LIBRARY_PATHlike that is a bad idea.
Yeah, we know quite a bit about some LD_LIBRARY_PATH woes ;-)
Quoted 3 lines
Just google for it and you'll find a load of stuff about it.
LD_LIBRARY_PATHshould only really be used not substitute a non-standard library - eg for testing.
In this situation, the issue is that we are running a *target* program, on the *build machine*, and we want it to pick target ilbraries, not the host libraries.
Of course, during cross-compilation, we can't run target code on the build machine, so we run it through qemu-user.
However, as Thomas explained and demonstrated, qemu-user intercepts and emulates the open()-like syscalls. Then it has a heuristic to resolve the opened path. This heuristic fails for us in some cases, however.
Thus we need a way to force-tell where to look for libraries.
So, yes, LD_LIBRARY_PATH is far from a panacea, but technically it makes things work.
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Thomas, All,
On 2022-10-31 21:39 +0100, Thomas Petazzoni spake thusly:
Quoted 3 lines
I finally took some time to investigate the NodeJS build issue that occurs on x86-64:
http://autobuild.buildroot.net/results/4f2/4f29c5e663b5d01af20d671baf6504332b8c1f7b/build-end.log
[--SNIP--]
Quoted 2 lines
After adding some traces into this function, I was able to understand what happens:
Great explanations, thanks a lot!
Quoted 8 lines
We see 3 possible ideas to resolve this problem:
(A) Change the behavior of Qemu to not fallback to unprefixed paths:
when -L is passed, all path-related system calls should see the paths prefixed by the -L option.
Issue with this is that this change is unlikely to get accepted by Qemu upstream. And there might be some side effects we have not really identified.
I am very skeptical of that solution...
Quoted 7 lines
(B) Create an empty $(STAGING_DIR)/etc/ld.so.cache. We have tested this solution and it works: it gets used instead of the host
/etc/ld.so.cache. Because $(STAGING_DIR)/etc/ld.so.cache is empty, there's no libc.so.6 match, so the target ELF interpreter goes through its normal library location resolution logic, which falls back to trying in/usr/liband /lib, which works as those paths ends up being prefixed with $(STAGING_DIR) by Qemu.
You'd also need an empty /etc/ld.so.cache.d/ maybe.
I was looking at the manpage for ld.so, and it has an option to disable looking in the cache:
--inhibit-cache
Do not use /etc/ld.so.cache.
However, we are not calling ld.so directly, so we'd need to pass it via an environment variable, but there's no such environment variable that would allow disable the cache.
I'm afraid an empty STAGING_DIR/etc/ld.so.cache might not be enough, but I have no fact to back this gut feeling...
Quoted 9 lines
(C) Pass
LD_LIBRARY_PATHpointing to $(STAGING_DIR)/lib and $(STAGING_DIR)/usr/lib in the Qemu wrapper. This works becauseLD_LIBRARY_PATHpaths have precedence over paths given by ld.so.cache.This is the solution already used by the GOI qemu wrapper in package/gobject-introspection/g-ir-scanner-qemuwrapper.in, and which was suggested in bug
https://bugs.busybox.net/show_bug.cgi?id=14366.
We know this has been working pretty well for the GOI wrapper, so if we were to use LD_LIBRARY_PATH, that would make for consistent wrappers.
Quoted 1 line
Any opinion? Or other ideas?
I would go for the LD_LIBRARY_PATH solution, if at least because it is consistent with the GOI wrapper.
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot