New: Embedded NPM fails to start with "no such file or directory" error

https://bugs.busybox.net/show_bug.cgi?id=9526

Bug ID: 9526
Summary: Embedded NPM fails to start with "no such file or
directory" error
Product: buildroot
Version: 2016.11
Hardware: All
OS: Linux
Status: NEW
Severity: major
Priority: P5
Component: Other
Assignee: unassigned at buildroot.uclibc.org
Reporter: bilge at scriptfusion.com
CC: buildroot at uclibc.org
Target Milestone: ---

Please see this Travis build log for consistent reproduction of this issue:

https://travis-ci.org/Docker-nano/Node.js/builds/187419495#L378

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #1 from bilge at scriptfusion.com --- For completeness, here is the relevant portion of the log.

$ docker run --rm "$PRODUCT" npm

panic: standard_init_linux.go:175: exec user process caused "no such file or directory" [recovered]

panic: standard_init_linux.go:175: exec user process caused "no such
file or directory"

goroutine 1 [running, locked to thread]:

panic(0x7ddea0, 0xc820120b60)

        /usr/local/go/src/runtime/panic.go:481 +0x3e6

github.com/urfave/cli.HandleAction.func1(0xc8200f32e8)

/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x38e

panic(0x7ddea0, 0xc820120b60)

        /usr/local/go/src/runtime/panic.go:443 +0x4e9

github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc8200f2bf8, 0xc82001a088, 0xc8200f2d08)

/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x136

github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc820058820, 0x7f969f965470, 0xc820120b60)

/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x5b1

main.glob.func8(0xc820076780, 0x0, 0x0)

        /go/src/github.com/opencontainers/runc/main_unix.go:26 +0x68

reflect.Value.call(0x7448a0, 0x8f0bd0, 0x13, 0x839998, 0x4, 0xc8200f3268, 0x1, 0x1, 0x0, 0x0, ...)

        /usr/local/go/src/reflect/value.go:435 +0x120d

reflect.Value.Call(0x7448a0, 0x8f0bd0, 0x13, 0xc8200f3268, 0x1, 0x1, 0x0, 0x0, 0x0)

        /usr/local/go/src/reflect/value.go:303 +0xb1

github.com/urfave/cli.HandleAction(0x7448a0, 0x8f0bd0, 0xc820076780, 0x0, 0x0)

/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x2ee

github.com/urfave/cli.Command.Run(0x83c828, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8d0180, 0x51, 0x0, ...)

/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xfec

github.com/urfave/cli.(*App).Run(0xc820001680, 0xc82000a100, 0x2, 0x2, 0x0, 0x0)

/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0xaa4

main.main()

        /go/src/github.com/opencontainers/runc/main.go:137 +0xe24

The command "docker run --rm "$PRODUCT" npm" exited with 2.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #2 from Thomas Petazzoni <[email protected]> --- Isn't this a docker-specific issue, see
https://github.com/docker/docker/issues/27940 ?

Please provide a way of reproducing this without all the Docker related complexity. Thanks!

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #3 from bilge at scriptfusion.com ---
Sorry, I don't know how to do run a rootfs without Docker. You can download the compiled product from
https://github.com/Docker-nano/Node.js/blob/product/6.7/rootfs.tar.xz

If you prefer to build it yourself, here is the configuration.

BR2_x86_64=y
BR2_WGET="wget --passive-ftp -nd -t 3 -nv" BR2_TOOLCHAIN_EXTERNAL=y BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y BR2_TOOLCHAIN_EXTERNAL_PATH="$(HOME)/toolchain" BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="$(ARCH)-nano-linux-uclibc" BR2_TOOLCHAIN_EXTERNAL_GCC_5=y BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_3=y BR2_TOOLCHAIN_EXTERNAL_WCHAR=y BR2_TOOLCHAIN_EXTERNAL_HAS_SSP=y BR2_TOOLCHAIN_EXTERNAL_CXX=y BR2_TARGET_GENERIC_HOSTNAME="Docker-nano" BR2_TARGET_GENERIC_ISSUE="Docker nano"
BR2_INIT_NONE=y
# BR2_TARGET_GENERIC_GETTY is not set BR2_ROOTFS_OVERLAY="rootfs_overlay" BR2_ROOTFS_POST_BUILD_SCRIPT="./post_build.sh"
# BR2_PACKAGE_BUSYBOX is not set
BR2_PACKAGE_NODEJS=y
BR2_PACKAGE_NODEJS_NPM=y
BR2_TARGET_ROOTFS_TAR_XZ=y

This uses the toolchain available from
https://github.com/Docker-nano/crosstool-NG/releases/download/2.0.0/x86_64-nano-linux-uclibc.tar.xz

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #4 from Thomas Petazzoni <[email protected]> --- What is your ./post-build.sh script doing ?

Remember: we need to be able to reproduce the issues that you face, otherwise, no way we can debug them.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #5 from bilge at scriptfusion.com ---
post-build.sh is located here
https://github.com/Docker-nano/Buildroot/blob/2016.11/in/post_build.sh.

Its contents I have copied in below.

It just trims the fat by removing files that are presumed to be useless for the majority of builds. I had not included it because it is thought to be inconsequential.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #6 from bilge at scriptfusion.com ---
I have found out why this happens. It is because /usr/lib/node_modules/npm/bin/npm-cli.js (the target NPM is linked to) begins with the line, "#!/usr/bin/env node", but /usr/bin/env does not exist! If this is manually patched to "#!/usr/bin/node" it works as expected.

I believe Buildroot should either include this patch, a copy of env or perhaps even some sort of env shim. In its current state NPM will always fail.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #7 from Peter Korsgaard <[email protected]> --- /usr/bin/env is provided by busybox or coreutils, so a fix is simply to enable one of those.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #8 from bilge at scriptfusion.com ---
Since the aim of the project is to keep things small, artificially bloating the build with the entirety of either Busybox or Coreutils is highly undesirable.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #9 from bilge at scriptfusion.com ---
Here is a Buildroot patch I wrote.

+++ x/deps/npm/bin/npm-cli.js
@@ -1,4 +1,4 @@
-#!/usr/bin/env node
+#!/usr/bin/node
;(function () { // wrapper in case we're in module_context mode
// windows: running "npm blah" in this folder will invoke WSH, not node. /*global WScript*/

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #10 from Peter Korsgaard <[email protected]> --- (In reply to bilge from comment #8)

Compared to nodejs, busybox is tiny - E.G. a busybox build with only the env applet is ~60K here.

Lots of software expects a minimum of basic UNIX functionality (coreutils things like ls/cp/mv/cat/env/.. and a shell). We do give you the option in Buildroot to disable both as there are use cases where it makes sense, but it really is an advanced case and you are expected to know what you are doing if you do so and handle any breakage yourself.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #11 from Peter Korsgaard <[email protected]> --- (In reply to bilge from comment #9)

I suggest you handle that with sed in your post-build script.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #12 from bilge at scriptfusion.com ---
I tend to agree that sed in post-build is better, because although the patch works, it applies to both the host and the target when we only want it to apply to the target. There isn't a problem with it being applied to the host until it tries to build NPM modules, in which case it dies horribly!

Regarding Busybox and Coreutils, I tried to build both but each failed in their own spectacular way. Right now the post-build script still seems easier. However, I noticed Busybox permits a configuration so perhaps it is possible to just build the env utility by itself?

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

Thomas Petazzoni <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

         Resolution|---                         |WONTFIX
             Status|NEW                         |RESOLVED

--- Comment #13 from Thomas Petazzoni <[email protected]> --- As Peter explained, we do expect a minimal set of utilities on the target, and we're not going to start patching upstream packages because something as basic as "env" might be missing. Users interested in such a fine-grained customization can do that in a post-build script.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #14 from bilge at scriptfusion.com ---
Tom, if that is your decision, I can respect it. However, there is still the outstanding issue that env is an unlisted dependency of NPM and even though I could build it myself (e.g. building Busybox/Coreutils as directed by Peter), there is no guarantee that it would actually build those dependencies _before_ NPM and may still fail!

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #15 from Peter Korsgaard <[email protected]> --- (In reply to bilge from comment #14)

Ehh, I'm getting confused here. Are you building on a system without /usr/bin/env? (E.G. in some kind of minimal container?)

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #16 from bilge at scriptfusion.com ---
Peter, that's exactly correct. There is nothing on the system other than what Buildroot builds, and as you can see from the configuration:

# BR2_PACKAGE_BUSYBOX is not set

There is no env on the target system unless Buildroot builds it, which it currently does not.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #17 from Peter Korsgaard <[email protected]> --- (In reply to bilge from comment #16)

Sorry, I still don't understand you. The Buildroot configuration is about what gets built FOR THE TARGET, not what is available on the build machine (except for host packages, but that is not applicable for busybox).

The build machine needs basic UNIX utilities to be able to build the packages (most, but not all get verified by Buildroot in support/dependencies.sh).

Can you explain more about your setup?

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #18 from bilge at scriptfusion.com ---
Peter, sorry I must have misunderstood you. The build machine is a standard Debian 8 image available as a Docker image from
https://hub.docker.com/r/dockernano/buildroot/. That build system certainly has env present as well as the following Debian packages: build-essential libncurses5-dev rsync cpio python unzip bc wget (as specified here:
https://github.com/Docker-nano/Buildroot/blob/2016.11/Dockerfile).

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #19 from bilge at scriptfusion.com ---
Peter, I think the point is that it does not matter whether env exists on the host or not; it must exist on the target in order for NPM to function on the target. Ergo, BusyBox must be built before NPM, which it is incidentally, but this is not guaranteed since there is no dependency constraint between NPM and BusyBox (or any other provider of env).

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #20 from Peter Korsgaard <[email protected]> --- (In reply to bilge from comment #19)

I think we are talking past eachother. Dependencies for on the target (E.G. the thing that will finally do something with the rootfs) are all runtime dependencies, so the build order doesn't matter.

So yeah, you simply need to enable busybox as well to make sure /usr/bin/env is also available in the rootfs when the target needs it at runtime. You can tweak the busybox config if you want to save some space, but you need to be careful to not remove stuff that is needed elsewhere (E.G. env)

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9526

--- Comment #21 from bilge at scriptfusion.com ---
Yeah, I'm sorry, you're right. It doesn't really matter what order they're built as long as env is only need on the target. I got confused because of the other bug (9531) where I thought I was being told env was also needed *on the target* in order to build modules. However, I believe that bug is not related to the env issue at all.

Regardless, this bug can be laid to rest, although I still feel a little bad that no change will be made to avoid someone else encountering the same problem, even if it's just a note in the documentation.

You are receiving this mail because:
You are on the CC list for the bug.