Skip to contentSkip to conversation

Yocto and NPM issues

DISCUSSION Svein Seldal 2 participants 5 messages Mar 21, 2018

lore
Share & tools
lore
#0 lore raw

Hi

I'm attempting to package a npm-based js application into a Rocko image, and I'm having some issues. For testing I use the official poky repo and the meta-openembedded repo (for installing nodejs 8.4.0). I'm building for qemux86-64.

Issue 1: NPM builds not working in Rocko

If I'm following the examples outlined in https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM, and run devtool add "npm://registry.npmjs.org;name=cute-files;version=1.0.2" and then bitbake cute-file. It fails on an npm error seen below. I found that this patch fixes the problem: https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/classes/npm.bbclass?id=d38e1e2c2ea4646b34ea6282d3d7620df5b0374b

Will there be more releases of rocko? Can I request that this patch is backported to the rocko branch please?

Issue 2: Workspace npm recipe does not work after bitbake -c cleanall

If using the devtool/workspace for the above recipe, and run bitbake -c cleanall cute-files, then try to rerun bitbake cute-files, it will warn on missing LICENSE files for the npm packages and fail on LIC_FILES_CHKSUM point to an invalid file QA error. E.g. something in line of:

WARNING: cute-files-1.0.2-r0 do_populate_lic: Could not copy license file /home/common/yocto/yocto-poky/build/workspace/sources/cute-files/node_modules/express/node_modules/serve-static/node_modules/send/node_modules/ms/license.md to /home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/license-destdir/cute-files/license.md: [Errno 2] No such file or directory: '/home/common/yocto/yocto-poky/build/workspace/sources/cute-files/node_modules/express/node_modules/serve-static/node_modules/send/node_modules/ms/license.md'
ERROR: cute-files-1.0.2-r0 do_populate_lic: QA Issue: cute-files: LIC_FILES_CHKSUM points to an invalid file: /home/common/yocto/yocto-poky/build/workspace/sources/cute-files/node_modules/express/node_modules/accepts/LICENSE [license-checksum]

Issue 3: npm recipe fails to download npm images after -c cleanall

If the recipe is copied into a normal layer and run without externalsrc, and runned after a -c cleanall, the following error is returned:

WARNING: cute-files-1.0.2-r0 do_fetch: Checksum failure encountered with download of npm://registry.npmjs.org;name=cute-files;version=1.0.2 - will attempt other sources if available
ERROR: cute-files-1.0.2-r0 do_fetch: Fetcher failure: Checksum mismatch!
File: 'commander-2.15.1.tgz' has sha1 checksum df46e867d0fc2aec66a34662b406a9ccafff5b0f when * was expected
ERROR: cute-files-1.0.2-r0 do_fetch: Fetcher failure for URL: 'npm://registry.npmjs.org;name=cute-files;version=1.0.2'. Unable to fetch URL from any source.
ERROR: cute-files-1.0.2-r0 do_fetch: Function failed: base_do_fetch
ERROR: Logfile of failure stored in: /home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/temp/log.do_fetch.20513
ERROR: Task (/home/common/yocto/yocto-poky/meta-foobar/recipes/cute-files/cute-files_1.0.2.bb:do_fetch) failed with exit code '1'

The only way to circumvent issue 2 and 3 after a -c cleanall is to use the devtool to download the packages, and then build the package as normal.

Best regards,
Svein

FAILURE Building cute-files:

ERROR: cute-files-1.0.2-r0 do_compile: Function failed: do_compile (log file is located at /home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/temp/log.do_compile.9536)
ERROR: Logfile of failure stored in: /home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/temp/log.do_compile.9536
Log data follows:

| DEBUG: Executing python function externalsrc_compile_prefunc
| NOTE: cute-files: compiling from external source tree /home/common/yocto/yocto-poky/build/workspace/sources/cute-files
| DEBUG: Python function externalsrc_compile_prefunc finished
| DEBUG: Executing shell function do_compile
| npm ERR! As of npm@5, the npm cache self-heals from corruption issues and data extracted from the cache is guaranteed to be valid. If you want to make sure everything is consistent, use 'npm cache verify' instead.
| npm ERR!
| npm ERR! If you're sure you want to delete the entire cache, rerun this command with --force.
|
| npm ERR! A complete log of this run can be found in:
| npm ERR!     /home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/npm_cache/_logs/2018-03-21T12_36_48_034Z-debug.log
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_compile (log file is located at /home/common/yocto/yocto-poky/build/tmp/work/core2-64-poky-linux/cute-files/1.0.2-r0/temp/log.do_compile.9536)

ERROR: Task (/home/common/yocto/yocto-poky/build/workspace/recipes/cute-files/cute-files_1.0.2.bb:do_compile) failed with exit code '1'

#1 lore raw
Quoted 3 lines · Svein Seldal

Hi

I'm attempting to package a npm-based js application into a Rocko image, and I'm having some issues. For testing I use the official poky repo and the meta-openembedded repo (for installing nodejs 8.4.0). I'm building for qemux86-64.

I know this is not an answer to your specific issues, but npm is a bit of maintenance nightmare as its workflow (downloading random stuff off the internet as an integral part of the build) clashes badly with how bitbake, and embedded world in general handle builds. We've discussed what to do about it, with no clear outcome, but we would like to have something that doesn't require constant fixing and hacks:

http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000480.html

Alex

#2 lore raw
Quoted 5 lines · Alexander Kanavin
Quoted 1 line

I'm attempting to package a npm-based js application into a Rocko image, and I'm having some issues. For testing I use the official poky repo and the meta-openembedded repo (for installing nodejs 8.4.0). I'm building for qemux86-64.

I know this is not an answer to your specific issues, but npm is a bit of maintenance nightmare as its workflow (downloading random stuff off the internet as an integral part of the build) clashes badly with how bitbake, and embedded world in general handle builds. We've discussed what to do about it, with no clear outcome, but we would like to have something that doesn't require constant fixing and hacks:

http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000480.html

Oh, and you might have more success using meta-nodejs, as it doesn't try to implement a custom fetcher, and just runs npm directly during the build. If you don't mind that that makes the build outcome essentially random from one invocation to the next.

https://github.com/imyller/meta-nodejs

Alex

#3 lore raw
In reply to Alexander Kanavin
Quoted 5 lines · Alexander Kanavin
Quoted 1 line

I'm attempting to package a npm-based js application into a Rocko image, and I'm having some issues. For testing I use the official poky repo and the meta-openembedded repo (for installing nodejs 8.4.0). I'm building for qemux86-64.

I know this is not an answer to your specific issues, but npm is a bit of maintenance nightmare as its workflow (downloading random stuff off the internet as an integral part of the build) clashes badly with how bitbake, and embedded world in general handle builds. We've discussed what to do about it, with no clear outcome, but we would like to have something that doesn't require constant fixing and hacks:

http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000480.html

I haven't read the post in full yet, but the principle of npm is inherently different from the philosophies of Yocto, granted.

One approach that we've been discussing as an alternative to the yocto+npm approach is to install the js app on a target machine and snapshot the whole resulting directory hierarchy. The product validation is then made on the modules as a whole. The recipe would need to be a simple file-installer.

The problem with that is that one needs a target deployment system that runs npm install on the actual target in order to make the correct files for the target. The output from that must be plugged into the oe build system somehow, so there will be many moving parts in this scheme too.

Quoted 6 lines

Oh, and you might have more success using meta-nodejs, as it doesn't try to implement a custom fetcher, and just runs npm directly during the build. If you don't mind that that makes the build outcome essentially random from one invocation to the next.

https://github.com/imyller/meta-nodejs

The meta-nodejs repo seems to be abandoned, as it does support any newer versions of nodejs. I asked on #yocto and I was suggested that this layer wasn't really necessary as meta-openembedded supports nodejs.

Best regards,
Svein

#4 lore raw
Quoted 8 lines · Svein Seldal
Quoted 6 lines

Oh, and you might have more success using meta-nodejs, as it doesn't try to implement a custom fetcher, and just runs npm directly during the build. If you don't mind that that makes the build outcome essentially random from one invocation to the next.

https://github.com/imyller/meta-nodejs

The meta-nodejs repo seems to be abandoned, as it does support any newer versions of nodejs. I asked on #yocto and I was suggested that this layer wasn't really necessary as meta-openembedded supports nodejs.

You have seen first-hand what 'supports' means in practice :-) As long as it's not regularly, automatically tested against a wide set of target npm packages, it just isn't going to work well.

I'd suggest you could take over the maintenance of meta-nodejs maybe?

Alex