Skip to contentSkip to conversation

nodejs: drop 'gyp' PACKAGECONFIG

PATCH Superseded André Draszik 2 participants 5 messages Mar 3, 2020

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 nodejs: drop 'gyp' PACKAGECONFIG

From: André Draszik <[email protected]>

During the python3 / nodejs update, the dependencies weren't updated, so
using system-gyp ends up trying to use the python2 version of system-
gyp, which will of course fail.
Fixing this to depend on the python3 version of gyp still doesn't
doesn't make things work, though:
ERROR: nodejs-native-12.14.1-r0 do_configure: Execution of '.../nodejs-native/12.14.1-r0/temp/run.do_configure.26054' failed with exit code 1:
gyp: Error importing pymod_do_mainmodule (ForEachFormat): No module named 'ForEachFormat' while loading dependencies of .../nodejs-native/12.14.1-r0/node-v12.14.1/node.gyp while trying to load .../nodejs-native/12.14.1-r0/node-v12.14.1/node.gyp Error running GYP

The reason is the following patch nodejs has applied to its version of gyp as of NodeJS v12 (commit fff922afee6e ("deps,build: compute torque_outputs in v8.gyp") upstream):

tools/gyp/pylib/gyp/input.py 2020-03-02 12:16:09.956707788 +0000 renamed from pylib/gyp/input.py 2020-03-02 12:36:30.788248197 +0000 +2 -0
@@ -890,6 +881,7 @@ def ExpandVariables(input, phase, variab
           oldwd = os.getcwd()  # Python doesn't like os.open('.'): no fchdir.
           if build_file_dir:  # build_file_dir may be None (see above).
             os.chdir(build_file_dir)
+          sys.path.append(os.getcwd())
           try:
 
             parsed_contents = shlex.split(contents)
@@ -900,6 +892,7 @@ def ExpandVariables(input, phase, variab
                              "module (%s): %s" % (parsed_contents[0], e))
             replacement = str(py_module.DoMain(parsed_contents[1:])).rstrip()
           finally:
+            sys.path.pop()
             os.chdir(oldwd)
           assert replacement != None
         elif command_string:

Since I'm not sure how to deal with that when using system-gyp, and because the original intention for using system-gyp was to make the previous nodejs version compatible with python3 by ultimately switching to the python3 version of system-gyp which isn't necessary anymore, and given nobody else seems to be using this PACKAGECONFIG, just drop it.

---
 meta-oe/recipes-devtools/nodejs/nodejs_12.14.1.bb | 3 ---
 1 file changed, 3 deletions(-)
meta-oe/recipes-devtools/nodejs/nodejs_12.14.1.bb +0 -3
index 6eb52c209..49bb71e28 100644
@@ -53,7 +53,6 @@ ARCHFLAGS ?= ""
 
 PACKAGECONFIG ??= "ares icu libuv zlib"
 PACKAGECONFIG[ares] = "--shared-cares,,c-ares"
-PACKAGECONFIG[gyp] = ",,gyp-py2-native"
 PACKAGECONFIG[icu] = "--with-intl=system-icu,--without-intl,icu"
 PACKAGECONFIG[libuv] = "--shared-libuv,,libuv"
 PACKAGECONFIG[nghttp2] = "--shared-nghttp2,,nghttp2"
@@ -82,8 +81,6 @@ python do_unpack() {
     shutil.rmtree(d.getVar('S') + '/deps/openssl', True)
     if 'ares' in d.getVar('PACKAGECONFIG'):
         shutil.rmtree(d.getVar('S') + '/deps/cares', True)
-    if 'gyp' in d.getVar('PACKAGECONFIG'):
-        shutil.rmtree(d.getVar('S') + '/tools/gyp', True)
     if 'libuv' in d.getVar('PACKAGECONFIG'):
         shutil.rmtree(d.getVar('S') + '/deps/uv', True)
     if 'nghttp2' in d.getVar('PACKAGECONFIG'):
Signature · 1 line
2.23.0.rc1
#1 lore raw

PATCH 2/3 brotli: allow building a -native version

From: André Draszik <[email protected]>

nodejs (-native) can be built against a system-installed version of the brotli libraries. Allow doing that by provide a -native version of brotli.

---
 meta-oe/recipes-extended/brotli/brotli_1.0.7.bb | 2 ++
 1 file changed, 2 insertions(+)
meta-oe/recipes-extended/brotli/brotli_1.0.7.bb +2 -0
index e4e454bda..70dbcaffb 100644
@@ -18,3 +18,5 @@ do_install_append () {
 		mv -v "${lib}" "$(echo ${lib} | sed s/-static//)"
 	done
 }
+
+BBCLASSEXTEND = "native"
Signature · 1 line
2.23.0.rc1
#2 lore raw

PATCH 3/3 nodejs: allow use of system brotli (and make default)

In reply to André Draszik

From: André Draszik <[email protected]>

Use system brotli via PACKAGECONFIG by default. So far, nodejs had been built using its embedded copy of brotli, which we generally try to avoid, for the known reasons (independent updates, cve & license checks, etc).

The nodejs patches to enable this have been submitted. brotli is in meta-oe, so enabling this by default should not be a problem.

---
 ...-passing-multiple-libs-to-pkg_config.patch | 41 ++++++++++++
 ...allow-use-of-system-installed-brotli.patch | 66 +++++++++++++++++++
 .../recipes-devtools/nodejs/nodejs_12.14.1.bb |  7 +-
 3 files changed, 113 insertions(+), 1 deletion(-)
 create mode 100644 meta-oe/recipes-devtools/nodejs/nodejs/0001-build-allow-passing-multiple-libs-to-pkg_config.patch
 create mode 100644 meta-oe/recipes-devtools/nodejs/nodejs/0002-build-allow-use-of-system-installed-brotli.patch
meta-oe/recipes-devtools/nodejs/nodejs/0001-build-allow-passing-multiple-libs-to-pkg_config.patch new file +41 -0
new file mode 100644
index 000000000..13edf229b
@@ -0,0 +1,41 @@
+From fdaa0e3bef93c5c72a7258b5f1e30718e7d81f9b Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Andr=C3=A9=20Draszik?= <[email protected]>
+Date: Mon, 2 Mar 2020 12:17:09 +0000
+Subject: [PATCH 1/2] build: allow passing multiple libs to pkg_config
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Sometimes it's necessary to pass multiple library names to pkg-config,
+e.g. the brotli shared libraries can be pulled in with
+    pkg-config libbrotlienc libbrotlidec
+
+Update the code to handle both, strings (as used so far), and lists
+of strings.
+
+Signed-off-by: André Draszik <[email protected]>
+---
+Upstream-Status: Submitted [https://github.com/nodejs/node/pull/32046]
+ configure.py | 6 +++++-
+ 1 file changed, 5 insertions(+), 1 deletion(-)
+
+diff --git a/configure.py b/configure.py
+index beb08df088..e3f78f2fed 100755
+--- a/configure.py
++++ b/configure.py
+@@ -680,7 +680,11 @@ def pkg_config(pkg):
+   retval = ()
+   for flag in ['--libs-only-l', '--cflags-only-I',
+                '--libs-only-L', '--modversion']:
+-    args += [flag, pkg]
++    args += [flag]
++    if isinstance(pkg, list):
++      args += pkg
++    else:
++      args += [pkg]
+     try:
+       proc = subprocess.Popen(shlex.split(pkg_config) + args,
+                               stdout=subprocess.PIPE)
+-- 
+2.25.0
+
meta-oe/recipes-devtools/nodejs/nodejs/0002-build-allow-use-of-system-installed-brotli.patch new file +66 -0

66 lines not on this page · raw message

meta-oe/recipes-devtools/nodejs/nodejs_12.14.1.bb +6 -1
index 49bb71e28..1ea438c5b 100644
@@ -20,6 +20,8 @@ SRC_URI = "http://nodejs.org/dist/v${PV}/node-v${PV}.tar.xz \
            file://0003-Install-both-binaries-and-use-libdir.patch \
            file://0004-v8-don-t-override-ARM-CFLAGS.patch \
            file://big-endian.patch \
+           file://0001-build-allow-passing-multiple-libs-to-pkg_config.patch \
+           file://0002-build-allow-use-of-system-installed-brotli.patch \
            "
 SRC_URI_append_class-target = " \
            file://0002-Using-native-binaries.patch \
@@ -51,8 +53,9 @@ ARCHFLAGS_arm = "${@bb.utils.contains('TUNE_FEATURES', 'callconvention-hard', '-
 GYP_DEFINES_append_mipsel = " mips_arch_variant='r1' "
 ARCHFLAGS ?= ""
 
-PACKAGECONFIG ??= "ares icu libuv zlib"
+PACKAGECONFIG ??= "ares brotli icu libuv zlib"
 PACKAGECONFIG[ares] = "--shared-cares,,c-ares"
+PACKAGECONFIG[brotli] = "--shared-brotli,,brotli"
 PACKAGECONFIG[icu] = "--with-intl=system-icu,--without-intl,icu"
 PACKAGECONFIG[libuv] = "--shared-libuv,,libuv"
 PACKAGECONFIG[nghttp2] = "--shared-nghttp2,,nghttp2"
@@ -81,6 +84,8 @@ python do_unpack() {
     shutil.rmtree(d.getVar('S') + '/deps/openssl', True)
     if 'ares' in d.getVar('PACKAGECONFIG'):
         shutil.rmtree(d.getVar('S') + '/deps/cares', True)
+    if 'brotli' in d.getVar('PACKAGECONFIG'):
+        shutil.rmtree(d.getVar('S') + '/deps/brotli', True)
     if 'libuv' in d.getVar('PACKAGECONFIG'):
         shutil.rmtree(d.getVar('S') + '/deps/uv', True)
     if 'nghttp2' in d.getVar('PACKAGECONFIG'):
Signature · 1 line
2.23.0.rc1
#3 lore raw

Re: [meta-oe][PATCH 3/3] nodejs: allow use of system brotli (and make default)

can you rebase it on top of master-next and resend

Quoted 181 lines · André Draszik

From: André Draszik <[email protected]>

Use system brotli via PACKAGECONFIG by default. So far, nodejs had been built using its embedded copy of brotli, which we generally try to avoid, for the known reasons (independent updates, cve & license checks, etc).

The nodejs patches to enable this have been submitted. brotli is in meta-oe, so enabling this by default should not be a problem.

---
 ...-passing-multiple-libs-to-pkg_config.patch | 41 ++++++++++++
 ...allow-use-of-system-installed-brotli.patch | 66 +++++++++++++++++++
 .../recipes-devtools/nodejs/nodejs_12.14.1.bb |  7 +-
 3 files changed, 113 insertions(+), 1 deletion(-)
 create mode 100644 meta-oe/recipes-devtools/nodejs/nodejs/0001-build-allow-passing-multiple-libs-to-pkg_config.patch
 create mode 100644 meta-oe/recipes-devtools/nodejs/nodejs/0002-build-allow-use-of-system-installed-brotli.patch
meta-oe/recipes-devtools/nodejs/nodejs/0001-build-allow-passing-multiple-libs-to-pkg_config.patch new file +41 -0
new file mode 100644
index 000000000..13edf229b
@@ -0,0 +1,41 @@
+From fdaa0e3bef93c5c72a7258b5f1e30718e7d81f9b Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Andr=C3=A9=20Draszik?= <[email protected]>
+Date: Mon, 2 Mar 2020 12:17:09 +0000
+Subject: [PATCH 1/2] build: allow passing multiple libs to pkg_config
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Sometimes it's necessary to pass multiple library names to pkg-config,
+e.g. the brotli shared libraries can be pulled in with
+    pkg-config libbrotlienc libbrotlidec
+
+Update the code to handle both, strings (as used so far), and lists
+of strings.
+
+Signed-off-by: André Draszik <[email protected]>
+---
+Upstream-Status: Submitted [https://github.com/nodejs/node/pull/32046]
+ configure.py | 6 +++++-
+ 1 file changed, 5 insertions(+), 1 deletion(-)
+
+diff --git a/configure.py b/configure.py
+index beb08df088..e3f78f2fed 100755
+--- a/configure.py
++++ b/configure.py
+@@ -680,7 +680,11 @@ def pkg_config(pkg):
+   retval = ()
+   for flag in ['--libs-only-l', '--cflags-only-I',
+                '--libs-only-L', '--modversion']:
+-    args += [flag, pkg]
++    args += [flag]
++    if isinstance(pkg, list):
++      args += pkg
++    else:
++      args += [pkg]
+     try:
+       proc = subprocess.Popen(shlex.split(pkg_config) + args,
+                               stdout=subprocess.PIPE)
+--
+2.25.0
+
meta-oe/recipes-devtools/nodejs/nodejs/0002-build-allow-use-of-system-installed-brotli.patch new file +66 -0
new file mode 100644
index 000000000..fc038f3aa
@@ -0,0 +1,66 @@
+From f0f927feee8cb1fb173835d5c3f6beb6bf7d5e54 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Andr=C3=A9=20Draszik?= <[email protected]>
+Date: Mon, 2 Mar 2020 12:17:35 +0000
+Subject: [PATCH 2/2] build: allow use of system-installed brotli
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+brotli is available as a shared library since 2016, so it makes sense
+to allow its use as a system-installed version.
+
+Some of the infrastructure was in place already (node.gyp and
+node.gypi), but some bits in the configure script here were missing.
+
+Add them, keeping the default as before, to use the bundled version.
+
+Refs: https://github.com/google/brotli/pull/421
+Signed-off-by: André Draszik <[email protected]>
+---
+Upstream-Status: Submitted [https://github.com/nodejs/node/pull/32046]
+ configure.py | 22 ++++++++++++++++++++++
+ 1 file changed, 22 insertions(+)
+
+diff --git a/configure.py b/configure.py
+index e3f78f2fed..0190e31b41 100755
+--- a/configure.py
++++ b/configure.py
+@@ -301,6 +301,27 @@ shared_optgroup.add_option('--shared-zlib-libpath',
+     dest='shared_zlib_libpath',
+     help='a directory to search for the shared zlib DLL')
+
++shared_optgroup.add_option('--shared-brotli',
++    action='store_true',
++    dest='shared_brotli',
++    help='link to a shared brotli DLL instead of static linking')
++
++shared_optgroup.add_option('--shared-brotli-includes',
++    action='store',
++    dest='shared_brotli_includes',
++    help='directory containing brotli header files')
++
++shared_optgroup.add_option('--shared-brotli-libname',
++    action='store',
++    dest='shared_brotli_libname',
++    default='brotlidec,brotlienc',
++    help='alternative lib name to link to [default: %default]')
++
++shared_optgroup.add_option('--shared-brotli-libpath',
++    action='store',
++    dest='shared_brotli_libpath',
++    help='a directory to search for the shared brotli DLL')
++
+ shared_optgroup.add_option('--shared-cares',
+     action='store_true',
+     dest='shared_cares',
+@@ -1692,6 +1713,7 @@ configure_napi(output)
+ configure_library('zlib', output)
+ configure_library('http_parser', output)
+ configure_library('libuv', output)
++configure_library('brotli', output, pkgname=['libbrotlidec', 'libbrotlienc'])
+ configure_library('cares', output, pkgname='libcares')
+ configure_library('nghttp2', output, pkgname='libnghttp2')
+ configure_v8(output)
+--
+2.25.0
+
meta-oe/recipes-devtools/nodejs/nodejs_12.14.1.bb +6 -2
index 49bb71e28..1ea438c5b 100644
@@ -20,6 +20,8 @@ SRC_URI = "http://nodejs.org/dist/v${PV}/node-v${PV}.tar.xz \
            file://0003-Install-both-binaries-and-use-libdir.patch \
            file://0004-v8-don-t-override-ARM-CFLAGS.patch \
            file://big-endian.patch \
+           file://0001-build-allow-passing-multiple-libs-to-pkg_config.patch \
+           file://0002-build-allow-use-of-system-installed-brotli.patch \
            "
 SRC_URI_append_class-target = " \
            file://0002-Using-native-binaries.patch \
@@ -51,8 +53,9 @@ ARCHFLAGS_arm = "${@bb.utils.contains('TUNE_FEATURES', 'callconvention-hard', '-
 GYP_DEFINES_append_mipsel = " mips_arch_variant='r1' "
 ARCHFLAGS ?= ""
 
-PACKAGECONFIG ??= "ares icu libuv zlib"
+PACKAGECONFIG ??= "ares brotli icu libuv zlib"
 PACKAGECONFIG[ares] = "--shared-cares,,c-ares"
+PACKAGECONFIG[brotli] = "--shared-brotli,,brotli"
 PACKAGECONFIG[icu] = "--with-intl=system-icu,--without-intl,icu"
 PACKAGECONFIG[libuv] = "--shared-libuv,,libuv"
 PACKAGECONFIG[nghttp2] = "--shared-nghttp2,,nghttp2"
@@ -81,6 +84,8 @@ python do_unpack() {
     shutil.rmtree(d.getVar('S') + '/deps/openssl', True)
     if 'ares' in d.getVar('PACKAGECONFIG'):
         shutil.rmtree(d.getVar('S') + '/deps/cares', True)
+    if 'brotli' in d.getVar('PACKAGECONFIG'):
+        shutil.rmtree(d.getVar('S') + '/deps/brotli', True)
     if 'libuv' in d.getVar('PACKAGECONFIG'):
         shutil.rmtree(d.getVar('S') + '/deps/uv', True)
     if 'nghttp2' in d.getVar('PACKAGECONFIG'):
--

2.23.0.rc1

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

#4 lore raw
In reply to André Draszik

can you rebase it on top of master-next and resend

Quoted 75 lines · André Draszik

From: André Draszik <[email protected]>

During the python3 / nodejs update, the dependencies weren't updated, so
using system-gyp ends up trying to use the python2 version of system-
gyp, which will of course fail.
Fixing this to depend on the python3 version of gyp still doesn't
doesn't make things work, though:
ERROR: nodejs-native-12.14.1-r0 do_configure: Execution of '.../nodejs-native/12.14.1-r0/temp/run.do_configure.26054' failed with exit code 1:
gyp: Error importing pymod_do_mainmodule (ForEachFormat): No module named 'ForEachFormat' while loading dependencies of .../nodejs-native/12.14.1-r0/node-v12.14.1/node.gyp while trying to load .../nodejs-native/12.14.1-r0/node-v12.14.1/node.gyp Error running GYP

The reason is the following patch nodejs has applied to its version of gyp as of NodeJS v12 (commit fff922afee6e ("deps,build: compute torque_outputs in v8.gyp") upstream):

tools/gyp/pylib/gyp/input.py 2020-03-02 12:16:09.956707788 +0000 renamed from pylib/gyp/input.py 2020-03-02 12:36:30.788248197 +0000 +2 -0
@@ -890,6 +881,7 @@ def ExpandVariables(input, phase, variab
           oldwd = os.getcwd()  # Python doesn't like os.open('.'): no fchdir.
           if build_file_dir:  # build_file_dir may be None (see above).
             os.chdir(build_file_dir)
+          sys.path.append(os.getcwd())
           try:
 
             parsed_contents = shlex.split(contents)
@@ -900,6 +892,7 @@ def ExpandVariables(input, phase, variab
                              "module (%s): %s" % (parsed_contents[0], e))
             replacement = str(py_module.DoMain(parsed_contents[1:])).rstrip()
           finally:
+            sys.path.pop()
             os.chdir(oldwd)
           assert replacement != None
         elif command_string:

Since I'm not sure how to deal with that when using system-gyp, and because the original intention for using system-gyp was to make the previous nodejs version compatible with python3 by ultimately switching to the python3 version of system-gyp which isn't necessary anymore, and given nobody else seems to be using this PACKAGECONFIG, just drop it.

---
 meta-oe/recipes-devtools/nodejs/nodejs_12.14.1.bb | 3 ---
 1 file changed, 3 deletions(-)
meta-oe/recipes-devtools/nodejs/nodejs_12.14.1.bb +0 -4
index 6eb52c209..49bb71e28 100644
@@ -53,7 +53,6 @@ ARCHFLAGS ?= ""
 
 PACKAGECONFIG ??= "ares icu libuv zlib"
 PACKAGECONFIG[ares] = "--shared-cares,,c-ares"
-PACKAGECONFIG[gyp] = ",,gyp-py2-native"
 PACKAGECONFIG[icu] = "--with-intl=system-icu,--without-intl,icu"
 PACKAGECONFIG[libuv] = "--shared-libuv,,libuv"
 PACKAGECONFIG[nghttp2] = "--shared-nghttp2,,nghttp2"
@@ -82,8 +81,6 @@ python do_unpack() {
     shutil.rmtree(d.getVar('S') + '/deps/openssl', True)
     if 'ares' in d.getVar('PACKAGECONFIG'):
         shutil.rmtree(d.getVar('S') + '/deps/cares', True)
-    if 'gyp' in d.getVar('PACKAGECONFIG'):
-        shutil.rmtree(d.getVar('S') + '/tools/gyp', True)
     if 'libuv' in d.getVar('PACKAGECONFIG'):
         shutil.rmtree(d.getVar('S') + '/deps/uv', True)
     if 'nghttp2' in d.getVar('PACKAGECONFIG'):
--

2.23.0.rc1

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel