Skip to contentSkip to conversation

insane.bbclass: Make do_qa_staging check shebangs

PATCH In review Paulo Neves 8 participants 64 messages Jun 6, 2022 – Jul 5, 2022 2 messages not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 insane.bbclass: Make do_qa_staging check shebangs

As reported in the bug report [1], there was no check for shebang sizes on native scripts and now this is fixed.

The path scope of qa_staging needed to be increased from just checking libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
meta/classes/insane.bbclass +15 -1
index 9ca84bace9..dd0cf2a769 100644
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(os.path.join(sysroot_destdir, sysroot_dir), d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
 }
 
Signature · 1 line
2.25.1
#1 lore raw

PATCH 2/3 oeqa/selftest: Add test for shebang overflow

Make sure we do not stage any executable with a bigger shebang than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 .../sysroot-test/sysroot-shebang-test_1.0.bb         | 12 ++++++++++++
 meta/lib/oeqa/selftest/cases/sysroot.py              | 10 ++++++++++
 2 files changed, 22 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb new file +12 -0
new file mode 100644
index 0000000000..6c834be897
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+    chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
meta/lib/oeqa/selftest/cases/sysroot.py +10 -0
index 315d1a61c2..9457f1e3ac 100644
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
 TESTSTRING:pn-sysroot-test-arch2 = "%s"
 """ % (uuid1, uuid2))
         bitbake("sysroot-test")
+
+    def test_sysroot_max_shebang(self):
+        """
+        Summary:   Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+        Expected:  Fail when a shebang bigger than the max shebang-size is reached.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+        res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
Signature · 1 line
2.25.1
#2 lore raw

PATCH 3/3 oeqa/selftest: Test staged .la and .pc files

In reply to Paulo Neves

These files are checked by qa_check_staged but there was no test cases for whether the tests actually worked. Now there are.

---
 .../sysroot-test/sysroot-la-test_1.0.bb       | 16 ++++++++++
 .../sysroot-test/sysroot-pc-test_1.0.bb       | 12 +++++++
 meta/lib/oeqa/selftest/cases/sysroot.py       | 32 +++++++++++++++++++
 3 files changed, 60 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb new file +16 -0
new file mode 100644
index 0000000000..21f06782fb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb new file +12 -0
new file mode 100644
index 0000000000..e748310fc4
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
meta/lib/oeqa/selftest/cases/sysroot.py +32 -0
index 9457f1e3ac..588fc8c713 100644
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
         expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
         res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
         self.assertTrue(expected in res.output, msg=res.output)
+
+    def test_sysroot_la(self):
+        """
+        Summary:   Check that workdir paths are not contained in .la files.
+        Expected:  Fail when a workdir path is found in the file content.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "la-test.la failed sanity test (workdir) in path"
+
+        res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+        res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+    def test_sysroot_pkgconfig(self):
+        """
+        Summary:   Check that tmpdir paths are not contained in .pc files.
+        Expected:  Fail when a tmpdir path is found in the file content.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "test.pc failed sanity test (tmpdir) in path"
+
+        res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+        self.assertTrue(expected in res.output, msg=res.output)
+
+        res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
Signature · 1 line
2.25.1
#3 lore raw

PATCH v2 1/5 python: Avoid shebang overflow on python-config.py

In reply to Paulo Neves

The native path may be too big, leading to shebang overflow. Just use the #!/usr/bin/env python3.

---
 ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
 .../recipes-devtools/python/python3_3.10.4.bb |  1 +
 2 files changed, 30 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +29 -0
new file mode 100644
index 0000000000..9f23278a30
@@ -0,0 +1,29 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <[email protected]>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Upstream-Status: Inappropriate [distribution]
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ 	@ # Substitution happens here, as the completely-expanded BINDIR
+ 	@ # is not available in configure
+ 	sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++	@ # Otherwise we might get huge shebangs with native paths
++	sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ 	@ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ 	LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ 	@  # In OpenEmbedded, always use the python version of the script, the shell
+-- 
+2.25.1
+
meta/recipes-devtools/python/python3_3.10.4.bb +1 -0
index 6bd3a6aba8..357025f856 100644
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
            file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
            file://deterministic_imports.patch \
+           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
            "
 
 SRC_URI:append:class-native = " \
Signature · 1 line
2.25.1
#4 lore raw

PATCH v2 2/5 gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2

A native path can cause a shebang overflow on gtkdoc-mkhtml. Replace it with /usr/bin/env.

---
 meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb +1 -0
index 150d2c0b23..392913fcc6 100644
@@ -43,6 +43,7 @@ do_install:append () {
         ${datadir}/gtk-doc/python/gtkdoc/config.py; do
         sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
             -e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+            -e '1s|^#!.*|#!/usr/bin/env python3|' \
             -i ${D}$fn
     done
 }
Signature · 1 line
2.25.1
#5 lore raw

PATCH v2 3/5 insane.bbclass: Make do_qa_staging check shebangs

In reply to Paulo Neves

As reported in the bug report [1], there was no check for shebang sizes on native scripts and now this is fixed.

The path scope of the qa_staging was increased from just checking libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
meta/classes/insane.bbclass +15 -1
index 9ca84bace9..b2951a48fe 100644
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(sysroot_destdir + sysroot_dir, d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
 }
 
Signature · 1 line
2.25.1
#6 lore raw

PATCH v2 4/5 oeqa/selftest: Add test for shebang overflow

In reply to Paulo Neves

Make sure we do not stage any executable with a bigger shebang than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 .../sysroot-test/sysroot-shebang-test_1.0.bb         | 12 ++++++++++++
 meta/lib/oeqa/selftest/cases/sysroot.py              | 10 ++++++++++
 2 files changed, 22 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb new file +12 -0
new file mode 100644
index 0000000000..6c834be897
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+    chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
meta/lib/oeqa/selftest/cases/sysroot.py +10 -0
index 315d1a61c2..9457f1e3ac 100644
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
 TESTSTRING:pn-sysroot-test-arch2 = "%s"
 """ % (uuid1, uuid2))
         bitbake("sysroot-test")
+
+    def test_sysroot_max_shebang(self):
+        """
+        Summary:   Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+        Expected:  Fail when a shebang bigger than the max shebang-size is reached.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+        res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
Signature · 1 line
2.25.1
#7 lore raw

Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow

Hi Paulo,

Quoted 6 lines · On Tue, 7 Jun 2022 17:11:22 +0200 "Paulo Neves" <[email protected]> wrote:

Make sure we do not stage any executable with a bigger shebang than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

This check seems to be working very well!! It triggered a huge amount of build failures on the autobuilders due to libcheck having a shebang too long in the checkmk script, e.g.:

#! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f

Here are a few logs:

https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors

It would be great if you could add another patch to your series to fix libcheck, and also to do 'bitbake world' to test as many packages as possible before discovering from the autobuilders.

Thank you!

Signature · 3 lines
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#8 lore raw

Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow

Quoted 15 lines · Luca Ceresoli

Hi Paulo,

Quoted 6 lines · On Tue, 7 Jun 2022 17:11:22 +0200 "Paulo Neves" <[email protected]> wrote:

Make sure we do not stage any executable with a bigger shebang than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

This check seems to be working very well!! It triggered a huge amount of build failures on the autobuilders due to libcheck having a shebang too long in the checkmk script, e.g.:

Quoted 8 lines

#! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f

Here are a few logs:

https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors

I am not seeing an immediate way to fix this optimally. The go-to solution to this class of issues is to just make
the #!/usr/bin/env <interpreter>. The issue is that there is an extra -f, which with /usr/bin/env, will not work. The awk manual also implies this is not optional for standalone scripts. I think we can create a wrapper, or maybe we already have such a wrapper?

Quoted 4 lines

It would be great if you could add another patch to your series to fix libcheck, and also to do 'bitbake world' to test as many packages as possible before discovering from the autobuilders.

It takes quite a while on my computer and often i get out of disk space. I tried asking the linaro guys for a tuxsuite token but no answer yet. If you have some way of getting resources to make builds let me know.

Quoted 1 line

Thank you!

Thank you
Paulo Neves

#9 lore raw

Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow

Quoted 32 lines · On Wed, 8 Jun 2022 16:45:20 +0200 "Paulo Neves" <[email protected]> wrote:
Quoted 15 lines · Luca Ceresoli

Hi Paulo,

Quoted 6 lines · On Tue, 7 Jun 2022 17:11:22 +0200 "Paulo Neves" <[email protected]> wrote:

Make sure we do not stage any executable with a bigger shebang than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

This check seems to be working very well!! It triggered a huge amount of build failures on the autobuilders due to libcheck having a shebang too long in the checkmk script, e.g.:

Quoted 8 lines

#! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f

Here are a few logs:

https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors

I am not seeing an immediate way to fix this optimally. The go-to solution to this class of issues is to just make
the #!/usr/bin/env <interpreter>. The issue is that there is an extra -f, which with /usr/bin/env, will not work. The awk manual also implies this is not optional for standalone scripts. I think we can create a wrapper, or maybe we already have such a wrapper?

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

It would be great if you could add another patch to your series to fix libcheck, and also to do 'bitbake world' to test as many packages as possible before discovering from the autobuilders.

It takes quite a while on my computer and often i get out of disk space. I tried asking the linaro guys for a tuxsuite token but no answer yet. If you have some way of getting resources to make builds let me know.

Did you enable rm_work?

Signature · 3 lines
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#11 lore raw

Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow

Hi Paulo,

Quoted 20 lines · On Wed, 8 Jun 2022 14:53:05 +0200 "Luca Ceresoli via lists.openembedded.org"…

Hi Paulo,

Quoted 6 lines · On Tue, 7 Jun 2022 17:11:22 +0200 "Paulo Neves" <[email protected]> wrote:

Make sure we do not stage any executable with a bigger shebang than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

This check seems to be working very well!! It triggered a huge amount of build failures on the autobuilders due to libcheck having a shebang too long in the checkmk script, e.g.:

#! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f

Here are a few logs:

Quoted 8 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors

It would be great if you could add another patch to your series to fix libcheck, and also to do 'bitbake world' to test as many packages as possible before discovering from the autobuilders.

Here are more failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/12/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/5340/steps/12/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/5340/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/13/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/13/logs/stdio

This time the error is:

stdio: ERROR: core-image-sato-1.0-r0 do_testsdk: The toolchain <...> is not built. Build it before running the tests: 'bitbake <image> -c populate_sdk' .

I'm not sure exactly how your code triggers such error, but it appeared when testing on the autobuilders with this patch series and disappeared when I removed only these 5 patches, thus it seems related.

Signature · 3 lines
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#12 lore raw

Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow

Hey Luca,

I sent a new patchset and from my tests it all should be good now.

Paulo Neves

Quoted 50 lines · Luca Ceresoli

Hi Paulo,

Quoted 19 lines · On Wed, 8 Jun 2022 14:53:05 +0200 "Luca Ceresoli via lists.openembedded.org"…

Hi Paulo,

Quoted 6 lines · On Tue, 7 Jun 2022 17:11:22 +0200 "Paulo Neves" <[email protected]> wrote:

Make sure we do not stage any executable with a bigger shebang than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

This check seems to be working very well!! It triggered a huge amount of build failures on the autobuilders due to libcheck having a shebang too long in the checkmk script, e.g.:

#! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f

Here are a few logs:

Quoted 8 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors

It would be great if you could add another patch to your series to fix libcheck, and also to do 'bitbake world' to test as many packages as possible before discovering from the autobuilders.

Here are more failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/12/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/5340/steps/12/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/5340/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/13/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/13/logs/stdio

This time the error is:

stdio: ERROR: core-image-sato-1.0-r0 do_testsdk: The toolchain <...> is not built. Build it before running the tests: 'bitbake <image> -c populate_sdk' .

I'm not sure exactly how your code triggers such error, but it appeared when testing on the autobuilders with this patch series and disappeared when I removed only these 5 patches, thus it seems related.

#13 lore raw

Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow

In reply to Paulo Neves

meta/lib/oeqa/selftest/cases/sysroot.py TESTSTRING:pn-sysroot-test-arch1 = "%s" · line 47

Quoted 3 lines
meta/lib/oeqa/selftest/cases/sysroot.py +3 -0
+        expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+        res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)

I err towards paranoia and would also check that res.status != 0.

Ross

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#14 lore raw

PATCH v2 5/5 oeqa/selftest: Test staged .la and .pc files

In reply to Paulo Neves

These files are checked by qa_check_staged but there was no test cases for whether the tests actually worked. Now there are.

---
 .../sysroot-test/sysroot-la-test_1.0.bb       | 16 ++++++++++
 .../sysroot-test/sysroot-pc-test_1.0.bb       | 12 +++++++
 meta/lib/oeqa/selftest/cases/sysroot.py       | 32 +++++++++++++++++++
 3 files changed, 60 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb new file +16 -0
new file mode 100644
index 0000000000..21f06782fb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb new file +12 -0
new file mode 100644
index 0000000000..e748310fc4
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
meta/lib/oeqa/selftest/cases/sysroot.py +32 -0
index 9457f1e3ac..588fc8c713 100644
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
         expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
         res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
         self.assertTrue(expected in res.output, msg=res.output)
+
+    def test_sysroot_la(self):
+        """
+        Summary:   Check that workdir paths are not contained in .la files.
+        Expected:  Fail when a workdir path is found in the file content.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "la-test.la failed sanity test (workdir) in path"
+
+        res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+        res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+    def test_sysroot_pkgconfig(self):
+        """
+        Summary:   Check that tmpdir paths are not contained in .pc files.
+        Expected:  Fail when a tmpdir path is found in the file content.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "test.pc failed sanity test (tmpdir) in path"
+
+        res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+        self.assertTrue(expected in res.output, msg=res.output)
+
+        res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
Signature · 1 line
2.25.1
#15 lore raw

Re: [OE-core] [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py

In reply to Paulo Neves

I'm not sure this is Inappropriate. Can you send this to upstream nevertheless?

Alex

Quoted 70 lines · Paulo Neves

The native path may be too big, leading to shebang overflow. Just use the #!/usr/bin/env python3.

---
 ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
 .../recipes-devtools/python/python3_3.10.4.bb |  1 +
 2 files changed, 30 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +29 -0
new file mode 100644
index 0000000000..9f23278a30
@@ -0,0 +1,29 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <[email protected]>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Upstream-Status: Inappropriate [distribution]
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+       @ # Substitution happens here, as the completely-expanded BINDIR
+       @ # is not available in configure
+       sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++      @ # Otherwise we might get huge shebangs with native paths
++      sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+       @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+       LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+       @  # In OpenEmbedded, always use the python version of the script, the shell
+--
+2.25.1
+
meta/recipes-devtools/python/python3_3.10.4.bb +1 -1
index 6bd3a6aba8..357025f856 100644
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
            file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
            file://deterministic_imports.patch \
+           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
            "
 
 SRC_URI:append:class-native = " \
--

2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#166670): https://lists.openembedded.org/g/openembedded-core/message/166670
Mute This Topic: https://lists.openembedded.org/mt/91602889/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#16 lore raw

Re: [OE-core] [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py

I think this patch will never be accepted because it hard-codes and depends on env. I recall that red hat for example does not accept the use of env, possibly for security reasons.

Paulo Neves

Quoted 95 lines · On Tue, Jun 7, 2022, 18:53 Alexander Kanavin <[email protected]> wrote:

I'm not sure this is Inappropriate. Can you send this to upstream nevertheless?

Alex

Quoted 10 lines · Paulo Neves

The native path may be too big, leading to shebang overflow. Just use the #!/usr/bin/env python3.

---
 ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
 .../recipes-devtools/python/python3_3.10.4.bb |  1 +
 2 files changed, 30 insertions(+)
 create mode 100644

meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

Quoted 2 lines

diff --git

a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

Quoted 4 lines

new file mode 100644
index 0000000000..9f23278a30
--- /dev/null
+++

b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

Quoted 6 lines
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +5 -0
@@ -0,0 +1,29 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <[email protected]>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+

meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch line 15

Quoted 10 lines
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +10 -0
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Upstream-Status: Inappropriate [distribution]
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index

f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644

meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

Quoted 3 lines
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +3 -0
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in

Misc/python-config.sh

meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

Quoted 3 lines
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +3 -0
+       @ # Substitution happens here, as the completely-expanded BINDIR
+       @ # is not available in configure
+       sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," <

$(srcdir)/Misc/python-config.in >python-config.py

meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

Quoted 3 lines
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +3 -0
++      @ # Otherwise we might get huge shebangs with native paths
++      sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+       @ # Replace makefile compat. variable references with shell

script compat. ones; $(VAR) -> ${VAR}

Review line 25

Quoted 1 line
(quoted diff) +1 -0
+       LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' <

Misc/python-config.sh >python-config

Review line 26

Quoted 1 line
(quoted diff) +1 -0
+       @  # In OpenEmbedded, always use the python version of the

script, the shell

Review

Quoted 4 lines
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +3 -0
+--
+2.25.1
+
(quoted diff) +0 -0

b/meta/recipes-devtools/python/python3_3.10.4.bb

meta/recipes-devtools/python/python3_3.10.4.bb SRC_URI = "

Quoted 4 lines

index 6bd3a6aba8..357025f856 100644

meta/recipes-devtools/python/python3_3.10.4.bb +0 -0
@@ -34,6 +34,7 @@ SRC_URI = "

http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \

Quoted 1 line

file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \

Quoted 1 line

file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \

Quoted 2 lines
            file://deterministic_imports.patch \

+ file://0001-Avoid-shebang-overflow-on-python-config.py.patch

\

Quoted 10 lines
            "

SRC_URI:append:class-native = " \
--
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#166670):

https://lists.openembedded.org/g/openembedded-core/message/166670

Quoted 3 lines

Mute This Topic: https://lists.openembedded.org/mt/91602889/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [

[email protected]]

Quoted 2 lines

-=-=-=-=-=-=-=-=-=-=-=-

#17 lore raw

Re: [OE-core] [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py

Please submit it nevertheless. If upstream says no, we'll just mark it as denied, but it’s not appropriate to declare it inappropriate a priori.

Alex

Quoted 105 lines · Paulo Neves

I think this patch will never be accepted because it hard-codes and depends on env. I recall that red hat for example does not accept the use of env, possibly for security reasons.

Paulo Neves

Quoted 95 lines · On Tue, Jun 7, 2022, 18:53 Alexander Kanavin <[email protected]> wrote:

I'm not sure this is Inappropriate. Can you send this to upstream nevertheless?

Alex

Quoted 10 lines · Paulo Neves

The native path may be too big, leading to shebang overflow. Just use the #!/usr/bin/env python3.

---
 ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
 .../recipes-devtools/python/python3_3.10.4.bb |  1 +
 2 files changed, 30 insertions(+)
 create mode 100644

meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

Quoted 2 lines

diff --git

a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

Quoted 4 lines

new file mode 100644
index 0000000000..9f23278a30
--- /dev/null
+++

b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

Quoted 6 lines
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +5 -0
@@ -0,0 +1,29 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <[email protected]>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+

meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch line 15

Quoted 10 lines
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +10 -0
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Upstream-Status: Inappropriate [distribution]
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index

f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644

meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

Quoted 3 lines
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +3 -0
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in

Misc/python-config.sh

meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

Quoted 3 lines
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +3 -0
+       @ # Substitution happens here, as the completely-expanded BINDIR
+       @ # is not available in configure
+       sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," <

$(srcdir)/Misc/python-config.in >python-config.py

meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

Quoted 3 lines
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +3 -0
++      @ # Otherwise we might get huge shebangs with native paths
++      sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+       @ # Replace makefile compat. variable references with shell

script compat. ones; $(VAR) -> ${VAR}

Review line 25

Quoted 1 line
(quoted diff) +1 -0
+       LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' <

Misc/python-config.sh >python-config

Review line 26

Quoted 1 line
(quoted diff) +1 -0
+       @  # In OpenEmbedded, always use the python version of the

script, the shell

Review

Quoted 4 lines
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +3 -0
+--
+2.25.1
+
(quoted diff) +0 -0

b/meta/recipes-devtools/python/python3_3.10.4.bb

meta/recipes-devtools/python/python3_3.10.4.bb SRC_URI = "

Quoted 4 lines

index 6bd3a6aba8..357025f856 100644

meta/recipes-devtools/python/python3_3.10.4.bb +0 -0
@@ -34,6 +34,7 @@ SRC_URI = "

http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \

Quoted 1 line

file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \

Quoted 1 line

file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \

Quoted 2 lines
            file://deterministic_imports.patch \

+

file://0001-Avoid-shebang-overflow-on-python-config.py.patch \

Quoted 10 lines
            "

SRC_URI:append:class-native = " \
--
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#166670):

https://lists.openembedded.org/g/openembedded-core/message/166670

Quoted 3 lines

Mute This Topic: https://lists.openembedded.org/mt/91602889/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [

[email protected]]

Quoted 2 lines

-=-=-=-=-=-=-=-=-=-=-=-

#18 lore raw

PATCH v3 1/7 python: Avoid shebang overflow on python-config.py

In reply to Paulo Neves

The native path may be too big, leading to shebang overflow. Just use the #!/usr/bin/env python3.

---
 ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
 .../recipes-devtools/python/python3_3.10.4.bb |  1 +
 2 files changed, 30 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +29 -0
new file mode 100644
index 0000000000..9f23278a30
@@ -0,0 +1,29 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <[email protected]>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Upstream-Status: Inappropriate [distribution]
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ 	@ # Substitution happens here, as the completely-expanded BINDIR
+ 	@ # is not available in configure
+ 	sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++	@ # Otherwise we might get huge shebangs with native paths
++	sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ 	@ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ 	LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ 	@  # In OpenEmbedded, always use the python version of the script, the shell
+-- 
+2.25.1
+
meta/recipes-devtools/python/python3_3.10.4.bb +1 -0
index 6bd3a6aba8..357025f856 100644
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
            file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
            file://deterministic_imports.patch \
+           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
            "
 
 SRC_URI:append:class-native = " \
Signature · 1 line
2.25.1
#19 lore raw

PATCH v3 2/7 gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2

A native path can cause a shebang overflow on gtkdoc-mkhtml. Replace it with /usr/bin/env.

---
 meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb +1 -0
index 150d2c0b23..392913fcc6 100644
@@ -43,6 +43,7 @@ do_install:append () {
         ${datadir}/gtk-doc/python/gtkdoc/config.py; do
         sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
             -e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+            -e '1s|^#!.*|#!/usr/bin/env python3|' \
             -i ${D}$fn
     done
 }
Signature · 1 line
2.25.1
#20 lore raw

PATCH v3 3/7 insane.bbclass: Make do_qa_staging check shebangs

In reply to Paulo Neves

As reported in the bug report [1], there was no check for shebang sizes on native scripts and now this is fixed.

The path scope of the qa_staging was increased from just checking libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
meta/classes/insane.bbclass +15 -1
index 9ca84bace9..b2951a48fe 100644
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(sysroot_destdir + sysroot_dir, d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
 }
 
Signature · 1 line
2.25.1
#21 lore raw

PATCH v3 4/7 oeqa/selftest: Add test for shebang overflow

In reply to Paulo Neves

Make sure we do not stage any executable with a bigger shebang than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 .../sysroot-test/sysroot-shebang-test_1.0.bb         | 12 ++++++++++++
 meta/lib/oeqa/selftest/cases/sysroot.py              | 10 ++++++++++
 2 files changed, 22 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb new file +12 -0
new file mode 100644
index 0000000000..6c834be897
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+    chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
meta/lib/oeqa/selftest/cases/sysroot.py +10 -0
index 315d1a61c2..9457f1e3ac 100644
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
 TESTSTRING:pn-sysroot-test-arch2 = "%s"
 """ % (uuid1, uuid2))
         bitbake("sysroot-test")
+
+    def test_sysroot_max_shebang(self):
+        """
+        Summary:   Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+        Expected:  Fail when a shebang bigger than the max shebang-size is reached.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+        res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
Signature · 1 line
2.25.1
#22 lore raw

PATCH v3 5/7 oeqa/selftest: Test staged .la and .pc files

In reply to Paulo Neves

These files are checked by qa_check_staged but there was no test cases for whether the tests actually worked. Now there are.

---
 .../sysroot-test/sysroot-la-test_1.0.bb       | 16 ++++++++++
 .../sysroot-test/sysroot-pc-test_1.0.bb       | 12 +++++++
 meta/lib/oeqa/selftest/cases/sysroot.py       | 32 +++++++++++++++++++
 3 files changed, 60 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb new file +16 -0
new file mode 100644
index 0000000000..21f06782fb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb new file +12 -0
new file mode 100644
index 0000000000..e748310fc4
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
meta/lib/oeqa/selftest/cases/sysroot.py +32 -0
index 9457f1e3ac..588fc8c713 100644
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
         expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
         res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
         self.assertTrue(expected in res.output, msg=res.output)
+
+    def test_sysroot_la(self):
+        """
+        Summary:   Check that workdir paths are not contained in .la files.
+        Expected:  Fail when a workdir path is found in the file content.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "la-test.la failed sanity test (workdir) in path"
+
+        res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+        res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+    def test_sysroot_pkgconfig(self):
+        """
+        Summary:   Check that tmpdir paths are not contained in .pc files.
+        Expected:  Fail when a tmpdir path is found in the file content.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "test.pc failed sanity test (tmpdir) in path"
+
+        res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+        self.assertTrue(expected in res.output, msg=res.output)
+
+        res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
Signature · 1 line
2.25.1
#23 lore raw

PATCH v3 6/7 utils: Add cmdline_shebang_wrapper util.

In reply to Paulo Neves

Useful to work around shebang relocation issues, where shebangs are too long or have arguments in them, thus preventing them from using the /usr/bin/env shebang.

---
 .../wrapper/cmdline-shebang-wrapper-test.bb   | 21 ++++++++++++
 .../recipes-test/wrapper/files/test.awk       |  2 ++
 meta/classes/utils.bbclass                    | 34 +++++++++++++++++++
 meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
 4 files changed, 68 insertions(+)
 create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
 create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
 create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb new file +21 -0
new file mode 100644
index 0000000000..302eea8901
@@ -0,0 +1,21 @@
+FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+    create_cmdline_shebang_wrapper ${D}${bindir}/test
+    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+        bbfatal "Wrapper is broken"
+    else
+        bbnote "Wrapper is good"
+    fi
+}
+
+BBCLASSEXTEND = "native"
meta-selftest/recipes-test/wrapper/files/test.awk new file +2 -0
new file mode 100644
index 0000000000..91429197b1
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
meta/classes/utils.bbclass +34 -0
index b4eb3d38ab..b617632d9f 100644
@@ -184,6 +184,40 @@ END
 	chmod +x $cmd
 }
 
+create_cmdline_shebang_wrapper () {
+	# Create a wrapper script where commandline options are needed
+	#
+	# These are useful to work around shebang relocation issues, where shebangs are too
+  # long or have arguments in them, thus preventing them from using the /usr/bin/env
+	# shebang
+	#
+	# Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+	cmd=$1
+	shift
+
+	echo "Generating wrapper script for $cmd"
+
+  # Strip #! and get remaining interpreter + arg
+  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
+  # strip the shebang from the real script as we do not want it to be usable anyway
+  tail -n +2 $cmd > $cmd.real
+	cmdname=$(basename $cmd)
+	dirname=$(dirname $cmd)
+	cmdoptions=$@
+	if [ "${base_prefix}" != "" ]; then
+		relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+		cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`
+	fi
+	cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
+END
+	chmod +x $cmd
+}
+
 create_wrapper () {
 	# Create a wrapper script where extra environment variables are needed
 	#
meta/lib/oeqa/selftest/cases/wrapper.py new file +11 -0
new file mode 100644
index 0000000000..6de63310c0
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+    def test_shebang_wrapper(self):
+        """
+        Summary:   Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
+        Expected:  Exit status to be 0.
+        Author:    Paulo Neves <[email protected]>
+        """
+        res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
Signature · 1 line
2.25.1
#24 lore raw

PATCH v3 7/7 libcheck: Fix too long shebang for native case.

In reply to Paulo Neves

It requires a shebang wrapper due to the fact that awk interpreter has an argument.

---
 meta/recipes-support/libcheck/libcheck_0.15.2.bb | 3 +++
 1 file changed, 3 insertions(+)
meta/recipes-support/libcheck/libcheck_0.15.2.bb +3 -0
index 188d689cc3..1393aa2a1c 100644
@@ -23,6 +23,9 @@ CACHED_CONFIGUREVARS += "ac_cv_path_AWK_PATH=${bindir}/gawk"
 
 RREPLACES:${PN} = "check (<= 0.9.5)"
 
+do_install:append:class-native() {
+    create_cmdline_shebang_wrapper ${D}${bindir}/checkmk
+}
 BBCLASSEXTEND = "native nativesdk"
 
 PACKAGES =+ "checkmk"
Signature · 1 line
2.25.1
#25 lore raw

Re: [OE-core] [PATCH v3 1/7] python: Avoid shebang overflow on python-config.py

In reply to Paulo Neves

As requested, please submit upstream.

Alex

Quoted 70 lines · Paulo Neves

The native path may be too big, leading to shebang overflow. Just use the #!/usr/bin/env python3.

---
 ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
 .../recipes-devtools/python/python3_3.10.4.bb |  1 +
 2 files changed, 30 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +29 -0
new file mode 100644
index 0000000000..9f23278a30
@@ -0,0 +1,29 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <[email protected]>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Upstream-Status: Inappropriate [distribution]
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+       @ # Substitution happens here, as the completely-expanded BINDIR
+       @ # is not available in configure
+       sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++      @ # Otherwise we might get huge shebangs with native paths
++      sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+       @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+       LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+       @  # In OpenEmbedded, always use the python version of the script, the shell
+--
+2.25.1
+
meta/recipes-devtools/python/python3_3.10.4.bb +1 -1
index 6bd3a6aba8..357025f856 100644
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
            file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
            file://deterministic_imports.patch \
+           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
            "
 
 SRC_URI:append:class-native = " \
--

2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#166813): https://lists.openembedded.org/g/openembedded-core/message/166813
Mute This Topic: https://lists.openembedded.org/mt/91678611/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#26 lore raw

Re: [OE-core] [PATCH v3 1/7] python: Avoid shebang overflow on python-config.py

As requested
https://github.com/python/cpython/pull/93760

Paulo Neves

Quoted 74 lines · Alexander Kanavin

As requested, please submit upstream.

Alex

Quoted 69 lines · Paulo Neves

The native path may be too big, leading to shebang overflow. Just use the #!/usr/bin/env python3.

---
  ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
  .../recipes-devtools/python/python3_3.10.4.bb |  1 +
  2 files changed, 30 insertions(+)

create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +29 -0
new file mode 100644
index 0000000000..9f23278a30
@@ -0,0 +1,29 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <[email protected]>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Upstream-Status: Inappropriate [distribution]
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+       @ # Substitution happens here, as the completely-expanded BINDIR
+       @ # is not available in configure
+       sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++      @ # Otherwise we might get huge shebangs with native paths
++      sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+       @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+       LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+       @  # In OpenEmbedded, always use the python version of the script, the shell
+--
+2.25.1
+
meta/recipes-devtools/python/python3_3.10.4.bb +1 -1
index 6bd3a6aba8..357025f856 100644
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
             file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
             file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
             file://deterministic_imports.patch \
+           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
             "
 
  SRC_URI:append:class-native = " \
--

2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#166813): https://lists.openembedded.org/g/openembedded-core/message/166813
Mute This Topic: https://lists.openembedded.org/mt/91678611/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#27 lore raw

Re: [OE-core] [PATCH v3 1/7] python: Avoid shebang overflow on python-config.py

Thanks, can you resend with updated Upstream-Status that includes the link?

Alex

Quoted 82 lines · Paulo Neves

As requested
https://github.com/python/cpython/pull/93760

Paulo Neves

Quoted 74 lines · Alexander Kanavin

As requested, please submit upstream.

Alex

Quoted 69 lines · Paulo Neves

The native path may be too big, leading to shebang overflow. Just use the #!/usr/bin/env python3.

---
  ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
  .../recipes-devtools/python/python3_3.10.4.bb |  1 +
  2 files changed, 30 insertions(+)

create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +29 -0
new file mode 100644
index 0000000000..9f23278a30
@@ -0,0 +1,29 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <[email protected]>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Upstream-Status: Inappropriate [distribution]
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+       @ # Substitution happens here, as the completely-expanded BINDIR
+       @ # is not available in configure
+       sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++      @ # Otherwise we might get huge shebangs with native paths
++      sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+       @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+       LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+       @  # In OpenEmbedded, always use the python version of the script, the shell
+--
+2.25.1
+
meta/recipes-devtools/python/python3_3.10.4.bb +1 -1
index 6bd3a6aba8..357025f856 100644
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
             file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
             file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
             file://deterministic_imports.patch \
+           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
             "
 
  SRC_URI:append:class-native = " \
--

2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#166813): https://lists.openembedded.org/g/openembedded-core/message/166813
Mute This Topic: https://lists.openembedded.org/mt/91678611/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#28 lore raw

PATCH v4 1/7 python: Avoid shebang overflow on python-config.py

In reply to Paulo Neves

The native path may be too big, leading to shebang overflow. Just use the #!/usr/bin/env python3.

---
 ...shebang-overflow-on-python-config.py.patch | 33 +++++++++++++++++++
 .../recipes-devtools/python/python3_3.10.4.bb |  1 +
 2 files changed, 34 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +33 -0
new file mode 100644
index 0000000000..921da8de7c
@@ -0,0 +1,33 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <[email protected]>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Denial reason: [1]
+
+Upstream-Status: Denied [distribution]
+
+[1] https://github.com/python/cpython/pull/93760#pullrequestreview-1005365737
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ 	@ # Substitution happens here, as the completely-expanded BINDIR
+ 	@ # is not available in configure
+ 	sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++	@ # Otherwise we might get huge shebangs with native paths
++	sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ 	@ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ 	LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ 	@  # In OpenEmbedded, always use the python version of the script, the shell
+-- 
+2.25.1
+
meta/recipes-devtools/python/python3_3.10.4.bb +1 -0
index 6bd3a6aba8..357025f856 100644
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
            file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
            file://deterministic_imports.patch \
+           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
            "
 
 SRC_URI:append:class-native = " \
Signature · 1 line
2.25.1
#29 lore raw

PATCH v4 2/7 gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2

A native path can cause a shebang overflow on gtkdoc-mkhtml. Replace it with /usr/bin/env.

---
 meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb +1 -0
index 150d2c0b23..392913fcc6 100644
@@ -43,6 +43,7 @@ do_install:append () {
         ${datadir}/gtk-doc/python/gtkdoc/config.py; do
         sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
             -e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+            -e '1s|^#!.*|#!/usr/bin/env python3|' \
             -i ${D}$fn
     done
 }
Signature · 1 line
2.25.1
#30 lore raw

PATCH v4 3/7 insane.bbclass: Make do_qa_staging check shebangs

In reply to Paulo Neves

As reported in the bug report [1], there was no check for shebang sizes on native scripts and now this is fixed.

The path scope of the qa_staging was increased from just checking libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
meta/classes/insane.bbclass +15 -1
index 9ca84bace9..b2951a48fe 100644
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(sysroot_destdir + sysroot_dir, d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
 }
 
Signature · 1 line
2.25.1
#31 lore raw

PATCH v4 4/7 oeqa/selftest: Add test for shebang overflow

In reply to Paulo Neves

Make sure we do not stage any executable with a bigger shebang than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 .../sysroot-test/sysroot-shebang-test_1.0.bb         | 12 ++++++++++++
 meta/lib/oeqa/selftest/cases/sysroot.py              | 10 ++++++++++
 2 files changed, 22 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb new file +12 -0
new file mode 100644
index 0000000000..6c834be897
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+    chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
meta/lib/oeqa/selftest/cases/sysroot.py +10 -0
index 315d1a61c2..9457f1e3ac 100644
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
 TESTSTRING:pn-sysroot-test-arch2 = "%s"
 """ % (uuid1, uuid2))
         bitbake("sysroot-test")
+
+    def test_sysroot_max_shebang(self):
+        """
+        Summary:   Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+        Expected:  Fail when a shebang bigger than the max shebang-size is reached.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+        res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
Signature · 1 line
2.25.1
#32 lore raw

PATCH v4 5/7 oeqa/selftest: Test staged .la and .pc files

In reply to Paulo Neves

These files are checked by qa_check_staged but there was no test cases for whether the tests actually worked. Now there are.

---
 .../sysroot-test/sysroot-la-test_1.0.bb       | 16 ++++++++++
 .../sysroot-test/sysroot-pc-test_1.0.bb       | 12 +++++++
 meta/lib/oeqa/selftest/cases/sysroot.py       | 32 +++++++++++++++++++
 3 files changed, 60 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb new file +16 -0
new file mode 100644
index 0000000000..21f06782fb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb new file +12 -0
new file mode 100644
index 0000000000..e748310fc4
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
meta/lib/oeqa/selftest/cases/sysroot.py +32 -0
index 9457f1e3ac..588fc8c713 100644
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
         expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
         res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
         self.assertTrue(expected in res.output, msg=res.output)
+
+    def test_sysroot_la(self):
+        """
+        Summary:   Check that workdir paths are not contained in .la files.
+        Expected:  Fail when a workdir path is found in the file content.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "la-test.la failed sanity test (workdir) in path"
+
+        res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+        res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+    def test_sysroot_pkgconfig(self):
+        """
+        Summary:   Check that tmpdir paths are not contained in .pc files.
+        Expected:  Fail when a tmpdir path is found in the file content.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "test.pc failed sanity test (tmpdir) in path"
+
+        res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+        self.assertTrue(expected in res.output, msg=res.output)
+
+        res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
Signature · 1 line
2.25.1
#33 lore raw

PATCH v4 6/7 utils: Add cmdline_shebang_wrapper util.

In reply to Paulo Neves

Useful to work around shebang relocation issues, where shebangs are too long or have arguments in them, thus preventing them from using the /usr/bin/env shebang.

---
 .../wrapper/cmdline-shebang-wrapper-test.bb   | 21 ++++++++++++
 .../recipes-test/wrapper/files/test.awk       |  2 ++
 meta/classes/utils.bbclass                    | 34 +++++++++++++++++++
 meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
 4 files changed, 68 insertions(+)
 create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
 create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
 create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb new file +21 -0
new file mode 100644
index 0000000000..302eea8901
@@ -0,0 +1,21 @@
+FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+    create_cmdline_shebang_wrapper ${D}${bindir}/test
+    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+        bbfatal "Wrapper is broken"
+    else
+        bbnote "Wrapper is good"
+    fi
+}
+
+BBCLASSEXTEND = "native"
meta-selftest/recipes-test/wrapper/files/test.awk new file +2 -0
new file mode 100644
index 0000000000..91429197b1
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
meta/classes/utils.bbclass +34 -0
index b4eb3d38ab..b617632d9f 100644
@@ -184,6 +184,40 @@ END
 	chmod +x $cmd
 }
 
+create_cmdline_shebang_wrapper () {
+	# Create a wrapper script where commandline options are needed
+	#
+	# These are useful to work around shebang relocation issues, where shebangs are too
+  # long or have arguments in them, thus preventing them from using the /usr/bin/env
+	# shebang
+	#
+	# Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+	cmd=$1
+	shift
+
+	echo "Generating wrapper script for $cmd"
+
+  # Strip #! and get remaining interpreter + arg
+  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
+  # strip the shebang from the real script as we do not want it to be usable anyway
+  tail -n +2 $cmd > $cmd.real
+	cmdname=$(basename $cmd)
+	dirname=$(dirname $cmd)
+	cmdoptions=$@
+	if [ "${base_prefix}" != "" ]; then
+		relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+		cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`
+	fi
+	cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
+END
+	chmod +x $cmd
+}
+
 create_wrapper () {
 	# Create a wrapper script where extra environment variables are needed
 	#
meta/lib/oeqa/selftest/cases/wrapper.py new file +11 -0
new file mode 100644
index 0000000000..6de63310c0
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+    def test_shebang_wrapper(self):
+        """
+        Summary:   Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
+        Expected:  Exit status to be 0.
+        Author:    Paulo Neves <[email protected]>
+        """
+        res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
Signature · 1 line
2.25.1
#34 lore raw

Re: [OE-core] [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util.

FILESEXTRAPATHS:prepend doesn't seem to be needed and LICENSE is really CLOSED?

Quoted 139 lines · Paulo Neves

Useful to work around shebang relocation issues, where shebangs are too long or have arguments in them, thus preventing them from using the /usr/bin/env shebang.

---
 .../wrapper/cmdline-shebang-wrapper-test.bb   | 21 ++++++++++++
 .../recipes-test/wrapper/files/test.awk       |  2 ++
 meta/classes/utils.bbclass                    | 34 +++++++++++++++++++
 meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
 4 files changed, 68 insertions(+)
 create mode 100644 meta-selftest/recipes-test/wrapper/
cmdline-shebang-wrapper-test.bb
 create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
 create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
(quoted diff) +0 -0

cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/ cmdline-shebang-wrapper-test.bb
new file mode 100644
index 0000000000..302eea8901

meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb new file +21 -0
@@ -0,0 +1,21 @@
+FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+    create_cmdline_shebang_wrapper ${D}${bindir}/test
+    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+        bbfatal "Wrapper is broken"
+    else
+        bbnote "Wrapper is good"
+    fi
+}
+
+BBCLASSEXTEND = "native"
meta-selftest/recipes-test/wrapper/files/test.awk new file +2 -0
new file mode 100644
index 0000000000..91429197b1
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
meta/classes/utils.bbclass +4 -0
index b4eb3d38ab..b617632d9f 100644
@@ -184,6 +184,40 @@ END
        chmod +x $cmd
 }
 
+create_cmdline_shebang_wrapper () {
+       # Create a wrapper script where commandline options are needed
+       #
+       # These are useful to work around shebang relocation issues, where

shebangs are too
+ # long or have arguments in them, thus preventing them from using the /usr/bin/env

meta/classes/utils.bbclass +12 -0
+       # shebang
+       #
+       # Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+       cmd=$1
+       shift
+
+       echo "Generating wrapper script for $cmd"
+
+  # Strip #! and get remaining interpreter + arg
+  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
+  # strip the shebang from the real script as we do not want it to be

usable anyway

meta/classes/utils.bbclass +6 -0
+  tail -n +2 $cmd > $cmd.real
+       cmdname=$(basename $cmd)
+       dirname=$(dirname $cmd)
+       cmdoptions=$@
+       if [ "${base_prefix}" != "" ]; then
+               relpath=`python3 -c "import os;

print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+ cmdoptions=`echo $@ | sed -e
"s:${base_prefix}:\\$realdir/$relpath:g"`

meta/classes/utils.bbclass +6 -0
+       fi
+       cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions

"\$@"

meta/classes/utils.bbclass +4 -0
+END
+       chmod +x $cmd
+}
+
 create_wrapper () {
        # Create a wrapper script where extra environment variables are

needed

        #
meta/lib/oeqa/selftest/cases/wrapper.py new file +7 -0
new file mode 100644
index 0000000000..6de63310c0
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+    def test_shebang_wrapper(self):
+        """
+        Summary:   Build a recipe which will fail if the

cmdline_shebang_wrapper function is defective.

meta/lib/oeqa/selftest/cases/wrapper.py new file +4 -0
+        Expected:  Exit status to be 0.
+        Author:    Paulo Neves <[email protected]>
+        """
+        res = bitbake("cmdline-shebang-wrapper-test -c install",

ignore_status=False)
--
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#166887):
https://lists.openembedded.org/g/openembedded-core/message/166887
Mute This Topic: https://lists.openembedded.org/mt/91748692/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#35 lore raw

Re: [OE-core] [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util.

Regarding the FILESEXTRAPATHS:prepend  i was convinced that ${THISDIR}/files was not automatically added and was needed for the test.awk file. I guess I have been cargo culting for some time on this. Will remove it.

This is a recipe for a test and if i add a license different than CLOSED than i need to add a license file, which seems a bit overkill for a one line. Am I wrong? What license should it even be? Dont Panic is a common string used in awk docs.

Paulo Neves

Quoted 141 lines · Martin Jansa

FILESEXTRAPATHS:prepend doesn't seem to be needed and LICENSE is really CLOSED?

On Tue, Jun 14, 2022 at 3:17 PM Paulo Neves <[email protected]> wrote:

    Useful to work around shebang relocation issues, where
    shebangs are too long or have arguments in them, thus preventing them
    from using the /usr/bin/env shebang.
    ---
     .../wrapper/cmdline-shebang-wrapper-test.bb
    <http://cmdline-shebang-wrapper-test.bb>  | 21 ++++++++++++
     .../recipes-test/wrapper/files/test.awk       |  2 ++
     meta/classes/utils.bbclass                    | 34
    +++++++++++++++++++
     meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
     4 files changed, 68 insertions(+)
     create mode 100644
    meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
    <http://cmdline-shebang-wrapper-test.bb>
     create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
     create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py

    diff --git
    a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
    <http://cmdline-shebang-wrapper-test.bb>
    b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
    <http://cmdline-shebang-wrapper-test.bb>
    new file mode 100644
    index 0000000000..302eea8901
    --- /dev/null
    +++
    b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
    <http://cmdline-shebang-wrapper-test.bb>
    @@ -0,0 +1,21 @@
    +FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
    +SUMMARY = "Check that create_cmdline_shebang works"
    +LICENSE = "CLOSED"
    +INHIBIT_DEFAULT_DEPS = "1"
    +
    +SRC_URI += "file://test.awk"
    +
    +EXCLUDE_FROM_WORLD = "1"
    +do_install() {
    +    install -d ${D}${bindir}
    +    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
    +    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
    +    create_cmdline_shebang_wrapper ${D}${bindir}/test
    +    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
    +        bbfatal "Wrapper is broken"
    +    else
    +        bbnote "Wrapper is good"
    +    fi
    +}
    +
    +BBCLASSEXTEND = "native"
    diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk
    b/meta-selftest/recipes-test/wrapper/files/test.awk
    new file mode 100644
    index 0000000000..91429197b1
    --- /dev/null
    +++ b/meta-selftest/recipes-test/wrapper/files/test.awk
    @@ -0,0 +1,2 @@
    +#! @AWK_BIN@ -f
    +BEGIN { print "Don't Panic!" }
    diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
    index b4eb3d38ab..b617632d9f 100644
    --- a/meta/classes/utils.bbclass
    +++ b/meta/classes/utils.bbclass
    @@ -184,6 +184,40 @@ END
            chmod +x $cmd
     }

    +create_cmdline_shebang_wrapper () {
    +       # Create a wrapper script where commandline options are needed
    +       #
    +       # These are useful to work around shebang relocation
    issues, where shebangs are too
    +  # long or have arguments in them, thus preventing them from
    using the /usr/bin/env
    +       # shebang
    +       #
    +       # Usage: create_cmdline_wrapper FILENAME <extra-options>
    +
    +       cmd=$1
    +       shift
    +
    +       echo "Generating wrapper script for $cmd"
    +
    +  # Strip #! and get remaining interpreter + arg
    +  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
    +  # strip the shebang from the real script as we do not want it
    to be usable anyway
    +  tail -n +2 $cmd > $cmd.real
    +       cmdname=$(basename $cmd)
    +       dirname=$(dirname $cmd)
    +       cmdoptions=$@
    +       if [ "${base_prefix}" != "" ]; then
    +               relpath=`python3 -c "import os;
    print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
    +               cmdoptions=`echo $@ | sed -e
    "s:${base_prefix}:\\$realdir/$relpath:g"`
    +       fi
    +       cat <<END >$cmd
    +#!/usr/bin/env bash
    +realpath=\`readlink -fn \$0\`
    +realdir=\`dirname \$realpath\`
    +exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real
    $cmdoptions "\$@"
    +END
    +       chmod +x $cmd
    +}
    +
     create_wrapper () {
            # Create a wrapper script where extra environment
    variables are needed
            #
    diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py
    b/meta/lib/oeqa/selftest/cases/wrapper.py
    new file mode 100644
    index 0000000000..6de63310c0
    --- /dev/null
    +++ b/meta/lib/oeqa/selftest/cases/wrapper.py
    @@ -0,0 +1,11 @@
    +from oeqa.selftest.case import OESelftestTestCase
    +from oeqa.utils.commands import bitbake
    +
    +class WrapperTests(OESelftestTestCase):
    +    def test_shebang_wrapper(self):
    +        """
    +        Summary:   Build a recipe which will fail if the
    cmdline_shebang_wrapper function is defective.
    +        Expected:  Exit status to be 0.
    +        Author:    Paulo Neves <[email protected]>
    +        """
    +        res = bitbake("cmdline-shebang-wrapper-test -c install",
    ignore_status=False)
    -- 
    2.25.1
#36 lore raw

Re: [OE-core] [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util.

You can use bitbake-getvar to see the default FILESPATH, but "files" next to recipe is included by default (I prefer to use BPN directory for slightly faster lookup - unless the files are shared between different recipes). FILESEXTRAPATHS is usually only needed from bbappends which add new files to SRC_URI.

For LICENSE you can use MIT and
LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420" like many "pure-metadata" recipes in oe-core.

Quoted 159 lines · Paulo Neves

Regarding the FILESEXTRAPATHS:prepend i was convinced that ${THISDIR}/files was not automatically added and was needed for the test.awk file. I guess I have been cargo culting for some time on this. Will remove it.

This is a recipe for a test and if i add a license different than CLOSED than i need to add a license file, which seems a bit overkill for a one line. Am I wrong? What license should it even be? Dont Panic is a common string used in awk docs.

Paulo Neves

On 6/14/22 15:39, Martin Jansa wrote:

FILESEXTRAPATHS:prepend doesn't seem to be needed and LICENSE is really CLOSED?

Quoted 139 lines · Paulo Neves

Useful to work around shebang relocation issues, where shebangs are too long or have arguments in them, thus preventing them from using the /usr/bin/env shebang.

---
 .../wrapper/cmdline-shebang-wrapper-test.bb   | 21 ++++++++++++
 .../recipes-test/wrapper/files/test.awk       |  2 ++
 meta/classes/utils.bbclass                    | 34 +++++++++++++++++++
 meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
 4 files changed, 68 insertions(+)
 create mode 100644 meta-selftest/recipes-test/wrapper/
cmdline-shebang-wrapper-test.bb
 create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
 create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
(quoted diff) +0 -0

cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/ cmdline-shebang-wrapper-test.bb
new file mode 100644
index 0000000000..302eea8901

meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb new file +21 -0
@@ -0,0 +1,21 @@
+FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+    create_cmdline_shebang_wrapper ${D}${bindir}/test
+    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+        bbfatal "Wrapper is broken"
+    else
+        bbnote "Wrapper is good"
+    fi
+}
+
+BBCLASSEXTEND = "native"
meta-selftest/recipes-test/wrapper/files/test.awk new file +2 -0
new file mode 100644
index 0000000000..91429197b1
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
meta/classes/utils.bbclass +4 -0
index b4eb3d38ab..b617632d9f 100644
@@ -184,6 +184,40 @@ END
        chmod +x $cmd
 }
 
+create_cmdline_shebang_wrapper () {
+       # Create a wrapper script where commandline options are needed
+       #
+       # These are useful to work around shebang relocation issues,

where shebangs are too
+ # long or have arguments in them, thus preventing them from using the /usr/bin/env

meta/classes/utils.bbclass +12 -0
+       # shebang
+       #
+       # Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+       cmd=$1
+       shift
+
+       echo "Generating wrapper script for $cmd"
+
+  # Strip #! and get remaining interpreter + arg
+  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
+  # strip the shebang from the real script as we do not want it to be

usable anyway

meta/classes/utils.bbclass +6 -0
+  tail -n +2 $cmd > $cmd.real
+       cmdname=$(basename $cmd)
+       dirname=$(dirname $cmd)
+       cmdoptions=$@
+       if [ "${base_prefix}" != "" ]; then
+               relpath=`python3 -c "import os;

print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+ cmdoptions=`echo $@ | sed -e
"s:${base_prefix}:\\$realdir/$relpath:g"`

meta/classes/utils.bbclass +6 -0
+       fi
+       cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions

"\$@"

meta/classes/utils.bbclass +4 -0
+END
+       chmod +x $cmd
+}
+
 create_wrapper () {
        # Create a wrapper script where extra environment variables are

needed

        #
meta/lib/oeqa/selftest/cases/wrapper.py new file +7 -0
new file mode 100644
index 0000000000..6de63310c0
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+    def test_shebang_wrapper(self):
+        """
+        Summary:   Build a recipe which will fail if the

cmdline_shebang_wrapper function is defective.

meta/lib/oeqa/selftest/cases/wrapper.py new file +4 -0
+        Expected:  Exit status to be 0.
+        Author:    Paulo Neves <[email protected]>
+        """
+        res = bitbake("cmdline-shebang-wrapper-test -c install",

ignore_status=False)
--
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#166887):
https://lists.openembedded.org/g/openembedded-core/message/166887
Mute This Topic: https://lists.openembedded.org/mt/91748692/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#37 lore raw

Re: [OE-core] [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util.

Thanks for the great tips. Yeah the habit of using the FILESPATH is that mostly I work on bbappends. Today i learned.

I sent a new v5 with the corrections you mention.

Paulo Neves

Quoted 171 lines · Martin Jansa

You can use bitbake-getvar to see the default FILESPATH, but "files" next to recipe is included by default (I prefer to use BPN directory for slightly faster lookup - unless the files are shared between different recipes). FILESEXTRAPATHS is usually only needed from bbappends which add new files to SRC_URI.

For LICENSE you can use MIT and
LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420" like many "pure-metadata" recipes in oe-core.

On Tue, Jun 14, 2022 at 3:46 PM Paulo Neves <[email protected]> wrote:

Regarding the FILESEXTRAPATHS:prepend  i was convinced that ${THISDIR}/files was not automatically added and was needed for the test.awk file. I guess I have been cargo culting for some time on this. Will remove it.

This is a recipe for a test and if i add a license different than CLOSED than i need to add a license file, which seems a bit overkill for a one line. Am I wrong? What license should it even be? Dont Panic is a common string used in awk docs.

Paulo Neves

Quoted 149 lines · Martin Jansa
    FILESEXTRAPATHS:prepend doesn't seem to be needed and LICENSE is
    really CLOSED?

    On Tue, Jun 14, 2022 at 3:17 PM Paulo Neves <[email protected]>
    wrote:

        Useful to work around shebang relocation issues, where
        shebangs are too long or have arguments in them, thus
        preventing them
        from using the /usr/bin/env shebang.
        ---
         .../wrapper/cmdline-shebang-wrapper-test.bb
        <http://cmdline-shebang-wrapper-test.bb>  | 21 ++++++++++++
         .../recipes-test/wrapper/files/test.awk       |  2 ++
         meta/classes/utils.bbclass                    | 34
        +++++++++++++++++++
         meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
         4 files changed, 68 insertions(+)
         create mode 100644
        meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
        <http://cmdline-shebang-wrapper-test.bb>
         create mode 100644
        meta-selftest/recipes-test/wrapper/files/test.awk
         create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py

        diff --git
        a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
        <http://cmdline-shebang-wrapper-test.bb>
        b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
        <http://cmdline-shebang-wrapper-test.bb>
        new file mode 100644
        index 0000000000..302eea8901
        --- /dev/null
        +++
        b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
        <http://cmdline-shebang-wrapper-test.bb>
        @@ -0,0 +1,21 @@
        +FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
        +SUMMARY = "Check that create_cmdline_shebang works"
        +LICENSE = "CLOSED"
        +INHIBIT_DEFAULT_DEPS = "1"
        +
        +SRC_URI += "file://test.awk"
        +
        +EXCLUDE_FROM_WORLD = "1"
        +do_install() {
        +    install -d ${D}${bindir}
        +    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
        +    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
        +    create_cmdline_shebang_wrapper ${D}${bindir}/test
        +    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
        +        bbfatal "Wrapper is broken"
        +    else
        +        bbnote "Wrapper is good"
        +    fi
        +}
        +
        +BBCLASSEXTEND = "native"
        diff --git
        a/meta-selftest/recipes-test/wrapper/files/test.awk
        b/meta-selftest/recipes-test/wrapper/files/test.awk
        new file mode 100644
        index 0000000000..91429197b1
        --- /dev/null
        +++ b/meta-selftest/recipes-test/wrapper/files/test.awk
        @@ -0,0 +1,2 @@
        +#! @AWK_BIN@ -f
        +BEGIN { print "Don't Panic!" }
        diff --git a/meta/classes/utils.bbclass
        b/meta/classes/utils.bbclass
        index b4eb3d38ab..b617632d9f 100644
        --- a/meta/classes/utils.bbclass
        +++ b/meta/classes/utils.bbclass
        @@ -184,6 +184,40 @@ END
                chmod +x $cmd
         }

        +create_cmdline_shebang_wrapper () {
        +       # Create a wrapper script where commandline options
        are needed
        +       #
        +       # These are useful to work around shebang relocation
        issues, where shebangs are too
        +  # long or have arguments in them, thus preventing them
        from using the /usr/bin/env
        +       # shebang
        +       #
        +       # Usage: create_cmdline_wrapper FILENAME <extra-options>
        +
        +       cmd=$1
        +       shift
        +
        +       echo "Generating wrapper script for $cmd"
        +
        +  # Strip #! and get remaining interpreter + arg
        +  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![
        ]*||g' )")"
        +  # strip the shebang from the real script as we do not want
        it to be usable anyway
        +  tail -n +2 $cmd > $cmd.real
        +       cmdname=$(basename $cmd)
        +       dirname=$(dirname $cmd)
        +       cmdoptions=$@
        +       if [ "${base_prefix}" != "" ]; then
        +               relpath=`python3 -c "import os;
        print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
        +               cmdoptions=`echo $@ | sed -e
        "s:${base_prefix}:\\$realdir/$relpath:g"`
        +       fi
        +       cat <<END >$cmd
        +#!/usr/bin/env bash
        +realpath=\`readlink -fn \$0\`
        +realdir=\`dirname \$realpath\`
        +exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real
        $cmdoptions "\$@"
        +END
        +       chmod +x $cmd
        +}
        +
         create_wrapper () {
                # Create a wrapper script where extra environment
        variables are needed
                #
        diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py
        b/meta/lib/oeqa/selftest/cases/wrapper.py
        new file mode 100644
        index 0000000000..6de63310c0
        --- /dev/null
        +++ b/meta/lib/oeqa/selftest/cases/wrapper.py
        @@ -0,0 +1,11 @@
        +from oeqa.selftest.case import OESelftestTestCase
        +from oeqa.utils.commands import bitbake
        +
        +class WrapperTests(OESelftestTestCase):
        +    def test_shebang_wrapper(self):
        +        """
        +        Summary:   Build a recipe which will fail if the
        cmdline_shebang_wrapper function is defective.
        +        Expected:  Exit status to be 0.
        +        Author:    Paulo Neves <[email protected]>
        +        """
        +        res = bitbake("cmdline-shebang-wrapper-test -c
        install", ignore_status=False)
        -- 
        2.25.1
#38 lore raw

PATCH v4 7/7 libcheck: Fix too long shebang for native case.

In reply to Paulo Neves

It requires a shebang wrapper due to the fact that awk interpreter has an argument.

---
 meta/recipes-support/libcheck/libcheck_0.15.2.bb | 3 +++
 1 file changed, 3 insertions(+)
meta/recipes-support/libcheck/libcheck_0.15.2.bb +3 -0
index 188d689cc3..1393aa2a1c 100644
@@ -23,6 +23,9 @@ CACHED_CONFIGUREVARS += "ac_cv_path_AWK_PATH=${bindir}/gawk"
 
 RREPLACES:${PN} = "check (<= 0.9.5)"
 
+do_install:append:class-native() {
+    create_cmdline_shebang_wrapper ${D}${bindir}/checkmk
+}
 BBCLASSEXTEND = "native nativesdk"
 
 PACKAGES =+ "checkmk"
Signature · 1 line
2.25.1
#39 lore raw

PATCH v5 1/7 python: Avoid shebang overflow on python-config.py

In reply to Paulo Neves

The native path may be too big, leading to shebang overflow. Just use the #!/usr/bin/env python3.

---
 ...shebang-overflow-on-python-config.py.patch | 33 +++++++++++++++++++
 .../recipes-devtools/python/python3_3.10.4.bb |  1 +
 2 files changed, 34 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +33 -0
new file mode 100644
index 0000000000..921da8de7c
@@ -0,0 +1,33 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <[email protected]>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Denial reason: [1]
+
+Upstream-Status: Denied [distribution]
+
+[1] https://github.com/python/cpython/pull/93760#pullrequestreview-1005365737
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ 	@ # Substitution happens here, as the completely-expanded BINDIR
+ 	@ # is not available in configure
+ 	sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++	@ # Otherwise we might get huge shebangs with native paths
++	sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ 	@ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ 	LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ 	@  # In OpenEmbedded, always use the python version of the script, the shell
+-- 
+2.25.1
+
meta/recipes-devtools/python/python3_3.10.4.bb +1 -0
index 6bd3a6aba8..357025f856 100644
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
            file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
            file://deterministic_imports.patch \
+           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
            "
 
 SRC_URI:append:class-native = " \
Signature · 1 line
2.25.1
#40 lore raw

PATCH v5 2/7 gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2

A native path can cause a shebang overflow on gtkdoc-mkhtml. Replace it with /usr/bin/env.

---
 meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb +1 -0
index 150d2c0b23..392913fcc6 100644
@@ -43,6 +43,7 @@ do_install:append () {
         ${datadir}/gtk-doc/python/gtkdoc/config.py; do
         sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
             -e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+            -e '1s|^#!.*|#!/usr/bin/env python3|' \
             -i ${D}$fn
     done
 }
Signature · 1 line
2.25.1
#41 lore raw

PATCH v5 3/7 insane.bbclass: Make do_qa_staging check shebangs

In reply to Paulo Neves

As reported in the bug report [1], there was no check for shebang sizes on native scripts and now this is fixed.

The path scope of the qa_staging was increased from just checking libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
meta/classes/insane.bbclass +15 -1
index 9ca84bace9..b2951a48fe 100644
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(sysroot_destdir + sysroot_dir, d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
 }
 
Signature · 1 line
2.25.1
#42 lore raw
Quoted 46 lines · Paulo Neves

As reported in the bug report [1], there was no check for shebang sizes on native scripts and now this is fixed.

The path scope of the qa_staging was increased from just checking libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
meta/classes/insane.bbclass +12 -0
index 9ca84bace9..b2951a48fe 100644
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}

meta/classes/insane.bbclass addtask do_package_qa_setscene · line 1158

Quoted 9 lines
meta/classes/insane.bbclass +3 -1
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(sysroot_destdir + sysroot_dir, d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
 }

I'm a little worried about the performance implications of this, we're going from scanning files in libdir to scanning nearly all files in the sysroots, reading from many of them. In isolation that doesn't seem much but I suspect the IO will impact builds overall.

That leaves me a little torn on this change :/

Cheers,

Richard

#43 lore raw

Yes, when I was making the change I also realized the scope of the check was quite bigger than before. I am going to deliver the final changes requested and leave it up to you to decide on the merge. Let me know your decision so I can close the bug as well.

Paulo Neves

Quoted 74 lines · On Fri, Jun 17, 2022, 18:50 Richard Purdie <…
Quoted 20 lines · Paulo Neves

As reported in the bug report [1], there was no check for shebang sizes on native scripts and now this is fixed.

The path scope of the qa_staging was increased from just checking libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
meta/classes/insane.bbclass +0 -0
index 9ca84bace9..b2951a48fe 100644

meta/classes/insane.bbclass def qa_check_staged(path,d): · line 630

Quoted 1 line
meta/classes/insane.bbclass +0 -0
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" %

d.getVar('PN'))

meta/classes/insane.bbclass def qa_check_staged(path,d):

Quoted 5 lines
meta/classes/insane.bbclass +3 -0
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" %

d.getVar('PN'))

meta/classes/insane.bbclass def qa_check_staged(path,d): · line 656

Quoted 7 lines
meta/classes/insane.bbclass +2 -0
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in

path %s" % (file,root)

Quoted 13 lines
meta/classes/insane.bbclass +7 -0
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}

meta/classes/insane.bbclass addtask do_package_qa_setscene · line 1157

Quoted 8 lines
meta/classes/insane.bbclass +3 -1
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(sysroot_destdir + sysroot_dir, d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the

package built above", d)

meta/classes/insane.bbclass addtask do_package_qa_setscene · line 1158

Quoted 1 line
meta/classes/insane.bbclass +0 -0
 }

I'm a little worried about the performance implications of this, we're going from scanning files in libdir to scanning nearly all files in the sysroots, reading from many of them. In isolation that doesn't seem much but I suspect the IO will impact builds overall.

That leaves me a little torn on this change :/

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#45 lore raw
Quoted 63 lines · Richard Purdie via lists.openembedded.org
Quoted 46 lines · Paulo Neves

As reported in the bug report [1], there was no check for shebang sizes on native scripts and now this is fixed.

The path scope of the qa_staging was increased from just checking libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
meta/classes/insane.bbclass +12 -0
index 9ca84bace9..b2951a48fe 100644
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}

meta/classes/insane.bbclass addtask do_package_qa_setscene · line 1158

Quoted 9 lines
meta/classes/insane.bbclass +3 -1
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(sysroot_destdir + sysroot_dir, d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
 }

I'm a little worried about the performance implications of this, we're going from scanning files in libdir to scanning nearly all files in the sysroots, reading from many of them. In isolation that doesn't seem much but I suspect the IO will impact builds overall.

Quoted 1 line

That leaves me a little torn on this change :/

Sorry about the delay on this. My concern isn't an issue as I wasn't remembering the exact context of this code correctly. The patches have therefore merged, thanks!

Cheers,

Richard

#46 lore raw

PATCH v5 4/7 oeqa/selftest: Add test for shebang overflow

In reply to Paulo Neves

Make sure we do not stage any executable with a bigger shebang than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 .../sysroot-test/sysroot-shebang-test_1.0.bb         | 12 ++++++++++++
 meta/lib/oeqa/selftest/cases/sysroot.py              | 10 ++++++++++
 2 files changed, 22 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb new file +12 -0
new file mode 100644
index 0000000000..6c834be897
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+    chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
meta/lib/oeqa/selftest/cases/sysroot.py +10 -0
index 315d1a61c2..9457f1e3ac 100644
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
 TESTSTRING:pn-sysroot-test-arch2 = "%s"
 """ % (uuid1, uuid2))
         bitbake("sysroot-test")
+
+    def test_sysroot_max_shebang(self):
+        """
+        Summary:   Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+        Expected:  Fail when a shebang bigger than the max shebang-size is reached.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+        res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
Signature · 1 line
2.25.1
#47 lore raw

PATCH v5 5/7 oeqa/selftest: Test staged .la and .pc files

In reply to Paulo Neves

These files are checked by qa_check_staged but there was no test cases for whether the tests actually worked. Now there are.

---
 .../sysroot-test/sysroot-la-test_1.0.bb       | 16 ++++++++++
 .../sysroot-test/sysroot-pc-test_1.0.bb       | 12 +++++++
 meta/lib/oeqa/selftest/cases/sysroot.py       | 32 +++++++++++++++++++
 3 files changed, 60 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb new file +16 -0
new file mode 100644
index 0000000000..21f06782fb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb new file +12 -0
new file mode 100644
index 0000000000..e748310fc4
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
meta/lib/oeqa/selftest/cases/sysroot.py +32 -0
index 9457f1e3ac..588fc8c713 100644
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
         expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
         res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
         self.assertTrue(expected in res.output, msg=res.output)
+
+    def test_sysroot_la(self):
+        """
+        Summary:   Check that workdir paths are not contained in .la files.
+        Expected:  Fail when a workdir path is found in the file content.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "la-test.la failed sanity test (workdir) in path"
+
+        res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+        res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+    def test_sysroot_pkgconfig(self):
+        """
+        Summary:   Check that tmpdir paths are not contained in .pc files.
+        Expected:  Fail when a tmpdir path is found in the file content.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "test.pc failed sanity test (tmpdir) in path"
+
+        res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+        self.assertTrue(expected in res.output, msg=res.output)
+
+        res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
Signature · 1 line
2.25.1
#48 lore raw

PATCH v5 6/7 utils: Add cmdline_shebang_wrapper util.

In reply to Paulo Neves

Useful to work around shebang relocation issues, where shebangs are too long or have arguments in them, thus preventing them from using the /usr/bin/env shebang.

---
 .../wrapper/cmdline-shebang-wrapper-test.bb   | 21 ++++++++++++
 .../recipes-test/wrapper/files/test.awk       |  2 ++
 meta/classes/utils.bbclass                    | 34 +++++++++++++++++++
 meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
 4 files changed, 68 insertions(+)
 create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
 create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
 create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb new file +21 -0
new file mode 100644
index 0000000000..c4126a41fc
@@ -0,0 +1,21 @@
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+    create_cmdline_shebang_wrapper ${D}${bindir}/test
+    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+        bbfatal "Wrapper is broken"
+    else
+        bbnote "Wrapper is good"
+    fi
+}
+
+BBCLASSEXTEND = "native"
meta-selftest/recipes-test/wrapper/files/test.awk new file +2 -0
new file mode 100644
index 0000000000..91429197b1
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
meta/classes/utils.bbclass +34 -0
index b4eb3d38ab..b617632d9f 100644
@@ -184,6 +184,40 @@ END
 	chmod +x $cmd
 }
 
+create_cmdline_shebang_wrapper () {
+	# Create a wrapper script where commandline options are needed
+	#
+	# These are useful to work around shebang relocation issues, where shebangs are too
+  # long or have arguments in them, thus preventing them from using the /usr/bin/env
+	# shebang
+	#
+	# Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+	cmd=$1
+	shift
+
+	echo "Generating wrapper script for $cmd"
+
+  # Strip #! and get remaining interpreter + arg
+  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
+  # strip the shebang from the real script as we do not want it to be usable anyway
+  tail -n +2 $cmd > $cmd.real
+	cmdname=$(basename $cmd)
+	dirname=$(dirname $cmd)
+	cmdoptions=$@
+	if [ "${base_prefix}" != "" ]; then
+		relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+		cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`
+	fi
+	cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
+END
+	chmod +x $cmd
+}
+
 create_wrapper () {
 	# Create a wrapper script where extra environment variables are needed
 	#
meta/lib/oeqa/selftest/cases/wrapper.py new file +11 -0
new file mode 100644
index 0000000000..6de63310c0
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+    def test_shebang_wrapper(self):
+        """
+        Summary:   Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
+        Expected:  Exit status to be 0.
+        Author:    Paulo Neves <[email protected]>
+        """
+        res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
Signature · 1 line
2.25.1
#49 lore raw

Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

Quoted 1 line · Paulo Neves via lists.openembedded.org

+create_cmdline_shebang_wrapper () {

The indentation in this function is a mix of tabs and spaces, please unify.

Quoted 1 line

+ argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”

[ ]* seems overly, complex, no need for the brackets. I’d anchor to the beginning of the line, and remove the g as there’s only one expected.

Might also be wise to assert that you found something here, so running this on an ELF fails with an obvious error.

Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#50 lore raw

RE: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

Quoted 13 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Ross Burton
Sent: den 15 juni 2022 12:52
To: [email protected]
Cc: [email protected]
Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

Quoted 1 line · Paulo Neves via lists.openembedded.org

+create_cmdline_shebang_wrapper () {

Quoted 7 lines

The indentation in this function is a mix of tabs and spaces, please unify.

Quoted 1 line

+ argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”

[ ]* seems overly, complex, no need for the brackets. I’d anchor to the beginning of the line, and remove the g as there’s only one expected.

This should work:

argument=$(basename "$(sed -ne 's/^#! *//p;q')" $cmd)

However, if the first line is something like "#!/usr/bin/env python3", then $argument will be "env python3" and somehow I assume that is not exactly what the rest of the code expects.

Quoted 5 lines

Might also be wise to assert that you found something here, so running this on an ELF fails with an obvious error.

Ross

//Peter

#51 lore raw

Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

Quoted 18 lines · Peter Kjellerstedt
Quoted 11 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Ross Burton
Sent: den 15 juni 2022 12:52
To: [email protected]
Cc: [email protected]
Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

On 14 Jun 2022, at 16:11, Paulo Neves via lists.openembedded.org <[email protected]> wrote:

Quoted 7 lines
Quoted 1 line

+create_cmdline_shebang_wrapper () {

The indentation in this function is a mix of tabs and spaces, please unify.

Quoted 1 line

+ argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”

[ ]* seems overly, complex, no need for the brackets. I’d anchor to the beginning of the line, and remove the g as there’s only one expected.

Quoted 7 lines

This should work:

argument=$(basename "$(sed -ne 's/^#! *//p;q')" $cmd)

However, if the first line is something like "#!/usr/bin/env python3", then $argument will be "env python3" and somehow I assume that is not exactly what the rest of the code expects.

The use case is that whatever is in shebang should be extracted and ran verbatim, so truncating like your suggestion can work but has slightly different semantic.

Quoted 6 lines
Quoted 4 lines

Might also be wise to assert that you found something here, so running this on an ELF fails with an obvious error.

Ross

//Peter

Paulo Neves

#52 lore raw

RE: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

Quoted 33 lines

-----Original Message-----
From: Paulo Neves <[email protected]>
Sent: den 15 juni 2022 15:51
To: Peter Kjellerstedt <[email protected]>; Ross Burton <[email protected]>
Cc: [email protected]
Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

Quoted 7 lines · Peter Kjellerstedt
Quoted 7 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Ross Burton
Sent: den 15 juni 2022 12:52
To: [email protected]
Cc: [email protected]
Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add

cmdline_shebang_wrapper

Quoted 15 lines
Quoted 11 lines

util.

Quoted 1 line · Paulo Neves via lists.openembedded.org

+create_cmdline_shebang_wrapper () {

The indentation in this function is a mix of tabs and spaces, please unify.

Quoted 1 line

+ argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”

[ ]* seems overly, complex, no need for the brackets. I’d anchor to the beginning of the line, and remove the g as there’s only one expected.

This should work:

argument=$(basename "$(sed -ne 's/^#! *//p;q')" $cmd)

Quoted 7 lines
Quoted 3 lines

However, if the first line is something like "#!/usr/bin/env python3", then $argument will be "env python3" and somehow I assume that is not exactly what the rest of the code expects.

The use case is that whatever is in shebang should be extracted and ran verbatim, so truncating like your suggestion can work but has slightly different semantic.

Umm, my suggested line above should give exactly the same output as yours, only with fewer commands...

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

Might also be wise to assert that you found something here, so running this on an ELF fails with an obvious error.

Ross

//Peter

Paulo Neves

//Peter

#53 lore raw

Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

You are right never mind.

Paulo Neves

Quoted 51 lines · Peter Kjellerstedt
Quoted 31 lines

-----Original Message-----
From: Paulo Neves <[email protected]>
Sent: den 15 juni 2022 15:51
To: Peter Kjellerstedt <[email protected]>; Ross Burton <[email protected]>
Cc: [email protected]
Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

Quoted 7 lines · Peter Kjellerstedt
Quoted 7 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Ross Burton
Sent: den 15 juni 2022 12:52
To: [email protected]
Cc: [email protected]
Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add

cmdline_shebang_wrapper

Quoted 13 lines
Quoted 11 lines

util.

Quoted 1 line · Paulo Neves via lists.openembedded.org

+create_cmdline_shebang_wrapper () {

The indentation in this function is a mix of tabs and spaces, please unify.

Quoted 1 line

+ argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”

[ ]* seems overly, complex, no need for the brackets. I’d anchor to the beginning of the line, and remove the g as there’s only one expected.

This should work:

Quoted 8 lines
Quoted 5 lines

argument=$(basename "$(sed -ne 's/^#! *//p;q')" $cmd)

However, if the first line is something like "#!/usr/bin/env python3", then $argument will be "env python3" and somehow I assume that is not exactly what the rest of the code expects.

The use case is that whatever is in shebang should be extracted and ran verbatim, so truncating like your suggestion can work but has slightly different semantic.

Umm, my suggested line above should give exactly the same output as yours, only with fewer commands...

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

Might also be wise to assert that you found something here, so running this on an ELF fails with an obvious error.

Ross

//Peter

Paulo Neves

//Peter

#54 lore raw

PATCH v5 7/7 libcheck: Fix too long shebang for native case.

In reply to Paulo Neves

It requires a shebang wrapper due to the fact that awk interpreter has an argument.

---
 meta/recipes-support/libcheck/libcheck_0.15.2.bb | 3 +++
 1 file changed, 3 insertions(+)
meta/recipes-support/libcheck/libcheck_0.15.2.bb +3 -0
index 188d689cc3..1393aa2a1c 100644
@@ -23,6 +23,9 @@ CACHED_CONFIGUREVARS += "ac_cv_path_AWK_PATH=${bindir}/gawk"
 
 RREPLACES:${PN} = "check (<= 0.9.5)"
 
+do_install:append:class-native() {
+    create_cmdline_shebang_wrapper ${D}${bindir}/checkmk
+}
 BBCLASSEXTEND = "native nativesdk"
 
 PACKAGES =+ "checkmk"
Signature · 1 line
2.25.1
#55 lore raw
In reply to Paulo Neves

meta/classes/insane.bbclass def qa_check_staged(path,d): · line 663

Quoted 5 lines
meta/classes/insane.bbclass +5 -0
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)

Where is package_qa_check_shebang_size() defined?

Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#56 lore raw

Never mind, can’t read :)

Quoted 11 lines · Ross Burton

meta/classes/insane.bbclass def qa_check_staged(path,d): · line 663

Quoted 6 lines
meta/classes/insane.bbclass +5 -0
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)

Where is package_qa_check_shebang_size() defined?

Ross

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#57 lore raw

:)

The function should probably be refactored so it does not imply it is only useful for packaging.

Paulo Neves

Quoted 13 lines · Ross Burton

Never mind, can’t read :)

Quoted 10 lines · Ross Burton

meta/classes/insane.bbclass def qa_check_staged(path,d): · line 663

Quoted 6 lines
meta/classes/insane.bbclass +5 -0
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)

Where is package_qa_check_shebang_size() defined?

Ross

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#58 lore raw

PATCH v6 1/7 python: Avoid shebang overflow on python-config.py

In reply to Paulo Neves

The native path may be too big, leading to shebang overflow. Just use the #!/usr/bin/env python3.

---
 ...shebang-overflow-on-python-config.py.patch | 33 +++++++++++++++++++
 .../recipes-devtools/python/python3_3.10.4.bb |  1 +
 2 files changed, 34 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch new file +33 -0
new file mode 100644
index 0000000000..921da8de7c
@@ -0,0 +1,33 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <[email protected]>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Denial reason: [1]
+
+Upstream-Status: Denied [distribution]
+
+[1] https://github.com/python/cpython/pull/93760#pullrequestreview-1005365737
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ 	@ # Substitution happens here, as the completely-expanded BINDIR
+ 	@ # is not available in configure
+ 	sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++	@ # Otherwise we might get huge shebangs with native paths
++	sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ 	@ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ 	LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ 	@  # In OpenEmbedded, always use the python version of the script, the shell
+-- 
+2.25.1
+
meta/recipes-devtools/python/python3_3.10.4.bb +1 -0
index 6bd3a6aba8..357025f856 100644
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
            file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
            file://deterministic_imports.patch \
+           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
            "
 
 SRC_URI:append:class-native = " \
Signature · 1 line
2.25.1
#59 lore raw

PATCH v6 2/7 gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2

A native path can cause a shebang overflow on gtkdoc-mkhtml. Replace it with /usr/bin/env.

---
 meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb +1 -0
index 150d2c0b23..392913fcc6 100644
@@ -43,6 +43,7 @@ do_install:append () {
         ${datadir}/gtk-doc/python/gtkdoc/config.py; do
         sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
             -e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+            -e '1s|^#!.*|#!/usr/bin/env python3|' \
             -i ${D}$fn
     done
 }
Signature · 1 line
2.25.1
#60 lore raw

PATCH v6 3/7 insane.bbclass: Make do_qa_staging check shebangs

In reply to Paulo Neves

As reported in the bug report [1], there was no check for shebang sizes on native scripts and now this is fixed.

The path scope of the qa_staging was increased from just checking libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
meta/classes/insane.bbclass +15 -1
index 9ca84bace9..b2951a48fe 100644
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(sysroot_destdir + sysroot_dir, d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
 }
 
Signature · 1 line
2.25.1
#61 lore raw

PATCH v6 4/7 oeqa/selftest: Add test for shebang overflow

In reply to Paulo Neves

Make sure we do not stage any executable with a bigger shebang than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

---
 .../sysroot-test/sysroot-shebang-test_1.0.bb         | 12 ++++++++++++
 meta/lib/oeqa/selftest/cases/sysroot.py              | 11 +++++++++++
 2 files changed, 23 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb new file +12 -0
new file mode 100644
index 0000000000..6c834be897
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+    chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
meta/lib/oeqa/selftest/cases/sysroot.py +11 -0
index 315d1a61c2..a17fd1818a 100644
@@ -35,3 +35,14 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
 TESTSTRING:pn-sysroot-test-arch2 = "%s"
 """ % (uuid1, uuid2))
         bitbake("sysroot-test")
+
+    def test_sysroot_max_shebang(self):
+        """
+        Summary:   Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+        Expected:  Fail when a shebang bigger than the max shebang-size is reached.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+        res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue(res.status != 0)
Signature · 1 line
2.25.1
#62 lore raw

PATCH v6 5/7 oeqa/selftest: Test staged .la and .pc files

In reply to Paulo Neves

These files are checked by qa_check_staged but there was no test cases for whether the tests actually worked. Now there are.

---
 .../sysroot-test/sysroot-la-test_1.0.bb       | 16 +++++++++
 .../sysroot-test/sysroot-pc-test_1.0.bb       | 12 +++++++
 meta/lib/oeqa/selftest/cases/sysroot.py       | 36 +++++++++++++++++++
 3 files changed, 64 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb new file +16 -0
new file mode 100644
index 0000000000..21f06782fb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb new file +12 -0
new file mode 100644
index 0000000000..e748310fc4
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
meta/lib/oeqa/selftest/cases/sysroot.py +36 -0
index a17fd1818a..294ba4a4a0 100644
@@ -46,3 +46,39 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
         res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
         self.assertTrue(expected in res.output, msg=res.output)
         self.assertTrue(res.status != 0)
+
+    def test_sysroot_la(self):
+        """
+        Summary:   Check that workdir paths are not contained in .la files.
+        Expected:  Fail when a workdir path is found in the file content.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "la-test.la failed sanity test (workdir) in path"
+
+        res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+        self.assertTrue(res.status != 0)
+
+        res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+        self.assertTrue(res.status != 0)
+
+    def test_sysroot_pkgconfig(self):
+        """
+        Summary:   Check that tmpdir paths are not contained in .pc files.
+        Expected:  Fail when a tmpdir path is found in the file content.
+        Author:    Paulo Neves <[email protected]>
+        """
+        expected = "test.pc failed sanity test (tmpdir) in path"
+
+        res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue(res.status != 0)
+
+        res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+        self.assertTrue(res.status != 0)
Signature · 1 line
2.25.1
#63 lore raw

PATCH v6 6/7 utils: Add cmdline_shebang_wrapper util.

In reply to Paulo Neves

Useful to work around shebang relocation issues, where shebangs are too long or have arguments in them, thus preventing them from using the /usr/bin/env shebang.

---
 .../wrapper/cmdline-shebang-wrapper-test.bb   | 21 ++++++++++++
 .../recipes-test/wrapper/files/test.awk       |  2 ++
 meta/classes/utils.bbclass                    | 34 +++++++++++++++++++
 meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
 4 files changed, 68 insertions(+)
 create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
 create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
 create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb new file +21 -0
new file mode 100644
index 0000000000..c4126a41fc
@@ -0,0 +1,21 @@
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+    create_cmdline_shebang_wrapper ${D}${bindir}/test
+    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+        bbfatal "Wrapper is broken"
+    else
+        bbnote "Wrapper is good"
+    fi
+}
+
+BBCLASSEXTEND = "native"
meta-selftest/recipes-test/wrapper/files/test.awk new file +2 -0
new file mode 100644
index 0000000000..91429197b1
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
meta/classes/utils.bbclass +34 -0
index b4eb3d38ab..b58c22771f 100644
@@ -184,6 +184,40 @@ END
 	chmod +x $cmd
 }
 
+create_cmdline_shebang_wrapper () {
+	# Create a wrapper script where commandline options are needed
+	#
+	# These are useful to work around shebang relocation issues, where shebangs are too
+	# long or have arguments in them, thus preventing them from using the /usr/bin/env
+	# shebang
+	#
+	# Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+	cmd=$1
+	shift
+
+	echo "Generating wrapper script for $cmd"
+
+	# Strip #! and get remaining interpreter + arg
+	argument="$(sed -ne 's/^#! *//p;q' $cmd)"
+	# strip the shebang from the real script as we do not want it to be usable anyway
+	tail -n +2 $cmd > $cmd.real
+	cmdname=$(basename $cmd)
+	dirname=$(dirname $cmd)
+	cmdoptions=$@
+	if [ "${base_prefix}" != "" ]; then
+		relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+		cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`
+	fi
+	cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
+END
+	chmod +x $cmd
+}
+
 create_wrapper () {
 	# Create a wrapper script where extra environment variables are needed
 	#
meta/lib/oeqa/selftest/cases/wrapper.py new file +11 -0
new file mode 100644
index 0000000000..6de63310c0
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+    def test_shebang_wrapper(self):
+        """
+        Summary:   Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
+        Expected:  Exit status to be 0.
+        Author:    Paulo Neves <[email protected]>
+        """
+        res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
Signature · 1 line
2.25.1
#64 lore raw

Re: [OE-core] [PATCH v6 6/7] utils: Add cmdline_shebang_wrapper util.

Quoted 80 lines · Paulo Neves

Useful to work around shebang relocation issues, where shebangs are too long or have arguments in them, thus preventing them from using the /usr/bin/env shebang.

---
  .../wrapper/cmdline-shebang-wrapper-test.bb   | 21 ++++++++++++
  .../recipes-test/wrapper/files/test.awk       |  2 ++
  meta/classes/utils.bbclass                    | 34 +++++++++++++++++++
  meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
  4 files changed, 68 insertions(+)

create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py

meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb new file +21 -0
new file mode 100644
index 0000000000..c4126a41fc
@@ -0,0 +1,21 @@
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+    create_cmdline_shebang_wrapper ${D}${bindir}/test
+    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+        bbfatal "Wrapper is broken"
+    else
+        bbnote "Wrapper is good"
+    fi
+}
+
+BBCLASSEXTEND = "native"
meta-selftest/recipes-test/wrapper/files/test.awk new file +2 -0
new file mode 100644
index 0000000000..91429197b1
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
meta/classes/utils.bbclass +24 -0
index b4eb3d38ab..b58c22771f 100644
@@ -184,6 +184,40 @@ END
  	chmod +x $cmd
  }
 
+create_cmdline_shebang_wrapper () {
+	# Create a wrapper script where commandline options are needed
+	#
+	# These are useful to work around shebang relocation issues, where shebangs are too
+	# long or have arguments in them, thus preventing them from using the /usr/bin/env
+	# shebang
+	#
+	# Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+	cmd=$1
+	shift
+
+	echo "Generating wrapper script for $cmd"
+
+	# Strip #! and get remaining interpreter + arg
+	argument="$(sed -ne 's/^#! *//p;q' $cmd)"
+	# strip the shebang from the real script as we do not want it to be usable anyway
+	tail -n +2 $cmd > $cmd.real
+	cmdname=$(basename $cmd)
+	dirname=$(dirname $cmd)
+	cmdoptions=$@
+	if [ "${base_prefix}" != "" ]; then
+		relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+		cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`

meta/classes/utils.bbclass END · line 220

Quoted 10 lines
meta/classes/utils.bbclass +10 -0
+	fi
+	cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
+END
+	chmod +x $cmd
+}
+

Maybe this has already been raised before, but the wrapper above does a couple of addtitional forks, which could be avoided if we use a python wrapper script instead.

Jacob

Quoted 31 lines
meta/classes/utils.bbclass +0 -0
  create_wrapper () {
  	# Create a wrapper script where extra environment variables are needed
  	#
meta/lib/oeqa/selftest/cases/wrapper.py new file +11 -1
new file mode 100644
index 0000000000..6de63310c0
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+    def test_shebang_wrapper(self):
+        """
+        Summary:   Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
+        Expected:  Exit status to be 0.
+        Author:    Paulo Neves <[email protected]>
+        """
+        res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
 
 
 
-=-=-=-=-=-=-=-=-=-=-=-

Links: You receive all messages sent to this group.
View/Reply Online (#167044): https://lists.openembedded.org/g/openembedded-core/message/167044
Mute This Topic: https://lists.openembedded.org/mt/91863579/4454410
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#65 lore raw

PATCH v6 7/7 libcheck: Fix too long shebang for native case.

In reply to Paulo Neves

It requires a shebang wrapper due to the fact that awk interpreter has an argument.

---
 meta/recipes-support/libcheck/libcheck_0.15.2.bb | 3 +++
 1 file changed, 3 insertions(+)
meta/recipes-support/libcheck/libcheck_0.15.2.bb +3 -0
index 188d689cc3..1393aa2a1c 100644
@@ -23,6 +23,9 @@ CACHED_CONFIGUREVARS += "ac_cv_path_AWK_PATH=${bindir}/gawk"
 
 RREPLACES:${PN} = "check (<= 0.9.5)"
 
+do_install:append:class-native() {
+    create_cmdline_shebang_wrapper ${D}${bindir}/checkmk
+}
 BBCLASSEXTEND = "native nativesdk"
 
 PACKAGES =+ "checkmk"
Signature · 1 line
2.25.1