Skip to contentSkip to conversation

bitbake.conf: Prevent pyc file creation in pseudo context for shell tasks

PATCH In review Peter Kjellerstedt 3 participants 3 messages Mar 19, 2021 – Mar 20, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bitbake.conf: Prevent pyc file creation in pseudo context for shell tasks

In commit 73d538f2 (bitbake.conf: Prevent pyc file generation in pseudo context), pyc generation was disabled for fakeroot python tasks. However, Python applications started from a fakeroot shell task were not affected by this and would still generate pyc files, which could trigger pseudo to abort. To avoid this, add PYTHONDONTWRITEBYTECODE=1 also to FAKEROOTENV.

Also add a test case to ensure that pyc files are not created by Python applications started from shell tasks that are executed under pseudo.

---
 .../pseudo-pyc-test/files/test-app1.py        |  5 +++
 .../pseudo-pyc-test/files/test-app2.py        |  4 +++
 ...-pyc-test.bb => pseudo-pyc-python-test.bb} |  0
 .../pseudo-pyc-test/pseudo-pyc-shell-test.bb  | 16 ++++++++++
 meta/conf/bitbake.conf                        |  2 +-
 meta/lib/oeqa/selftest/cases/pseudo.py        | 32 ++++++++++++-------
 6 files changed, 47 insertions(+), 12 deletions(-)
 create mode 100755 meta-selftest/recipes-test/pseudo-pyc-test/files/test-app1.py
 create mode 100755 meta-selftest/recipes-test/pseudo-pyc-test/files/test-app2.py
 rename meta-selftest/recipes-test/pseudo-pyc-test/{pseudo-pyc-test.bb => pseudo-pyc-python-test.bb} (100%)
 create mode 100644 meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-shell-test.bb
meta-selftest/recipes-test/pseudo-pyc-test/files/test-app1.py new file +5 -0
new file mode 100755
index 0000000000..7578870765
@@ -0,0 +1,5 @@
+#!/usr/bin/env python3
+import sys
+
+import pseudo_pyc_test1
+print(pseudo_pyc_test1.STRING)
meta-selftest/recipes-test/pseudo-pyc-test/files/test-app2.py new file +4 -0
new file mode 100755
index 0000000000..2ee1e24e52
@@ -0,0 +1,4 @@
+#!/usr/bin/env python3
+
+import pseudo_pyc_test2
+print(pseudo_pyc_test2.STRING)
meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-python-test.bb renamed from meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-test.bb +0 -0
similarity index 100%
rename from meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-test.bb
rename to meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-python-test.bb
meta-selftest/recipes-test/pseudo-pyc-test/pseudo-pyc-shell-test.bb new file +16 -0
new file mode 100644
index 0000000000..1e0c532a3a
@@ -0,0 +1,16 @@
+SUMMARY = "pseudo env test"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI = "file://test-app1.py"
+SRC_URI += "file://test-app2.py"
+
+do_compile() {
+	${WORKDIR}/test-app1.py
+}
+
+do_install() {
+	${WORKDIR}/test-app2.py
+}
meta/conf/bitbake.conf +1 -1
index ecd4d1638e..3a158d0e6f 100644
@@ -694,7 +694,7 @@ export PSEUDO_DISABLED = "1"
 #export PSEUDO_LIBDIR = "${STAGING_DIR_NATIVE}$PSEUDOBINDIR/../lib/pseudo/lib
 FAKEROOTBASEENV = "PSEUDO_BINDIR=${PSEUDO_SYSROOT}${bindir_native} PSEUDO_LIBDIR=${PSEUDO_SYSROOT}${prefix_native}/lib/pseudo/lib PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=1 PYTHONDONTWRITEBYTECODE=1"
 FAKEROOTCMD = "${PSEUDO_SYSROOT}${bindir_native}/pseudo"
-FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=0"
+FAKEROOTENV = "PSEUDO_PREFIX=${PSEUDO_SYSROOT}${prefix_native} PSEUDO_LOCALSTATEDIR=${PSEUDO_LOCALSTATEDIR} PSEUDO_PASSWD=${PSEUDO_PASSWD} PSEUDO_NOSYMLINKEXP=1 PSEUDO_IGNORE_PATHS=${@oe.path.canonicalize(d.getVar('PSEUDO_IGNORE_PATHS'))} PSEUDO_DISABLED=0 PYTHONDONTWRITEBYTECODE=1"
 FAKEROOTNOENV = "PSEUDO_UNLOAD=1"
 FAKEROOTDIRS = "${PSEUDO_LOCALSTATEDIR}"
 FAKEROOTLOGS = "${WORKDIR}/pseudo/pseudo.log"
meta/lib/oeqa/selftest/cases/pseudo.py +21 -11
index 33593d5ce9..4265b39ecf 100644
@@ -9,19 +9,29 @@ from oeqa.utils.commands import bitbake, get_test_layer
 from oeqa.selftest.case import OESelftestTestCase
 
 class Pseudo(OESelftestTestCase):
+    def setUp(self):
+        metaselftestpath = get_test_layer()
+        lib_path = os.path.join(metaselftestpath, 'lib')
+        self.write_config('export PYTHONPATH="%s"' % lib_path)
 
-    def test_pseudo_pyc_creation(self):
-        self.write_config("")
+        self.pycache_path = os.path.join(lib_path, '__pycache__')
+        if os.path.exists(self.pycache_path):
+            shutil.rmtree(self.pycache_path)
 
-        metaselftestpath = get_test_layer()
-        pycache_path = os.path.join(metaselftestpath, 'lib/__pycache__')
-        if os.path.exists(pycache_path):
-            shutil.rmtree(pycache_path)
+    def test_pseudo_pyc_creation_in_python_task(self):
+        bitbake('pseudo-pyc-python-test -c install')
+
+        test1_pyc_present = len(glob.glob(os.path.join(self.pycache_path, 'pseudo_pyc_test1.*.pyc')))
+        self.assertTrue(test1_pyc_present, 'python test1 pyc file missing, should be created outside of pseudo context.')
+
+        test2_pyc_present = len(glob.glob(os.path.join(self.pycache_path, 'pseudo_pyc_test2.*.pyc')))
+        self.assertFalse(test2_pyc_present, 'python test2 pyc file present, should not be created in pseudo context.')
 
-        bitbake('pseudo-pyc-test -c install')
+    def test_pseudo_pyc_creation_in_shell_task(self):
+        bitbake('pseudo-pyc-shell-test -c install')
 
-        test1_pyc_present = len(glob.glob(os.path.join(pycache_path, 'pseudo_pyc_test1.*.pyc')))
-        self.assertTrue(test1_pyc_present, 'test1 pyc file missing, should be created outside of pseudo context.')
+        test1_pyc_present = len(glob.glob(os.path.join(self.pycache_path, 'pseudo_pyc_test1.*.pyc')))
+        self.assertTrue(test1_pyc_present, 'shell test1 pyc file missing, should be created outside of pseudo context.')
 
-        test2_pyc_present = len(glob.glob(os.path.join(pycache_path, 'pseudo_pyc_test2.*.pyc')))
-        self.assertFalse(test2_pyc_present, 'test2 pyc file present, should not be created in pseudo context.')
+        test2_pyc_present = len(glob.glob(os.path.join(self.pycache_path, 'pseudo_pyc_test2.*.pyc')))
+        self.assertFalse(test2_pyc_present, 'shell test2 pyc file present, should not be created in pseudo context.')
#1 lore raw
Quoted 7 lines · Peter Kjellerstedt

In commit 73d538f2 (bitbake.conf: Prevent pyc file generation in pseudo context), pyc generation was disabled for fakeroot python tasks. However, Python applications started from a fakeroot shell task were not affected by this and would still generate pyc files, which could trigger pseudo to abort. To avoid this, add PYTHONDONTWRITEBYTECODE=1 also to FAKEROOTENV.

Quoted 5 lines

Also add a test case to ensure that pyc files are not created by Python applications started from shell tasks that are executed under pseudo.

It is looking like this might not be quite so simple.

https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/3225

Basically if we do what this patch does, it disables pyc files everywhere including target recipes/output.

Cheers,

Richard

#2 lore raw
Quoted 12 lines · Richard Purdie
Quoted 11 lines · Peter Kjellerstedt

In commit 73d538f2 (bitbake.conf: Prevent pyc file generation in pseudo context), pyc generation was disabled for fakeroot python tasks. However, Python applications started from a fakeroot shell task were not affected by this and would still generate pyc files, which could trigger pseudo to abort. To avoid this, add PYTHONDONTWRITEBYTECODE=1 also to FAKEROOTENV.

Also add a test case to ensure that pyc files are not created by Python applications started from shell tasks that are executed under pseudo.

Quoted 9 lines
Quoted 1 line

It is looking like this might not be quite so simple.

https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/3225

Basically if we do what this patch does, it disables pyc files everywhere including target recipes/output.

seems to break builds meson as well see
https://errors.yoctoproject.org/Errors/Details/574281/

Quoted 8 lines

Cheers,

Richard