Skip to contentSkip to conversation

patch review request

PATCH New Anuj Mittal 1 participant 21 messages Jan 21, 2021

lore
Share & tools
lore
#0 lore raw

cover patch review request

Please review these next set of changes for gatesgarth. Builds cleanly on autobuilder.

Thanks,

Anuj

The following changes since commit c7bf9aebd002fc6bc23a57bc5e9863382bd17ae7:

systemd: upgrade 246.6 -> 246.9 (2021-01-15 00:05:41 +0800)

are available in the Git repository at:

git://push.openembedded.org/openembedded-core-contrib anujm/gatesgarth

Adrian Herrera (1):
scripts: oe-run-native, fix *-native directories

Andrey Mozzhuhin (1):
toolchain-shar-extract.sh: Handle special characters in script path

Chris Laplante (1):
systemd.bbclass: improve error message when a service unit specified
in SYSTEMD_SERVICE is not found

Mans Rullgard (1):
boost: drop arm-intrinsics.patch

Marek Vasut (2):
meta: toolchain-shar-relocate.sh: Do not use $target_sdk_dir as regex
meta: toolchain-shar-relocate.sh: Filter out post-relocate-setup
script

Mikko Rapeli (1):
zip: whitelist CVE-2018-13410 and CVE-2018-13684

Nathan Rossi (1):
gcc: Backport patch to resolve i*86 tune configuration overrides

Oleksiy Obitotskyy (1):
flex: Fix --noline option behavior

Oleksiy Obitotskyy yIEf0zt.mo (1):
toolchain-shar-relocate.sh: Fix handling files with colons

Paul Barker (7):
wic: Add workdir argument
wic: Ensure internal workdir is not reused
wic: Allow exec_native_cmd to run HOSTTOOLS
image_types_wic: Move wic working directory
wic: Update pseudo db when excluding content from rootfs
wic: Copy rootfs dir if fstab needs updating
wic: Optimise fstab modification for ext2/3/4 and msdos partitions

Tomasz Dziendzielski (2):
lib/oe/utils: Return empty string in parallel_make
devtool: Fix source extraction for gcc shared source

Yi Fan Yu (1):
binutils: Fix CVE-2020-35448

 meta/classes/image_types_wic.bbclass          |  10 +-
 meta/classes/systemd.bbclass                  |   3 +-
 meta/files/toolchain-shar-extract.sh          |  12 +-
 meta/files/toolchain-shar-relocate.sh         |   7 +-
 meta/lib/oe/utils.py                          |   2 +-
 .../binutils/binutils-2.35.1.inc              |   1 +
 .../binutils/binutils/CVE-2020-35448.patch    |  85 +++++++
 ...directives-if-gen_line_dirs-is-false.patch |  32 +++
 meta/recipes-devtools/flex/flex_2.6.4.bb      |   1 +
 meta/recipes-devtools/gcc/gcc-10.2.inc        |   2 +-
 ...0001-lib-Remove-i-86-march-overrides.patch | 113 ----------
 ...libgomp-libitc-Fix-bootstrap-PR70454.patch | 208 ++++++++++++++++++
 meta/recipes-extended/zip/zip_3.0.bb          |   6 +
 .../boost/boost/arm-intrinsics.patch          |  55 -----
 meta/recipes-support/boost/boost_1.74.0.bb    |   2 +-
 scripts/lib/devtool/standard.py               |  30 ++-
 scripts/lib/wic/misc.py                       |   7 +-
 scripts/lib/wic/partition.py                  |  32 ++-
 scripts/lib/wic/plugins/imager/direct.py      |  48 ++--
 scripts/lib/wic/plugins/source/rootfs.py      |  33 ++-
 scripts/oe-run-native                         |   2 +-
 scripts/wic                                   |   2 +
 22 files changed, 464 insertions(+), 229 deletions(-)
 create mode 100644 meta/recipes-devtools/binutils/binutils/CVE-2020-35448.patch
 create mode 100644 meta/recipes-devtools/flex/flex/0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch
 delete mode 100644 meta/recipes-devtools/gcc/gcc/0001-lib-Remove-i-86-march-overrides.patch
 create mode 100644 meta/recipes-devtools/gcc/gcc/0001-libatomic-libgomp-libitc-Fix-bootstrap-PR70454.patch
 delete mode 100644 meta/recipes-support/boost/boost/arm-intrinsics.patch
Signature · 1 line
2.29.2
#1 lore raw

PATCH 1/20 zip: whitelist CVE-2018-13410 and CVE-2018-13684

From: Mikko Rapeli <[email protected]>

https://nvd.nist.gov/vuln/detail/CVE-2018-13410 is disputed and also Debian considers it not a vulnerability:

https://security-tracker.debian.org/tracker/CVE-2018-13410

http://seclists.org/fulldisclosure/2018/Jul/24
"Negligible security impact, would involve that a untrusted party controls the -TT value."

https://nvd.nist.gov/vuln/detail/CVE-2018-13684 is not for zip, also Debian concludes this:

https://security-tracker.debian.org/tracker/CVE-2018-13684

"NOT-FOR-US: smart contract implementation for ZIP"

---
 meta/recipes-extended/zip/zip_3.0.bb | 6 ++++++
 1 file changed, 6 insertions(+)
meta/recipes-extended/zip/zip_3.0.bb +6 -0
index c00a932763..97e5e57533 100644
@@ -19,6 +19,12 @@ UPSTREAM_VERSION_UNKNOWN = "1"
 SRC_URI[md5sum] = "7b74551e63f8ee6aab6fbc86676c0d37"
 SRC_URI[sha256sum] = "f0e8bb1f9b7eb0b01285495a2699df3a4b766784c1765a8f1aeedf63c0806369"
 
+# Disputed and also Debian doesn't consider a vulnerability
+CVE_CHECK_WHITELIST += "CVE-2018-13410"
+
+# Not for zip but for smart contract implementation for it
+CVE_CHECK_WHITELIST += "CVE-2018-13684"
+
 # zip.inc sets CFLAGS, but what Makefile actually uses is
 # CFLAGS_NOOPT.  It will also force -O3 optimization, overriding
 # whatever we set.
Signature · 1 line
2.29.2
#2 lore raw

PATCH 2/20 scripts: oe-run-native, fix *-native directories

In reply to Anuj Mittal

From: Adrian Herrera <[email protected]>

This fixes a crash with "find" when running a native tool and *-native directories do not exist under the binary directory in the sysroot. This happened because the directory wildcard was passed as part of the root directory.
The directory wildcard is now passed by "-name", which returns an empty result if no matching directory.

Change-Id Iba7acd8bbd7e0beb4d25c984f6af7a4fd21486e6
(cherry picked from commit f6c90ed0ad24b7d4f892e22e088b1578824eb1d3)
---
 scripts/oe-run-native | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/oe-run-native +1 -1
index 4e63e69cc4..22958d97e7 100755
@@ -43,7 +43,7 @@ fi
 OLD_PATH=$PATH
 
 # look for a tool only in native sysroot
-PATH=$OECORE_NATIVE_SYSROOT/usr/bin:$OECORE_NATIVE_SYSROOT/bin:$OECORE_NATIVE_SYSROOT/usr/sbin:$OECORE_NATIVE_SYSROOT/sbin$(find $OECORE_NATIVE_SYSROOT/usr/bin/*-native -maxdepth 1 -type d -printf ":%p")
+PATH=$OECORE_NATIVE_SYSROOT/usr/bin:$OECORE_NATIVE_SYSROOT/bin:$OECORE_NATIVE_SYSROOT/usr/sbin:$OECORE_NATIVE_SYSROOT/sbin$(find $OECORE_NATIVE_SYSROOT/usr/bin -maxdepth 1 -name "*-native" -type d -printf ":%p")
 tool_find=`/usr/bin/which $tool 2>/dev/null`
 
 if [ -n "$tool_find" ] ; then
Signature · 1 line
2.29.2
#3 lore raw

PATCH 3/20 toolchain-shar-extract.sh: Handle special characters in script path

In reply to Anuj Mittal

From: Andrey Mozzhuhin <[email protected]>

Extracting SDK archive may fail if the script is run using a path with special characters such as space or asterisk. This is because the shell interprets such characters after expanding the $0 variable.

Added quotes to all uses of the shell variable $0 to fix this.

(cherry picked from commit 0453acbbd45604537090ec7a3295b34309e6eecb)
---
 meta/files/toolchain-shar-extract.sh | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
meta/files/toolchain-shar-extract.sh +6 -6
index bea6d4189a..dd9342758b 100644
@@ -95,7 +95,7 @@ while getopts ":yd:npDRSl" OPT; do
 		listcontents=1
 		;;
 	*)
-		echo "Usage: $(basename $0) [-y] [-d <dir>]"
+		echo "Usage: $(basename "$0") [-y] [-d <dir>]"
 		echo "  -y         Automatic yes to all prompts"
 		echo "  -d <dir>   Install the SDK to <dir>"
 		echo "======== Extensible SDK only options ============"
@@ -111,17 +111,17 @@ while getopts ":yd:npDRSl" OPT; do
 	esac
 done
 
-payload_offset=$(($(grep -na -m1 "^MARKER:$" $0|cut -d':' -f1) + 1))
+payload_offset=$(($(grep -na -m1 "^MARKER:$" "$0"|cut -d':' -f1) + 1))
 if [ "$listcontents" = "1" ] ; then
     if [ @SDK_ARCHIVE_TYPE@ = "zip" ]; then
-        tail -n +$payload_offset $0 > sdk.zip
+        tail -n +$payload_offset "$0" > sdk.zip
         if unzip -l sdk.zip;then
             rm sdk.zip
         else
             rm sdk.zip && exit 1
         fi
     else
-        tail -n +$payload_offset $0| tar tvJ || exit 1
+        tail -n +$payload_offset "$0"| tar tvJ || exit 1
     fi
     exit
 fi
@@ -242,14 +242,14 @@ fi
 
 printf "Extracting SDK..."
 if [ @SDK_ARCHIVE_TYPE@ = "zip" ]; then
-    tail -n +$payload_offset $0 > sdk.zip
+    tail -n +$payload_offset "$0" > sdk.zip
     if $SUDO_EXEC unzip $EXTRA_TAR_OPTIONS sdk.zip -d $target_sdk_dir;then
         rm sdk.zip
     else
         rm sdk.zip && exit 1
     fi
 else
-    tail -n +$payload_offset $0| $SUDO_EXEC tar mxJ -C $target_sdk_dir --checkpoint=.2500 $EXTRA_TAR_OPTIONS || exit 1
+    tail -n +$payload_offset "$0"| $SUDO_EXEC tar mxJ -C $target_sdk_dir --checkpoint=.2500 $EXTRA_TAR_OPTIONS || exit 1
 fi
 echo "done"
 
Signature · 1 line
2.29.2
#4 lore raw

PATCH 4/20 systemd.bbclass: improve error message when a service unit specified in SYSTEMD_SERVICE is not found

In reply to Anuj Mittal

From: Chris Laplante <[email protected]>

The previous message was fairly useless without diving into the bbclass.

(cherry picked from commit ef6117b148be65536e89409a83cbfd22049c652e)
---
 meta/classes/systemd.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
meta/classes/systemd.bbclass +2 -1
index 9e8a82c9f1..a4bff732b9 100644
@@ -174,7 +174,8 @@ python systemd_populate_packages() {
                 if path_found != '':
                     systemd_add_files_and_parse(pkg_systemd, path_found, service, keys)
                 else:
-                    bb.fatal("SYSTEMD_SERVICE_%s value %s does not exist" % (pkg_systemd, service))
+                    bb.fatal("Didn't find service unit '{0}', specified in SYSTEMD_SERVICE_{1}. {2}".format(
+                        service, pkg_systemd, "Also looked for service unit '{0}'.".format(base) if base is not None else ""))
 
     def systemd_create_presets(pkg, action):
         presetf = oe.path.join(d.getVar("PKGD"), d.getVar("systemd_unitdir"), "system-preset/98-%s.preset" % pkg)
Signature · 1 line
2.29.2
#5 lore raw

PATCH 5/20 boost: drop arm-intrinsics.patch

In reply to Anuj Mittal

From: Mans Rullgard <[email protected]>

This patch makes gcc produce broken code. It is unclear why it is there in the first place. Drop it.

(cherry picked from commit 5f3cace37496fe1dc4fd045f688f7d441505c437)
---
 .../boost/boost/arm-intrinsics.patch          | 55 -------------------
 meta/recipes-support/boost/boost_1.74.0.bb    |  2 +-
 2 files changed, 1 insertion(+), 56 deletions(-)
 delete mode 100644 meta/recipes-support/boost/boost/arm-intrinsics.patch
meta/recipes-support/boost/boost/arm-intrinsics.patch deleted +0 -55
deleted file mode 100644
index fe85c69a82..0000000000
@@ -1,55 +0,0 @@
-Upstream-Status: Backport
-
-8/17/2010 - rebased to 1.44 by Qing He <[email protected]>
-
-diff --git a/boost/smart_ptr/detail/atomic_count_sync.hpp b/boost/smart_ptr/detail/atomic_count_sync.hpp
-index b6359b5..78b1cc2 100644
---- a/boost/smart_ptr/detail/atomic_count_sync.hpp
-+++ b/boost/smart_ptr/detail/atomic_count_sync.hpp
-@@ -33,17 +33,46 @@ public:
- 
-     long operator++()
-     {
-+#ifdef __ARM_ARCH_7A__
-+       int v1, tmp;
-+       asm volatile ("1:                 \n\t"
-+                     "ldrex   %0, %1     \n\t"
-+                     "add     %0 ,%0, #1 \n\t"
-+                     "strex   %2, %0, %1 \n\t"
-+                     "cmp     %2, #0     \n\t"
-+                     "bne     1b         \n\t"
-+                     : "=&r" (v1), "+Q"(value_), "=&r"(tmp)
-+                    );
-+#else
-         return __sync_add_and_fetch( &value_, 1 );
-+#endif
-     }
- 
-     long operator--()
-     {
-+#ifdef __ARM_ARCH_7A__
-+       int v1, tmp;
-+       asm volatile ("1:                 \n\t"
-+                     "ldrex   %0, %1     \n\t"
-+                     "sub     %0 ,%0, #1 \n\t"
-+                     "strex   %2, %0, %1 \n\t"
-+                     "cmp     %2, #0     \n\t"
-+                     "bne     1b         \n\t"
-+                     : "=&r" (v1), "+Q"(value_), "=&r"(tmp)
-+                    );
-+       return value_;
-+#else
-         return __sync_add_and_fetch( &value_, -1 );
-+#endif
-     }
- 
-     operator long() const
-     {
-+#if __ARM_ARCH_7A__
-+        return value_;
-+#else
-         return __sync_fetch_and_add( &value_, 0 );
-+#endif
-     }
- 
- private:
meta/recipes-support/boost/boost_1.74.0.bb +1 -1
index b01b390a59..9188303195 100644
@@ -1,7 +1,7 @@
 require boost-${PV}.inc
 require boost.inc
 
-SRC_URI += "file://arm-intrinsics.patch \
+SRC_URI += " \
            file://boost-CVE-2012-2677.patch \
            file://boost-math-disable-pch-for-gcc.patch \
            file://0001-Apply-boost-1.62.0-no-forced-flags.patch.patch \
Signature · 1 line
2.29.2
#6 lore raw

PATCH 6/20 binutils: Fix CVE-2020-35448

In reply to Anuj Mittal

From: Yi Fan Yu <[email protected]>

Fix related to a buffer overflow in bfd library

CVE Details https://nvd.nist.gov/vuln/detail/CVE-2020-35448

Upstream Tracking https://sourceware.org/bugzilla/show_bug.cgi?id=26574

Patch from Upstream

    https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;
    h=8642dafaef21aa6747cec01df1977e9c52eb4679
(cherry picked from commit cb83312131f6c4f69d89d639085e07ea1f53167e)
---
 .../binutils/binutils-2.35.1.inc              |  1 +
 .../binutils/binutils/CVE-2020-35448.patch    | 85 +++++++++++++++++++
 2 files changed, 86 insertions(+)
 create mode 100644 meta/recipes-devtools/binutils/binutils/CVE-2020-35448.patch
meta/recipes-devtools/binutils/binutils-2.35.1.inc +1 -0
index c92cb75543..775af2b8f2 100644
@@ -43,5 +43,6 @@ SRC_URI = "\
      file://0016-Check-for-clang-before-checking-gcc-version.patch \
      file://0017-gas-improve-reproducibility-for-stabs-debugging-data.patch \
      file://0001-aarch64-Return-an-error-on-conditional-branch-to-an-.patch \
+     file://CVE-2020-35448.patch \
 "
 S  = "${WORKDIR}/git"
meta/recipes-devtools/binutils/binutils/CVE-2020-35448.patch new file +85 -0

85 lines not on this page · raw message

Signature · 1 line
2.29.2
#7 lore raw

PATCH 7/20 meta: toolchain-shar-relocate.sh: Do not use $target_sdk_dir as regex

In reply to Anuj Mittal

From: Marek Vasut <[email protected]>

The $target_sdk_dir path might contain special characters, for example if the path is /opt/poky/3.2+snapshot . Prevent grep from interpreting those as part of the regex by using the -F parameter and multiple -e parameters to specify which strings to filter out. Also note that the previous regex was using asterisk as wildcard (e.g. environment-setup-*), but that should have been regex (e.g. environment-setup-.*, with dot) to match correctly, this is also fixed by this change.

Fixes 9721378688 ("toolchain-shar-template.sh: Make relocation optional.")
Cc Joshua Watt <[email protected]>
Cc Krzysztof Zawadzki <[email protected]>
Cc Randy Witt <[email protected]>
Cc Richard Purdie <[email protected]>
Cc Ross Burton <[email protected]>
(cherry picked from commit 19d9fa7ab6c851000bc5d24281739e1b2bb8f057)
---
 meta/files/toolchain-shar-relocate.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
meta/files/toolchain-shar-relocate.sh +3 -1
index e3c10018ef..9c358a53e2 100644
@@ -56,7 +56,9 @@ for replace in "$target_sdk_dir -maxdepth 1" "$native_sysroot"; do
 	$SUDO_EXEC find $replace -type f
 done | xargs -n100 file | grep ":.*\(ASCII\|script\|source\).*text" | \
     awk -F':' '{printf "\"%s\"\n", $1}' | \
-    grep -Ev "$target_sdk_dir/(environment-setup-*|relocate_sdk*|${0##*/})" | \
+    grep -Fv -e "$target_sdk_dir/environment-setup-" \
+             -e "$target_sdk_dir/relocate_sdk" \
+             -e "$target_sdk_dir/${0##*/}" | \
     xargs -n100 $SUDO_EXEC sed -i \
         -e "s:$DEFAULT_INSTALL_DIR:$target_sdk_dir:g" \
         -e "s:^#! */usr/bin/perl.*:#! /usr/bin/env perl:g" \
Signature · 1 line
2.29.2
#9 lore raw

PATCH 9/20 lib/oe/utils: Return empty string in parallel_make

In reply to Anuj Mittal

From: Tomasz Dziendzielski <[email protected]>

In cmake.bbclass we set CMAKE_BUILD_PARALLEL_LEVEL using parallel_make function and if PARALLEL_MAKE is set to empty string then this variable is exported as "None" causing cmake to fail with:
"'CMAKE_BUILD_PARALLEL_LEVEL' environment variable
invalid number 'None' given."

(cherry picked from commit 2f790ded554a52ac18d1c28002142f9c62abec8b)
---
 meta/lib/oe/utils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/utils.py +1 -1
index 468c76f30f..9a2187e36f 100644
@@ -193,7 +193,7 @@ def parallel_make(d, makeinst=False):
 
         return int(v)
 
-    return None
+    return ''
 
 def parallel_make_argument(d, fmt, limit=None, makeinst=False):
     """
Signature · 1 line
2.29.2
#10 lore raw

PATCH 10/20 gcc: Backport patch to resolve i*86 tune configuration overrides

In reply to Anuj Mittal

From: Nathan Rossi <[email protected]>

Upstream has merged a change to handle detection of target overrides so that they are only applied as required. Backport this patch from upstream and remove the OE specific patch.

(cherry picked from commit d430f425e1a5725e93833a652e5c6988576f7347)
---
 meta/recipes-devtools/gcc/gcc-10.2.inc        |   2 +-
 ...0001-lib-Remove-i-86-march-overrides.patch | 113 ----------
 ...libgomp-libitc-Fix-bootstrap-PR70454.patch | 208 ++++++++++++++++++
 3 files changed, 209 insertions(+), 114 deletions(-)
 delete mode 100644 meta/recipes-devtools/gcc/gcc/0001-lib-Remove-i-86-march-overrides.patch
 create mode 100644 meta/recipes-devtools/gcc/gcc/0001-libatomic-libgomp-libitc-Fix-bootstrap-PR70454.patch
meta/recipes-devtools/gcc/gcc-10.2.inc +1 -1
index 87dc988d70..82f180db77 100644
@@ -69,7 +69,7 @@ SRC_URI = "\
            file://0002-aarch64-Introduce-SLS-mitigation-for-RET-and-BR-inst.patch \
            file://0003-aarch64-Mitigate-SLS-for-BLR-instruction.patch \
            file://0001-aarch64-Fix-up-__aarch64_cas16_acq_rel-fallback.patch \
-           file://0001-lib-Remove-i-86-march-overrides.patch \
+           file://0001-libatomic-libgomp-libitc-Fix-bootstrap-PR70454.patch \
 "
 SRC_URI[sha256sum] = "b8dd4368bb9c7f0b98188317ee0254dd8cc99d1e3a18d0ff146c855fe16c1d8c"
 
meta/recipes-devtools/gcc/gcc/0001-lib-Remove-i-86-march-overrides.patch deleted +0 -113

113 lines not on this page · raw message

meta/recipes-devtools/gcc/gcc/0001-libatomic-libgomp-libitc-Fix-bootstrap-PR70454.patch new file +208 -0

208 lines not on this page · raw message

Signature · 1 line
2.29.2
#8 lore raw

PATCH 8/20 meta: toolchain-shar-relocate.sh: Filter out post-relocate-setup script

In reply to Anuj Mittal

From: Marek Vasut <[email protected]>

The toolchain-shar-extract.sh script updates the SDK relocation paths in post-relocate-setup.sh, so avoid doing this twice. This is generally not a problem, unless the SDK path is a subset of the SDK relocation path, in which case the resulting path is substituted twice. To trigger the issue,

  $ ./tmp/deploy/sdk/poky-glibc-x86_64-core-image-base-core2-64-qemux86-64-toolchain-3.2+snapshot.sh -y -d /home/oe/.local/opt/poky/3.2+snapshot
which generates relocation path
  /home/oe/.local/home/oe/.local/opt/poky/3.2+snapshot
instead of
  /home/oe/.local/opt/poky/3.2+snapshot
Fixes 93ec145f42 ("toolchain-shar-extract: Add post-relocate scripts")
Cc Joshua Watt <[email protected]>
Cc Krzysztof Zawadzki <[email protected]>
Cc Randy Witt <[email protected]>
Cc Richard Purdie <[email protected]>
Cc Ross Burton <[email protected]>
(cherry picked from commit 5000aabe6ac336e7b424dafa1bf76271dee6a6f1)
---
 meta/files/toolchain-shar-relocate.sh | 1 +
 1 file changed, 1 insertion(+)
meta/files/toolchain-shar-relocate.sh +1 -0
index 9c358a53e2..94d288ce05 100644
@@ -58,6 +58,7 @@ done | xargs -n100 file | grep ":.*\(ASCII\|script\|source\).*text" | \
     awk -F':' '{printf "\"%s\"\n", $1}' | \
     grep -Fv -e "$target_sdk_dir/environment-setup-" \
              -e "$target_sdk_dir/relocate_sdk" \
+             -e "$target_sdk_dir/post-relocate-setup" \
              -e "$target_sdk_dir/${0##*/}" | \
     xargs -n100 $SUDO_EXEC sed -i \
         -e "s:$DEFAULT_INSTALL_DIR:$target_sdk_dir:g" \
Signature · 1 line
2.29.2
#11 lore raw

PATCH 11/20 wic: Add workdir argument

In reply to Anuj Mittal

From: Paul Barker <[email protected]>

This allows the path for the temporary workdir used by wic to be set when running wic from bitbake or directly from the command line.

(cherry picked from commit 2e7314ac3a3cab89de93c932e8efc75de0feb0a7)
---
 scripts/lib/wic/plugins/imager/direct.py | 2 +-
 scripts/wic                              | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)
scripts/lib/wic/plugins/imager/direct.py +1 -1
index 0ca67b77d5..b329568c7a 100644
@@ -62,7 +62,7 @@ class DirectPlugin(ImagerPlugin):
 
         self.name = "%s-%s" % (os.path.splitext(os.path.basename(wks_file))[0],
                                strftime("%Y%m%d%H%M"))
-        self.workdir = tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.')
+        self.workdir = options.workdir or tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.')
         self._image = None
         self.ptable_format = self.ks.bootloader.ptable
         self.parts = self.ks.partitions
scripts/wic +2 -0
index 24700f380f..a741aed364 100755
@@ -312,6 +312,8 @@ def wic_init_parser_create(subparser):
 
     subparser.add_argument("-o", "--outdir", dest="outdir", default='.',
                       help="name of directory to create image in")
+    subparser.add_argument("-w", "--workdir",
+                      help="temporary workdir to use for intermediate files")
     subparser.add_argument("-e", "--image-name", dest="image_name",
                       help="name of the image to use the artifacts from "
                            "e.g. core-image-sato")
Signature · 1 line
2.29.2
#12 lore raw

PATCH 12/20 wic: Ensure internal workdir is not reused

In reply to Anuj Mittal

From: Paul Barker <[email protected]>

If a path is specified for the internal wic working directory using the -w/--workdir argument then it must not already exist. Re-using a previous workdir could easily result in rootfs and intermediate files from a previous build being added to the current image.

(cherry picked from commit 2e40c8d4109024ff704c5ce40d98050ca7f34dd5)
---
 scripts/lib/wic/plugins/imager/direct.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
scripts/lib/wic/plugins/imager/direct.py +11 -1
index b329568c7a..f107e60089 100644
@@ -62,7 +62,7 @@ class DirectPlugin(ImagerPlugin):
 
         self.name = "%s-%s" % (os.path.splitext(os.path.basename(wks_file))[0],
                                strftime("%Y%m%d%H%M"))
-        self.workdir = options.workdir or tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.')
+        self.workdir = self.setup_workdir(options.workdir)
         self._image = None
         self.ptable_format = self.ks.bootloader.ptable
         self.parts = self.ks.partitions
@@ -78,6 +78,16 @@ class DirectPlugin(ImagerPlugin):
         self._image = PartitionedImage(image_path, self.ptable_format,
                                        self.parts, self.native_sysroot)
 
+    def setup_workdir(self, workdir):
+        if workdir:
+            if os.path.exists(workdir):
+                raise WicError("Internal workdir '%s' specified in wic arguments already exists!" % (workdir))
+
+            os.makedirs(workdir)
+            return workdir
+        else:
+            return tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.')
+
     def do_create(self):
         """
         Plugin entry point.
Signature · 1 line
2.29.2
#13 lore raw

PATCH 13/20 wic: Allow exec_native_cmd to run HOSTTOOLS

In reply to Anuj Mittal

From: Paul Barker <[email protected]>

This allows programs from HOSTTOOLS (e.g. 'install', 'rm', 'mv', etc) to be more easily executed by wic. Without this change only programs from an actual *-native recipe built by bitbake can be executed by wic.

(cherry picked from commit 8eb186acdecfbb3151c9a0ab148358e3fe5cce39)
---
 meta/classes/image_types_wic.bbclass | 2 +-
 scripts/lib/wic/misc.py              | 7 +++++--
 2 files changed, 6 insertions(+), 3 deletions(-)
meta/classes/image_types_wic.bbclass +1 -1
index 286e0f5d54..000ee4249f 100644
@@ -3,7 +3,7 @@
 WICVARS ?= "\
            BBLAYERS IMGDEPLOYDIR DEPLOY_DIR_IMAGE FAKEROOTCMD IMAGE_BASENAME IMAGE_EFI_BOOT_FILES IMAGE_BOOT_FILES \
            IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD INITRD_LIVE ISODIR RECIPE_SYSROOT_NATIVE \
-           ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS \
+           ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS HOSTTOOLS_DIR \
            KERNEL_IMAGETYPE MACHINE INITRAMFS_IMAGE INITRAMFS_IMAGE_BUNDLE INITRAMFS_LINK_NAME APPEND \
            ASSUME_PROVIDED PSEUDO_IGNORE_PATHS"
 
scripts/lib/wic/misc.py +5 -2
index 4b08d649c6..75b219cd3f 100644
@@ -138,9 +138,12 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
     if pseudo:
         cmd_and_args = pseudo + cmd_and_args
 
-    native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin" % \
+    hosttools_dir = get_bitbake_var("HOSTTOOLS_DIR")
+
+    native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin:%s" % \
                    (native_sysroot, native_sysroot,
-                    native_sysroot, native_sysroot)
+                    native_sysroot, native_sysroot,
+                    hosttools_dir)
 
     native_cmd_and_args = "export PATH=%s:$PATH;%s" % \
                    (native_paths, cmd_and_args)
Signature · 1 line
2.29.2
#14 lore raw

PATCH 14/20 image_types_wic: Move wic working directory

In reply to Anuj Mittal

From: Paul Barker <[email protected]>

By default the wic working directory is placed under the output directory. When invoking wic under bitbake, the wic output directory is added to PSEUDO_PATHS_IGNORE to avoid issues with files being removed from outside a pseudo environment (see oe-core commit ad8f5532ff).

However, wic will copy the rootfs directory into its working directory if it needs to add or remove content before creating a filesystem image. This copy of the rootfs directory must be tracked by pseudo in order to keep the permissions correct in the resulting image. So we can't have the wic working directory under a path in PSEUDO_PATHS_IGNORE unless we like broken permissions.

To fix this the new '-w' argument to wic is used to move the working directory away from the output directory.

Note that wic deletes the temporary working directory automatically when it finishes creating an image so users won't normally see the 'tmp-wic' directory under WORKDIR.

Fixes [Yocto #14129]

(cherry picked from commit 4d07169499c47fa9dc759e6f81843416a6dc25c5)
---
 meta/classes/image_types_wic.bbclass | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
meta/classes/image_types_wic.bbclass +7 -1
index 000ee4249f..49be1da77a 100644
@@ -29,11 +29,17 @@ WIC_CREATE_EXTRA_ARGS ?= ""
 IMAGE_CMD_wic () {
 	out="${IMGDEPLOYDIR}/${IMAGE_NAME}"
 	build_wic="${WORKDIR}/build-wic"
+	tmp_wic="${WORKDIR}/tmp-wic"
 	wks="${WKS_FULL_PATH}"
+	if [ -e "$tmp_wic" ]; then
+		# Ensure we don't have any junk leftover from a previously interrupted
+		# do_image_wic execution
+		rm -rf "$tmp_wic"
+	fi
 	if [ -z "$wks" ]; then
 		bbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
 	fi
-	BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" ${WIC_CREATE_EXTRA_ARGS}
+	BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" -w "$tmp_wic" ${WIC_CREATE_EXTRA_ARGS}
 	mv "$build_wic/$(basename "${wks%.wks}")"*.direct "$out${IMAGE_NAME_SUFFIX}.wic"
 }
 IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
Signature · 1 line
2.29.2
#15 lore raw

PATCH 15/20 wic: Update pseudo db when excluding content from rootfs

In reply to Anuj Mittal

From: Paul Barker <[email protected]>

To exclude content from the rootfs, wic makes a copy (using hardlinks if possible) of the rootfs directory and associated pseudo db, then removes files & directories as needed. However if these files and directories are removed using the python functions os.remove and shutil.rmtree, the copied pseudo db will not be updated correctly. For files copied from the original rootfs, if hardlinks were used successfully when copying the rootfs this should mean that the relevant inodes can't be reused and so the risk of pseudo aborts should be avoided. However, this logic doesn't apply for directories (as they can't be hardlinked) or for files added via the '--include-path' argument (as they weren't present in the original rootfs) and so there remains some risk of inodes being reused and the pseudo db becoming corrupted.

To fix this, use the 'rm' command under pseudo when removing files & directories from the copied rootfs to ensure that the copied pseudo db is updated.

(cherry picked from commit d5db7e268947f0392c2126137571a44acd29ccd6)
---
 scripts/lib/wic/plugins/source/rootfs.py | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)
scripts/lib/wic/plugins/source/rootfs.py +8 -5
index f1db83f8a1..afb1eb9202 100644
@@ -199,17 +199,20 @@ class RootfsPlugin(SourcePlugin):
                 if not os.path.lexists(full_path):
                     continue
 
+                if new_pseudo:
+                    pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)
+                else:
+                    pseudo = None
                 if path.endswith(os.sep):
                     # Delete content only.
                     for entry in os.listdir(full_path):
                         full_entry = os.path.join(full_path, entry)
-                        if os.path.isdir(full_entry) and not os.path.islink(full_entry):
-                            shutil.rmtree(full_entry)
-                        else:
-                            os.remove(full_entry)
+                        rm_cmd = "rm -rf %s" % (full_entry)
+                        exec_native_cmd(rm_cmd, native_sysroot, pseudo)
                 else:
                     # Delete whole directory.
-                    shutil.rmtree(full_path)
+                    rm_cmd = "rm -rf %s" % (full_path)
+                    exec_native_cmd(rm_cmd, native_sysroot, pseudo)
 
         part.prepare_rootfs(cr_workdir, oe_builddir,
                             new_rootfs or part.rootfs_dir, native_sysroot,
Signature · 1 line
2.29.2
#16 lore raw

PATCH 16/20 wic: Copy rootfs dir if fstab needs updating

In reply to Anuj Mittal

From: Paul Barker <[email protected]>

By default, wic updates the /etc/fstab in the rootfs to include details of additional partitions described in the selected wks file. If this modification is performed in place, other tasks which create an image file from the rootfs directory (e.g. do_image_tar and do_image_ext4) will pick up the modified fstab file which would not be appropriate for those images as they do not include the additional partitions described in the wks file. wic does undo modifications to the fstab file once it has finished creating the filesystem image, however this leaves open a race condition if one of the other tasks reads the contents of the fstab file from the rootfs directory between the point where wic modifies the fstab file and the point where wic restores the files original content.

This could be solved by adding a lockfile for tasks which use the rootfs directory to ensure that no other such task is reading the rootfs directory while do_image_wic is running. This would serialize several do_image_* tasks and result in slower builds, especially for large images. Another drawback of this solution is that it is hard to selectively optimise - adding lockfiles to do_image_* tasks would result in these tasks always being serialized even if no fstab modification will take place.

An alternative solution is to copy the rootfs directory when fstab needs to be modified. The code to do this in wic already exists as it is needed when including or excluding content in the rootfs. This still results in an impact on build times but the copy uses hardlinks if possible (so little data is actually copied) and we can make selective optimisations to improve things. The rootfs copy will only take place if fstab modification is required (or if it was already needed to include or exclude rootfs content). We can also follow up with further optimisations after this commit. So this second solution is chosen.

Fixes [Yocto #13994]

(cherry picked from commit ce682a73b7447652f898ce1d1d0416a456df5416)
---
 scripts/lib/wic/partition.py             |  5 +++-
 scripts/lib/wic/plugins/imager/direct.py | 36 ++++++++----------------
 scripts/lib/wic/plugins/source/rootfs.py | 17 +++++++++--
 3 files changed, 29 insertions(+), 29 deletions(-)
scripts/lib/wic/partition.py +4 -1
index 286c7867cb..f59eceb23d 100644
@@ -54,6 +54,7 @@ class Partition():
         self.uuid = args.uuid
         self.fsuuid = args.fsuuid
         self.type = args.type
+        self.updated_fstab_path = None
 
         self.lineno = lineno
         self.source_file = ""
@@ -118,11 +119,13 @@ class Partition():
         return self.fixed_size if self.fixed_size else self.size
 
     def prepare(self, creator, cr_workdir, oe_builddir, rootfs_dir,
-                bootimg_dir, kernel_dir, native_sysroot):
+                bootimg_dir, kernel_dir, native_sysroot, updated_fstab_path):
         """
         Prepare content for individual partitions, depending on
         partition command parameters.
         """
+        self.updated_fstab_path = updated_fstab_path
+
         if not self.source:
             if not self.size and not self.fixed_size:
                 raise WicError("The %s partition has a size of zero. Please "
scripts/lib/wic/plugins/imager/direct.py +11 -25
index f107e60089..7e1c1c03ab 100644
@@ -58,7 +58,7 @@ class DirectPlugin(ImagerPlugin):
         self.compressor = options.compressor
         self.bmap = options.bmap
         self.no_fstab_update = options.no_fstab_update
-        self.original_fstab = None
+        self.updated_fstab_path = None
 
         self.name = "%s-%s" % (os.path.splitext(os.path.basename(wks_file))[0],
                                strftime("%Y%m%d%H%M"))
@@ -100,11 +100,8 @@ class DirectPlugin(ImagerPlugin):
         finally:
             self.cleanup()
 
-    def _write_fstab(self, image_rootfs):
-        """overriden to generate fstab (temporarily) in rootfs. This is called
-        from _create, make sure it doesn't get called from
-        BaseImage.create()
-        """
+    def update_fstab(self, image_rootfs):
+        """Assume partition order same as in wks"""
         if not image_rootfs:
             return
 
@@ -114,18 +111,9 @@ class DirectPlugin(ImagerPlugin):
 
         with open(fstab_path) as fstab:
             fstab_lines = fstab.readlines()
-            self.original_fstab = fstab_lines.copy()
-
-        if self._update_fstab(fstab_lines, self.parts):
-            with open(fstab_path, "w") as fstab:
-                fstab.writelines(fstab_lines)
-        else:
-            self.original_fstab = None
 
-    def _update_fstab(self, fstab_lines, parts):
-        """Assume partition order same as in wks"""
         updated = False
-        for part in parts:
+        for part in self.parts:
             if not part.realnum or not part.mountpoint \
                or part.mountpoint == "/":
                 continue
@@ -154,7 +142,10 @@ class DirectPlugin(ImagerPlugin):
             fstab_lines.append(line)
             updated = True
 
-        return updated
+        if updated:
+            self.updated_fstab_path = os.path.join(self.workdir, "fstab")
+            with open(self.updated_fstab_path, "w") as f:
+                f.writelines(fstab_lines)
 
     def _full_path(self, path, name, extention):
         """ Construct full file path to a file we generate. """
@@ -170,7 +161,7 @@ class DirectPlugin(ImagerPlugin):
         a partitioned image.
         """
         if not self.no_fstab_update:
-            self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
+            self.update_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
 
         for part in self.parts:
             # get rootfs size from bitbake variable if it's not set in .ks file
@@ -283,12 +274,6 @@ class DirectPlugin(ImagerPlugin):
             if os.path.isfile(path):
                 shutil.move(path, os.path.join(self.outdir, fname))
 
-        #Restore original fstab
-        if self.original_fstab:
-            fstab_path = self.rootfs_dir.get("ROOTFS_DIR") + "/etc/fstab"
-            with open(fstab_path, "w") as fstab:
-                fstab.writelines(self.original_fstab)
-
         # remove work directory
         shutil.rmtree(self.workdir, ignore_errors=True)
 
@@ -368,7 +353,8 @@ class PartitionedImage():
             # sizes before we can add them and do the layout.
             part.prepare(imager, imager.workdir, imager.oe_builddir,
                          imager.rootfs_dir, imager.bootimg_dir,
-                         imager.kernel_dir, imager.native_sysroot)
+                         imager.kernel_dir, imager.native_sysroot,
+                         imager.updated_fstab_path)
 
             # Converting kB to sectors for parted
             part.size_sec = part.disk_size * 1024 // self.sector_size
scripts/lib/wic/plugins/source/rootfs.py +14 -3
index afb1eb9202..6fd415af5b 100644
@@ -103,9 +103,9 @@ class RootfsPlugin(SourcePlugin):
         new_rootfs = None
         new_pseudo = None
         # Handle excluded paths.
-        if part.exclude_path or part.include_path or part.change_directory:
-            # We need a new rootfs directory we can delete files from. Copy to
-            # workdir.
+        if part.exclude_path or part.include_path or part.change_directory or part.updated_fstab_path:
+            # We need a new rootfs directory we can safely modify without
+            # interfering with other tasks. Copy to workdir.
             new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
 
             if os.path.lexists(new_rootfs):
@@ -214,6 +214,17 @@ class RootfsPlugin(SourcePlugin):
                     rm_cmd = "rm -rf %s" % (full_path)
                     exec_native_cmd(rm_cmd, native_sysroot, pseudo)
 
+            has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab"))
+            if part.updated_fstab_path and has_fstab:
+                fstab_path = os.path.join(new_rootfs, "etc/fstab")
+                # Assume that fstab should always be owned by root with fixed permissions
+                install_cmd = "install -m 0644 %s %s" % (part.updated_fstab_path, fstab_path)
+                if new_pseudo:
+                    pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)
+                else:
+                    pseudo = None
+                exec_native_cmd(install_cmd, native_sysroot, pseudo)
+
         part.prepare_rootfs(cr_workdir, oe_builddir,
                             new_rootfs or part.rootfs_dir, native_sysroot,
                             pseudo_dir = new_pseudo or pseudo_dir)
Signature · 1 line
2.29.2
#17 lore raw

PATCH 17/20 wic: Optimise fstab modification for ext2/3/4 and msdos partitions

In reply to Anuj Mittal

From: Paul Barker <[email protected]>

The fix for [Yocto #13994] required the rootfs directory to be copied (using hardlinks if possible) when modifying the fstab file under wic.

We can optimise this copy away for filesystems where we have the tools to modify the contents of the partition image after it is created. For ext2/3/4 filesystems we have the debugfs tool and for msdos/vfat filesystems we have the mcopy tool. So for any of these filesystems we skip the modification of the fstab file in the rootfs directory (and skip the associated copy unless it is otherwise necessary) and update the contents of fstab directly in the partition image.

(cherry picked from commit 5fb8ae0e9159597d7eaa9307a3a8543800bf9405)
---
 scripts/lib/wic/partition.py             | 27 +++++++++++++++++++-----
 scripts/lib/wic/plugins/source/rootfs.py |  9 +++++---
 2 files changed, 28 insertions(+), 8 deletions(-)
scripts/lib/wic/partition.py +22 -5
index f59eceb23d..e574f40c47 100644
@@ -55,6 +55,8 @@ class Partition():
         self.fsuuid = args.fsuuid
         self.type = args.type
         self.updated_fstab_path = None
+        self.has_fstab = False
+        self.update_fstab_in_rootfs = False
 
         self.lineno = lineno
         self.source_file = ""
@@ -125,6 +127,8 @@ class Partition():
         partition command parameters.
         """
         self.updated_fstab_path = updated_fstab_path
+        if self.updated_fstab_path and not (self.fstype.startswith("ext") or self.fstype == "msdos"):
+            self.update_fstab_in_rootfs = True
 
         if not self.source:
             if not self.size and not self.fixed_size:
@@ -250,7 +254,7 @@ class Partition():
 
         prefix = "ext" if self.fstype.startswith("ext") else self.fstype
         method = getattr(self, "prepare_rootfs_" + prefix)
-        method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo)
+        method(rootfs, cr_workdir, oe_builddir, rootfs_dir, native_sysroot, pseudo)
         self.source_file = rootfs
 
         # get the rootfs size in the right units for kickstart (kB)
@@ -258,7 +262,7 @@ class Partition():
         out = exec_cmd(du_cmd)
         self.size = int(out.split()[0])
 
-    def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir,
+    def prepare_rootfs_ext(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
                            native_sysroot, pseudo):
         """
         Prepare content for an ext2/3/4 rootfs partition.
@@ -282,10 +286,19 @@ class Partition():
             (self.fstype, extraopts, rootfs, label_str, self.fsuuid, rootfs_dir)
         exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
 
+        if self.updated_fstab_path and self.has_fstab:
+            debugfs_script_path = os.path.join(cr_workdir, "debugfs_script")
+            with open(debugfs_script_path, "w") as f:
+                f.write("cd etc\n")
+                f.write("rm fstab\n")
+                f.write("write %s fstab\n" % (self.updated_fstab_path))
+            debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs)
+            exec_native_cmd(debugfs_cmd, native_sysroot)
+
         mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
         exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
 
-    def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir,
+    def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
                              native_sysroot, pseudo):
         """
         Prepare content for a btrfs rootfs partition.
@@ -308,7 +321,7 @@ class Partition():
              self.mkfs_extraopts, self.fsuuid, rootfs)
         exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
 
-    def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir,
+    def prepare_rootfs_msdos(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
                              native_sysroot, pseudo):
         """
         Prepare content for a msdos/vfat rootfs partition.
@@ -337,12 +350,16 @@ class Partition():
         mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir)
         exec_native_cmd(mcopy_cmd, native_sysroot)
 
+        if self.updated_fstab_path and self.has_fstab:
+            mcopy_cmd = "mcopy -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path)
+            exec_native_cmd(mcopy_cmd, native_sysroot)
+
         chmod_cmd = "chmod 644 %s" % rootfs
         exec_cmd(chmod_cmd)
 
     prepare_rootfs_vfat = prepare_rootfs_msdos
 
-    def prepare_rootfs_squashfs(self, rootfs, oe_builddir, rootfs_dir,
+    def prepare_rootfs_squashfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
                                 native_sysroot, pseudo):
         """
         Prepare content for a squashfs rootfs partition.
scripts/lib/wic/plugins/source/rootfs.py +6 -3
index 6fd415af5b..96d940a91d 100644
@@ -94,6 +94,7 @@ class RootfsPlugin(SourcePlugin):
                                "it is not a valid path, exiting" % part.rootfs_dir)
 
         part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+        part.has_fstab = os.path.exists(os.path.join(part.rootfs_dir, "etc/fstab"))
         pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
         if not os.path.lexists(pseudo_dir):
             logger.warn("%s folder does not exist. "
@@ -103,7 +104,7 @@ class RootfsPlugin(SourcePlugin):
         new_rootfs = None
         new_pseudo = None
         # Handle excluded paths.
-        if part.exclude_path or part.include_path or part.change_directory or part.updated_fstab_path:
+        if part.exclude_path or part.include_path or part.change_directory or part.update_fstab_in_rootfs:
             # We need a new rootfs directory we can safely modify without
             # interfering with other tasks. Copy to workdir.
             new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
@@ -214,8 +215,10 @@ class RootfsPlugin(SourcePlugin):
                     rm_cmd = "rm -rf %s" % (full_path)
                     exec_native_cmd(rm_cmd, native_sysroot, pseudo)
 
-            has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab"))
-            if part.updated_fstab_path and has_fstab:
+            # Update part.has_fstab here as fstab may have been added or
+            # removed by the above modifications.
+            part.has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab"))
+            if part.update_fstab_in_rootfs and part.has_fstab:
                 fstab_path = os.path.join(new_rootfs, "etc/fstab")
                 # Assume that fstab should always be owned by root with fixed permissions
                 install_cmd = "install -m 0644 %s %s" % (part.updated_fstab_path, fstab_path)
Signature · 1 line
2.29.2
#18 lore raw

PATCH 18/20 toolchain-shar-relocate.sh: Fix handling files with colons

In reply to Anuj Mittal

From: "Oleksiy Obitotskyy yIEf0zt.mo" <[email protected]>

Files could contain colons in name and we should not use colons (':') as field separator. E.g. perl/python man pages packages.

(cherry picked from commit 50d2ed0689f1aed6f33b4992d37e2e991c99eb07)
---
 meta/files/toolchain-shar-relocate.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/files/toolchain-shar-relocate.sh +1 -1
index 94d288ce05..5433741296 100644
@@ -55,7 +55,7 @@ fi
 for replace in "$target_sdk_dir -maxdepth 1" "$native_sysroot"; do
 	$SUDO_EXEC find $replace -type f
 done | xargs -n100 file | grep ":.*\(ASCII\|script\|source\).*text" | \
-    awk -F':' '{printf "\"%s\"\n", $1}' | \
+    awk -F': ' '{printf "\"%s\"\n", $1}' | \
     grep -Fv -e "$target_sdk_dir/environment-setup-" \
              -e "$target_sdk_dir/relocate_sdk" \
              -e "$target_sdk_dir/post-relocate-setup" \
Signature · 1 line
2.29.2
#19 lore raw

PATCH 19/20 devtool: Fix source extraction for gcc shared source

In reply to Anuj Mittal

From: Tomasz Dziendzielski <[email protected]>

If do_patch task is disabled then prepare do_configure dependencies to fetch external sources and create symlink to ${S} in devtool workspace.

[YOCTO #13036]

(cherry picked from commit 9e47319bfe62d289b90f7545a64dbdc1cbde7f1d)
---
 scripts/lib/devtool/standard.py | 30 ++++++++++++++++++++++++++----
 1 file changed, 26 insertions(+), 4 deletions(-)
scripts/lib/devtool/standard.py +26 -4
index 261d642d4a..f7d8a82117 100644
@@ -589,6 +589,16 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
             else:
                 task = 'do_patch'
 
+                if 'noexec' in (d.getVarFlags(task, False) or []) or 'task' not in (d.getVarFlags(task, False) or []):
+                    logger.info('The %s recipe has %s disabled. Running only '
+                                       'do_configure task dependencies' % (pn, task))
+
+                    if 'depends' in d.getVarFlags('do_configure', False):
+                        pn = d.getVarFlags('do_configure', False)['depends']
+                        pn = pn.replace('${PV}', d.getVar('PV'))
+                        pn = pn.replace('${COMPILERDEP}', d.getVar('COMPILERDEP'))
+                        task = None
+
             # Run the fetch + unpack tasks
             res = tinfoil.build_targets(pn,
                                         task,
@@ -600,6 +610,17 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
         if not res:
             raise DevtoolError('Extracting source for %s failed' % pn)
 
+        if not is_kernel_yocto and ('noexec' in (d.getVarFlags('do_patch', False) or []) or 'task' not in (d.getVarFlags('do_patch', False) or [])):
+            workshareddir = d.getVar('S')
+            if os.path.islink(srctree):
+                os.unlink(srctree)
+
+            os.symlink(workshareddir, srctree)
+
+            # The initial_rev file is created in devtool_post_unpack function that will not be executed if
+            # do_unpack/do_patch tasks are disabled so we have to directly say that source extraction was successful
+            return True, True
+
         try:
             with open(os.path.join(tempdir, 'initial_rev'), 'r') as f:
                 initial_rev = f.read()
@@ -847,10 +868,11 @@ def modify(args, config, basepath, workspace):
             if not initial_rev:
                 return 1
             logger.info('Source tree extracted to %s' % srctree)
-            # Get list of commits since this revision
-            (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree)
-            commits = stdout.split()
-            check_commits = True
+            if os.path.exists(os.path.join(srctree, '.git')):
+                # Get list of commits since this revision
+                (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree)
+                commits = stdout.split()
+                check_commits = True
         else:
             if os.path.exists(os.path.join(srctree, '.git')):
                 # Check if it's a tree previously extracted by us. This is done
Signature · 1 line
2.29.2
#20 lore raw

PATCH 20/20 flex: Fix --noline option behavior

In reply to Anuj Mittal

From: Oleksiy Obitotskyy <[email protected]>

Option --noline or -L does not handled properly.
So generated code contains #line directives with
file absolute path and prevents to create
reproducible builds.

(cherry picked from commit 0bf2cb7dc2123f220accf1542c2ae4c4b4b8275a)
---
 ...directives-if-gen_line_dirs-is-false.patch | 32 +++++++++++++++++++
 meta/recipes-devtools/flex/flex_2.6.4.bb      |  1 +
 2 files changed, 33 insertions(+)
 create mode 100644 meta/recipes-devtools/flex/flex/0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch
meta/recipes-devtools/flex/flex/0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch new file +32 -0
new file mode 100644
index 0000000000..c8202b6bd5
@@ -0,0 +1,32 @@
+From 440f3f55739468cd26e22f31871eca8cbbd53294 Mon Sep 17 00:00:00 2001
+From: Oleksiy Obitotskyy <[email protected]>
+Date: Wed, 6 Jan 2021 06:12:14 -0800
+Subject: [PATCH] Emit no #line directives if gen_line_dirs is false
+
+If we set --noline we should not print line directives.
+But setting --noline means gen_line_dirs is false.
+
+Upstream-Status: Submitted
+Signed-off-by: Oleksiy Obitotskyy <[email protected]>
+---
+ src/buf.c | 4 ++--
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+
+diff --git a/src/buf.c b/src/buf.c
+index 185083c..4439e28 100644
+--- a/src/buf.c
++++ b/src/buf.c
+@@ -95,8 +95,8 @@ struct Buf *buf_linedir (struct Buf *buf, const char* filename, int lineno)
+     const char *src;
+     size_t tsz;
+ 
+-    if (gen_line_dirs)
+-	return buf;
++    if (!gen_line_dirs)
++        return buf;
+ 
+     tsz = strlen("#line \"\"\n")                +   /* constant parts */
+                2 * strlen (filename)            +   /* filename with possibly all backslashes escaped */
+-- 
+2.26.2.Cisco
+
meta/recipes-devtools/flex/flex_2.6.4.bb +1 -0
index 3d57572865..1d43d2228a 100644
@@ -16,6 +16,7 @@ SRC_URI = "https://github.com/westes/flex/releases/download/v${PV}/flex-${PV}.ta
            ${@bb.utils.contains('PTEST_ENABLED', '1', '', 'file://disable-tests.patch', d)} \
            file://0001-build-AC_USE_SYSTEM_EXTENSIONS-in-configure.ac.patch \
            file://check-funcs.patch \
+           file://0001-Emit-no-line-directives-if-gen_line_dirs-is-false.patch \
            "
 
 SRC_URI[md5sum] = "2882e3179748cc9f9c23ec593d6adc8d"
Signature · 1 line
2.29.2