overlayfs: Fix QA failure
PATCH v2 1/4 overlayfs: Fix QA failure
From: Vyacheslav Yurkov <[email protected]>
The QA check for skipped mounts happens twice, at parsing stage and at rootfs postprocessing. Fix the former, because it didn't happen properly.
--- meta/lib/oe/overlayfs.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/lib/oe/overlayfs.py +2 -1
@@ -36,7 +36,8 @@ def unitFileList(d): # check that we have required mount points set first requiredMountPoints = d.getVarFlags('OVERLAYFS_WRITABLE_PATHS') for mountPoint in requiredMountPoints: - if mountPoint not in overlayMountPoints: + qaSkip = (d.getVarFlag("OVERLAYFS_QA_SKIP", mountPoint) or "").split() + if mountPoint not in overlayMountPoints and not "mount-configured" in qaSkip: bb.fatal("Missing required mount point for OVERLAYFS_MOUNT_POINT[%s] in your MACHINE configuration" % mountPoint) for mountPoint in overlayMountPoints:
PATCH v2 2/4 overlayfs: oe-selftest: Make the test more deterministic
From: Vyacheslav Yurkov <[email protected]>
The test orignally was written under assumption that poky distro is used. When poky-altcft is used for example, then systemd is already set in DISTRO_FETURES, which the test did not expect.
--- meta/lib/oeqa/selftest/cases/overlayfs.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
meta/lib/oeqa/selftest/cases/overlayfs.py +3 -2
@@ -5,7 +5,7 @@ # from oeqa.selftest.case import OESelftestTestCase -from oeqa.utils.commands import bitbake, runqemu, get_bb_vars +from oeqa.utils.commands import bitbake, runqemu, get_bb_vars, get_bb_var from oeqa.core.decorator import OETestTag from oeqa.core.decorator.data import skipIfNotMachine
@@ -46,7 +46,8 @@ inherit overlayfs res = bitbake('core-image-minimal', ignore_status=True) line = getline(res, "overlayfs-user was skipped: missing required distro features") self.assertTrue("overlayfs" in res.output, msg=res.output) - self.assertTrue("systemd" in res.output, msg=res.output) + if not "systemd" in get_bb_var('DISTRO_FEATURES'): + self.assertTrue("systemd" in res.output, msg=res.output) self.assertTrue("ERROR: Required build target 'core-image-minimal' has no buildable providers." in res.output, msg=res.output) def test_not_all_units_installed(self):
Re: [OE-core] [PATCH v2 2/4] overlayfs: oe-selftest: Make the test more deterministic
Quoted 5 lines · Vyacheslav Yurkov via lists.openembedded.org
From: Vyacheslav Yurkov <[email protected]>
The test orignally was written under assumption that poky distro is used. When poky-altcft is used for example, then systemd is already set in
DISTRO_FETURES, which the test did not expect.
The commit title says this is about determinism, but it's actually about supporting init systems other than systemd in the test case. I also think "selftest" should come before the component name, so perhaps it should be:
selftest: overlayfs: Support non-systemd init systems
Quoted 20 lines
Signed-off-by Vyacheslav Yurkov <[email protected]>--- meta/lib/oeqa/selftest/cases/overlayfs.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)meta/lib/oeqa/selftest/cases/overlayfs.py +1
-1@@ -5,7 +5,7 @@ # from oeqa.selftest.case import OESelftestTestCase -from oeqa.utils.commands import bitbake, runqemu, get_bb_vars +from oeqa.utils.commands import bitbake, runqemu, get_bb_vars, get_bb_var from oeqa.core.decorator import OETestTag from oeqa.core.decorator.data import skipIfNotMachine
meta/lib/oeqa/selftest/cases/overlayfs.py inherit overlayfs · line 50
Quoted 6 lines
meta/lib/oeqa/selftest/cases/overlayfs.py +2
-1@@ -46,7 +46,8 @@ inherit overlayfs res = bitbake('core-image-minimal', ignore_status=True) line = getline(res, "overlayfs-user was skipped: missing required distro features") self.assertTrue("overlayfs" in res.output, msg=res.output) - self.assertTrue("systemd" in res.output, msg=res.output) + if not "systemd" in get_bb_var('DISTRO_FEATURES'): + self.assertTrue("systemd" in res.output, msg=res.output)
The inverse logic here confused a few people on the patch review call. Could we add a comment at the same time, something like:
# If systemd is not enabled, we should see it listed as a missing
# distro feature
That is just re-stating what the code does, but I think it's worthwhile here to avoid confusion.
Best regards,
Signature · 1 line
Paul Barker
Re: [OE-core] [PATCH v2 2/4] overlayfs: oe-selftest: Make the test more deterministic
Quoted 7 lines · Paul Barker
Quoted 5 lines · Vyacheslav Yurkov via lists.openembedded.org
From: Vyacheslav Yurkov <[email protected]>
The test orignally was written under assumption that poky distro is used. When poky-altcft is used for example, then systemd is already set in
DISTRO_FETURES, which the test did not expect.
Quoted 6 lines
The commit title says this is about determinism, but it's actually about supporting init systems other than systemd in the test case. I also think "selftest" should come before the component name, so perhaps it should be:
selftest: overlayfs: Support non-systemd init systems
The commit has nothing to do with non-systemd init systems, because the whole feature only works with systemd.
Quoted 29 lines
Quoted 20 lines
Signed-off-by Vyacheslav Yurkov <[email protected]>--- meta/lib/oeqa/selftest/cases/overlayfs.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)meta/lib/oeqa/selftest/cases/overlayfs.py +1
-1@@ -5,7 +5,7 @@ # from oeqa.selftest.case import OESelftestTestCase -from oeqa.utils.commands import bitbake, runqemu, get_bb_vars +from oeqa.utils.commands import bitbake, runqemu, get_bb_vars, get_bb_var from oeqa.core.decorator import OETestTag from oeqa.core.decorator.data import skipIfNotMachinemeta/lib/oeqa/selftest/cases/overlayfs.py inherit overlayfs · line 50
Quoted 6 lines
meta/lib/oeqa/selftest/cases/overlayfs.py +2
-1@@ -46,7 +46,8 @@ inherit overlayfs res = bitbake('core-image-minimal', ignore_status=True) line = getline(res, "overlayfs-user was skipped: missing required distro features") self.assertTrue("overlayfs" in res.output, msg=res.output) - self.assertTrue("systemd" in res.output, msg=res.output) + if not "systemd" in get_bb_var('DISTRO_FEATURES'): + self.assertTrue("systemd" in res.output, msg=res.output)The inverse logic here confused a few people on the patch review call. Could we add a comment at the same time, something like:
Quoted 5 lines
# If systemd is not enabled, we should see it listed as a missing
# distro featureThat is just re-stating what the code does, but I think it's worthwhile here to avoid confusion.
This is already written a few lines above in the test description.
Quoted 3 lines
Best regards,
Thanks,
Slava
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [OE-core] [PATCH v2 2/4] overlayfs: oe-selftest: Make the test more deterministic
Just want to double check, what is the reason the patch was dropped from master-next? Is that only the component order in the caption?
Slava
Quoted 53 lines · Vyacheslav Yurkov via lists.openembedded.org
Quoted 7 lines · Paul Barker
Quoted 5 lines · Vyacheslav Yurkov via lists.openembedded.org
From: Vyacheslav Yurkov <[email protected]>
The test orignally was written under assumption that poky distro is used. When poky-altcft is used for example, then systemd is already set in
DISTRO_FETURES, which the test did not expect.Quoted 6 lines
The commit title says this is about determinism, but it's actually about supporting init systems other than systemd in the test case. I also think "selftest" should come before the component name, so perhaps it should be:
selftest: overlayfs: Support non-systemd init systems
The commit has nothing to do with non-systemd init systems, because the whole feature only works with systemd.
Quoted 28 lines
Quoted 17 lines
Signed-off-by Vyacheslav Yurkov <[email protected]>---
meta/lib/oeqa/selftest/cases/overlayfs.py| 5 +++--1 file changed, 3 insertions(+), 2 deletions(-)meta/lib/oeqa/selftest/cases/overlayfs.py +0
-0@@ -5,7 +5,7 @@ #from oeqa.selftest.case import OESelftestTestCase
meta/lib/oeqa/selftest/cases/overlayfs.py +1
-1-from oeqa.utils.commands import bitbake, runqemu, get_bb_vars +from oeqa.utils.commands import bitbake, runqemu, get_bb_vars,
get_bb_var
from oeqa.core.decorator import OETestTagQuoted 8 lines
from oeqa.core.decorator.data import skipIfNotMachine
@@ -46,7 +46,8 @@ inherit overlayfs
res =bitbake('core-image-minimal', ignore_status=True)
line =getline(res, "overlayfs-user was skipped: missing required distro features")self.assertTrue("overlayfs" in res.output, msg=res.output)
-self.assertTrue("systemd" in res.output, msg=res.output)
+ if not "systemd" inget_bb_var('DISTRO_FEATURES'):
+self.assertTrue("systemd" in res.output, msg=res.output)The inverse logic here confused a few people on the patch review call. Could we add a comment at the same time, something like:
Quoted 5 lines
# If systemd is not enabled, we should see it listed as a missing
# distro featureThat is just re-stating what the code does, but I think it's worthwhile here to avoid confusion.
This is already written a few lines above in the test description.
PATCH v2 3/4 overlayfs: Remove helper unit
In reply to [email protected]From: Vyacheslav Yurkov <[email protected]>
Since systemd v256 workdir and upperdir are automatically created if they don't exist, so we don't need an additional helper unit anymore.
Ref: https://github.com/systemd/systemd/commit/6c75eff6afd90596d09a33a9275be292db4e3380
--- meta/classes-recipe/overlayfs.bbclass | 10 ---------- meta/classes-recipe/rootfs-postcommands.bbclass | 13 +------------ meta/files/overlayfs-create-dirs.service.in | 13 ------------- meta/files/overlayfs-create-dirs.sh | 11 ----------- meta/files/overlayfs-unit.mount.in | 3 +-- meta/lib/oe/overlayfs.py | 4 ---- 6 files changed, 2 insertions(+), 52 deletions(-) delete mode 100644 meta/files/overlayfs-create-dirs.service.in delete mode 100644 meta/files/overlayfs-create-dirs.sh
meta/classes-recipe/overlayfs.bbclass +0 -10
@@ -55,27 +55,21 @@ REQUIRED_DISTRO_FEATURES += "systemd overlayfs" inherit systemd features_check -OVERLAYFS_CREATE_DIRS_TEMPLATE ??= "${COREBASE}/meta/files/overlayfs-create-dirs.service.in" OVERLAYFS_MOUNT_UNIT_TEMPLATE ??= "${COREBASE}/meta/files/overlayfs-unit.mount.in" OVERLAYFS_ALL_OVERLAYS_TEMPLATE ??= "${COREBASE}/meta/files/overlayfs-all-overlays.service.in" python do_create_overlayfs_units() { from oe.overlayfs import mountUnitName - with open(d.getVar("OVERLAYFS_CREATE_DIRS_TEMPLATE"), "r") as f: - CreateDirsUnitTemplate = f.read() with open(d.getVar("OVERLAYFS_MOUNT_UNIT_TEMPLATE"), "r") as f: MountUnitTemplate = f.read() with open(d.getVar("OVERLAYFS_ALL_OVERLAYS_TEMPLATE"), "r") as f: AllOverlaysTemplate = f.read() def prepareUnits(data, lower): - from oe.overlayfs import helperUnitName - args = { 'DATA_MOUNT_POINT': data, 'DATA_MOUNT_UNIT': mountUnitName(data), - 'CREATE_DIRS_SERVICE': helperUnitName(lower), 'LOWERDIR': lower, }
@@ -83,10 +77,6 @@ python do_create_overlayfs_units() { with open(os.path.join(d.getVar('WORKDIR'), mountUnitName(lower)), 'w') as f: f.write(MountUnitTemplate.format(**args)) - bb.debug(1, "Generate helper systemd unit %s" % helperUnitName(lower)) - with open(os.path.join(d.getVar('WORKDIR'), helperUnitName(lower)), 'w') as f: - f.write(CreateDirsUnitTemplate.format(**args)) - def prepareGlobalUnit(dependentUnits): from oe.overlayfs import allOverlaysUnitName args = {
meta/classes-recipe/rootfs-postcommands.bbclass +1 -12
@@ -47,7 +47,7 @@ ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("DISTRO_FEATURES", "systemd" ROOTFS_POSTPROCESS_COMMAND += 'empty_var_volatile' -ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("DISTRO_FEATURES", "overlayfs", "overlayfs_qa_check overlayfs_postprocess", "", d)}' +ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("DISTRO_FEATURES", "overlayfs", "overlayfs_qa_check", "", d)}' inherit image-artifact-names
@@ -570,14 +570,3 @@ python overlayfs_qa_check() { if not allUnitExist: bb.fatal('Not all mount paths and units are installed in the image') } - -python overlayfs_postprocess() { - import shutil - - # install helper script - helperScriptName = "overlayfs-create-dirs.sh" - helperScriptSource = oe.path.join(d.getVar("COREBASE"), "meta/files", helperScriptName) - helperScriptDest = oe.path.join(d.getVar("IMAGE_ROOTFS"), "/usr/sbin/", helperScriptName) - shutil.copyfile(helperScriptSource, helperScriptDest) - os.chmod(helperScriptDest, 0o755) -}
meta/files/overlayfs-create-dirs.service.in deleted +0 -13
@@ -1,13 +0,0 @@ -[Unit] -Description=Overlayfs directories setup {LOWERDIR} -RequiresMountsFor={DATA_MOUNT_POINT} -DefaultDependencies=no - -[Service] -Type=oneshot -ExecStart=/usr/sbin/overlayfs-create-dirs.sh {LOWERDIR} {DATA_MOUNT_POINT} -RemainAfterExit=true -StandardOutput=journal - -[Install] -WantedBy=multi-user.target
meta/files/overlayfs-create-dirs.sh deleted +0 -11
@@ -1,11 +0,0 @@ -#!/bin/sh -# This script is intended to be used sorely by overlayfs-create-dirs.service -# Usage: overlayfs-create-dirs.sh <LOWERDIR> <DATA_MOUNT_POINT> - -lowerdir=$1 -datamountpoint=$2 -mkdir -p ${datamountpoint}/upper${lowerdir} -mkdir -p ${datamountpoint}/workdir${lowerdir} -if [ -d "$lowerdir" ]; then - chown $(stat -c "%U:%G" ${lowerdir}) ${datamountpoint}/upper${lowerdir} -fi
meta/files/overlayfs-unit.mount.in +1 -2
@@ -1,7 +1,6 @@ [Unit] Description=Overlayfs mount unit {LOWERDIR} -Requires={CREATE_DIRS_SERVICE} -After={CREATE_DIRS_SERVICE} +RequiresMountsFor={DATA_MOUNT_POINT} [Mount] What=overlay
meta/lib/oe/overlayfs.py +0 -4
@@ -23,9 +23,6 @@ def allOverlaysUnitName(d): def mountUnitName(unit): return escapeSystemdUnitName(unit) + '.mount' -def helperUnitName(unit): - return escapeSystemdUnitName(unit) + '-create-upper-dir.service' - def unitFileList(d): fileList = [] overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
@@ -47,7 +44,6 @@ def unitFileList(d): continue for path in mountPointList.split(): fileList.append(mountUnitName(path)) - fileList.append(helperUnitName(path)) fileList.append(allOverlaysUnitName(d))
PATCH v2 4/4 oe-selftest: overlayfs: Add a demo case for /etc
In reply to [email protected]From: Vyacheslav Yurkov <[email protected]>
/etc is a special directory. It's possible to create a mount unit for it, but many system services that start early at boot time will not rescan its content when you mount overlay on top. The added test case demonstrates this by adding a sample systemd service and enabling it in overlay.
--- meta/lib/oeqa/selftest/cases/overlayfs.py | 98 +++++++++++++++++++++++ 1 file changed, 98 insertions(+)
meta/lib/oeqa/selftest/cases/overlayfs.py +98 -0
@@ -263,6 +263,104 @@ EOT self._test_correct_image('systemd-machine-units', systemd_machine_unit_append) + @skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently") + def test_etc_mount(self): + """ + Summary: /etc is not supposed to be used with overlayfs.bbclass + Expected: Observe inconsistencies after using etc overlay with a mount unit + Author: Vyacheslav Yurkov <[email protected]> + """ + systemd_machine_unit_append = """ +SYSTEMD_SERVICE:${PN} += " \ + data.mount \ + etc.mount \ +" + +do_install:append() { + install -d ${D}${systemd_system_unitdir} + install -d ${D}${ROOT_HOME} + cat <<EOT > ${D}${systemd_system_unitdir}/etc.mount +[Unit] +Description=OverlayFS mount for /etc directory +DefaultDependencies=no +RequiresMountsFor=/data + +[Mount] +What=overlay +Where=/etc +Type=overlay +Options=lowerdir=/etc,upperdir=/data/overlay/etc,workdir=/data/overlay-workdir/etc +[Install] +WantedBy=local-fs.target +EOT + + cat <<EOT >${D}${systemd_system_unitdir}/data.mount +[Unit] +Description=Persistent storage partition + +[Mount] +What=/dev/sda3 +Where=/data +Type=ext4 +Options=defaults + +[Install] +WantedBy=local-fs.target +EOT + cat <<EOT > ${D}${ROOT_HOME}/test-daemon.service +[Unit] +Description=My one-shot task +After=local-fs.target + +[Service] +Type=oneshot +ExecStart=/usr/bin/echo test +RemainAfterExit=yes + +[Install] +WantedBy=multi-user.target +EOT +} + +FILES:${PN} += "${ROOT_HOME}" +""" + + config = """ +IMAGE_INSTALL:append = " systemd-machine-units" +DISTRO_FEATURES:append = " overlayfs" + +# Use systemd as init manager +INIT_MANAGER = "systemd" + +# enable overlayfs in the kernel +KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc" + +IMAGE_FSTYPES += "wic" +WKS_FILE = "overlayfs_etc.wks.in" +OVERLAYFS_ROOTFS_TYPE = "ext4" +""" + + self.write_config(config) + + machine_inc = """ +OVERLAYFS_MOUNT_POINT[etc] = "/etc" +OVERLAYFS_QA_SKIP[mnt-overlay] = "mount-configured" +""" + self.set_machine_config(machine_inc) + self.write_recipeinc('systemd-machine-units', systemd_machine_unit_append) + bitbake('core-image-minimal') + with runqemu('core-image-minimal', image_fstype='wic', discard_writes=False) as qemu: + test_daemon_path = get_bb_var('ROOT_HOME') + '/test-daemon.service' + status, output = qemu.run_serial('cp -f ' + test_daemon_path + ' /etc/systemd/system/') + status, output = qemu.run_serial("systemctl enable test-daemon") + status, output = qemu.run_serial("sync") + with runqemu('core-image-minimal', image_fstype='wic') as qemu: + # Check the test service status. It's enabled and supposed to start, but it didn't + status, output = qemu.run_serial("systemctl is-enabled test-daemon") + self.assertTrue("enabled" in output, msg=output) + status, output = qemu.run_serial("systemctl is-active test-daemon") + self.assertTrue("inactive" in output, msg=output) + @OETestTag("runqemu") class OverlayFSEtcRunTimeTests(OESelftestTestCase): """overlayfs-etc class tests"""
Re: [OE-core] [PATCH v2 4/4] oe-selftest: overlayfs: Add a demo case for /etc
Quoted 109 lines · Vyacheslav Yurkov via lists.openembedded.org
From: Vyacheslav Yurkov <[email protected]>
/etc is a special directory. It's possible to create a mount unit for it, but many system services that start early at boot time will not rescan its content when you mount overlay on top. The added test case demonstrates this by adding a sample systemd service and enabling it in overlay.
Signed-off-by Vyacheslav Yurkov <[email protected]>--- meta/lib/oeqa/selftest/cases/overlayfs.py | 98 +++++++++++++++++++++++ 1 file changed, 98 insertions(+)meta/lib/oeqa/selftest/cases/overlayfs.py +88
-0@@ -263,6 +263,104 @@ EOT self._test_correct_image('systemd-machine-units', systemd_machine_unit_append) + @skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently") + def test_etc_mount(self): + """ + Summary: /etc is not supposed to be used with overlayfs.bbclass + Expected: Observe inconsistencies after using etc overlay with a mount unit + Author: Vyacheslav Yurkov <[email protected]> + """ + systemd_machine_unit_append = """ +SYSTEMD_SERVICE:${PN} += " \ + data.mount \ + etc.mount \ +" + +do_install:append() { + install -d ${D}${systemd_system_unitdir} + install -d ${D}${ROOT_HOME} + cat <<EOT > ${D}${systemd_system_unitdir}/etc.mount +[Unit] +Description=OverlayFS mount for /etc directory +DefaultDependencies=no +RequiresMountsFor=/data + +[Mount] +What=overlay +Where=/etc +Type=overlay +Options=lowerdir=/etc,upperdir=/data/overlay/etc,workdir=/data/overlay-workdir/etc +[Install] +WantedBy=local-fs.target +EOT + + cat <<EOT >${D}${systemd_system_unitdir}/data.mount +[Unit] +Description=Persistent storage partition + +[Mount] +What=/dev/sda3 +Where=/data +Type=ext4 +Options=defaults + +[Install] +WantedBy=local-fs.target +EOT + cat <<EOT > ${D}${ROOT_HOME}/test-daemon.service +[Unit] +Description=My one-shot task +After=local-fs.target + +[Service] +Type=oneshot +ExecStart=/usr/bin/echo test +RemainAfterExit=yes + +[Install] +WantedBy=multi-user.target +EOT +} + +FILES:${PN} += "${ROOT_HOME}" +""" + + config = """ +IMAGE_INSTALL:append = " systemd-machine-units" +DISTRO_FEATURES:append = " overlayfs" + +# Use systemd as init manager +INIT_MANAGER = "systemd" + +# enable overlayfs in the kernel +KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc" + +IMAGE_FSTYPES += "wic" +WKS_FILE = "overlayfs_etc.wks.in" +OVERLAYFS_ROOTFS_TYPE = "ext4" +""" + + self.write_config(config) + + machine_inc = """ +OVERLAYFS_MOUNT_POINT[etc] = "/etc" +OVERLAYFS_QA_SKIP[mnt-overlay] = "mount-configured" +""" + self.set_machine_config(machine_inc) + self.write_recipeinc('systemd-machine-units', systemd_machine_unit_append) + bitbake('core-image-minimal') + with runqemu('core-image-minimal', image_fstype='wic', discard_writes=False) as qemu: + test_daemon_path = get_bb_var('ROOT_HOME') + '/test-daemon.service'
meta/lib/oeqa/selftest/cases/overlayfs.py EOT · line 363
Quoted 10 lines
meta/lib/oeqa/selftest/cases/overlayfs.py +10
-0+ status, output = qemu.run_serial('cp -f ' + test_daemon_path + ' /etc/systemd/system/') + status, output = qemu.run_serial("systemctl enable test-daemon") + status, output = qemu.run_serial("sync") + with runqemu('core-image-minimal', image_fstype='wic') as qemu: + # Check the test service status. It's enabled and supposed to start, but it didn't + status, output = qemu.run_serial("systemctl is-enabled test-daemon") + self.assertTrue("enabled" in output, msg=output) + status, output = qemu.run_serial("systemctl is-active test-daemon") + self.assertTrue("inactive" in output, msg=output) +
What is actually being tested here? It looks like we're just confirming that incorrect behaviour occurs. Why do we want to ensure that the behaviour stays incorrect?
If someone is installing an 'etc.mount' unit into the rootfs that behaves like this then there's probably nothing we can do within Yocto Project to help them. I suppose you could use overlayfs for /etc if you set it up from an initramfs before pivoting to the main rootfs.
Best regards,
Signature · 1 line
Paul Barker
Re: [OE-core] [PATCH v2 4/4] oe-selftest: overlayfs: Add a demo case for /etc
Quoted 125 lines · Paul Barker
Quoted 109 lines · Vyacheslav Yurkov via lists.openembedded.org
From: Vyacheslav Yurkov <[email protected]>
/etc is a special directory. It's possible to create a mount unit for it, but many system services that start early at boot time will not rescan its content when you mount overlay on top. The added test case demonstrates this by adding a sample systemd service and enabling it in overlay.
Signed-off-by Vyacheslav Yurkov <[email protected]>--- meta/lib/oeqa/selftest/cases/overlayfs.py | 98 +++++++++++++++++++++++ 1 file changed, 98 insertions(+)meta/lib/oeqa/selftest/cases/overlayfs.py +88
-0@@ -263,6 +263,104 @@ EOT self._test_correct_image('systemd-machine-units', systemd_machine_unit_append) + @skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently") + def test_etc_mount(self): + """ + Summary: /etc is not supposed to be used with overlayfs.bbclass + Expected: Observe inconsistencies after using etc overlay with a mount unit + Author: Vyacheslav Yurkov <[email protected]> + """ + systemd_machine_unit_append = """ +SYSTEMD_SERVICE:${PN} += " \ + data.mount \ + etc.mount \ +" + +do_install:append() { + install -d ${D}${systemd_system_unitdir} + install -d ${D}${ROOT_HOME} + cat <<EOT > ${D}${systemd_system_unitdir}/etc.mount +[Unit] +Description=OverlayFS mount for /etc directory +DefaultDependencies=no +RequiresMountsFor=/data + +[Mount] +What=overlay +Where=/etc +Type=overlay +Options=lowerdir=/etc,upperdir=/data/overlay/etc,workdir=/data/overlay-workdir/etc +[Install] +WantedBy=local-fs.target +EOT + + cat <<EOT >${D}${systemd_system_unitdir}/data.mount +[Unit] +Description=Persistent storage partition + +[Mount] +What=/dev/sda3 +Where=/data +Type=ext4 +Options=defaults + +[Install] +WantedBy=local-fs.target +EOT + cat <<EOT > ${D}${ROOT_HOME}/test-daemon.service +[Unit] +Description=My one-shot task +After=local-fs.target + +[Service] +Type=oneshot +ExecStart=/usr/bin/echo test +RemainAfterExit=yes + +[Install] +WantedBy=multi-user.target +EOT +} + +FILES:${PN} += "${ROOT_HOME}" +""" + + config = """ +IMAGE_INSTALL:append = " systemd-machine-units" +DISTRO_FEATURES:append = " overlayfs" + +# Use systemd as init manager +INIT_MANAGER = "systemd" + +# enable overlayfs in the kernel +KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc" + +IMAGE_FSTYPES += "wic" +WKS_FILE = "overlayfs_etc.wks.in" +OVERLAYFS_ROOTFS_TYPE = "ext4" +""" + + self.write_config(config) + + machine_inc = """ +OVERLAYFS_MOUNT_POINT[etc] = "/etc" +OVERLAYFS_QA_SKIP[mnt-overlay] = "mount-configured" +""" + self.set_machine_config(machine_inc) + self.write_recipeinc('systemd-machine-units', systemd_machine_unit_append) + bitbake('core-image-minimal') + with runqemu('core-image-minimal', image_fstype='wic', discard_writes=False) as qemu: + test_daemon_path = get_bb_var('ROOT_HOME') + '/test-daemon.service'meta/lib/oeqa/selftest/cases/overlayfs.py EOT · line 363
Quoted 10 lines
meta/lib/oeqa/selftest/cases/overlayfs.py +10
-0+ status, output = qemu.run_serial('cp -f ' + test_daemon_path + ' /etc/systemd/system/') + status, output = qemu.run_serial("systemctl enable test-daemon") + status, output = qemu.run_serial("sync") + with runqemu('core-image-minimal', image_fstype='wic') as qemu: + # Check the test service status. It's enabled and supposed to start, but it didn't + status, output = qemu.run_serial("systemctl is-enabled test-daemon") + self.assertTrue("enabled" in output, msg=output) + status, output = qemu.run_serial("systemctl is-active test-daemon") + self.assertTrue("inactive" in output, msg=output) +What is actually being tested here? It looks like we're just confirming that incorrect behaviour occurs. Why do we want to ensure that the behaviour stays incorrect?
Quoted 7 lines
If someone is installing an 'etc.mount' unit into the rootfs that behaves like this then there's probably nothing we can do within Yocto Project to help them. I suppose you could use overlayfs for /etc if you set it up from an initramfs before pivoting to the main rootfs.
Best regards,
Correct, and this is what overlayfs-etc class is for. I've seen attempts to use the mount unit instead and this is just a proof of concept that it doesn't work.
Slava
Quoted 15 lines · Vyacheslav Yurkov via lists.openembedded.org
From: Vyacheslav Yurkov <[email protected]>
The QA check for skipped mounts happens twice, at parsing stage and at rootfs postprocessing. Fix the former, because it didn't happen properly.
Signed-off-by Vyacheslav Yurkov <[email protected]>--- meta/lib/oe/overlayfs.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)meta/lib/oe/overlayfs.py +0
-0
meta/lib/oe/overlayfs.py def unitFileList(d): · line 43
Quoted 9 lines
meta/lib/oe/overlayfs.py +2
-1@@ -36,7 +36,8 @@ def unitFileList(d): # check that we have required mount points set first requiredMountPoints = d.getVarFlags('OVERLAYFS_WRITABLE_PATHS') for mountPoint in requiredMountPoints: - if mountPoint not in overlayMountPoints: + qaSkip = (d.getVarFlag("OVERLAYFS_QA_SKIP", mountPoint) or "").split() + if mountPoint not in overlayMountPoints and not "mount-configured" in qaSkip: bb.fatal("Missing required mount point for OVERLAYFS_MOUNT_POINT[%s] in your MACHINE configuration" % mountPoint) for mountPoint in overlayMountPoints:
Thanks for the patches! The review has been delayed a bit by the holidays...
We need more explanation in the commit message of why the QA check was happening twice and why it wasn't happening properly at parsing time.
Also, when you send a v2 (or v3, etc) patch series, please include a note on what has changed since the previous version, either after the commit message delimiter (the first '---' line) or in a cover letter for the series. That makes review much easier!
Best regards,
Signature · 1 line
Paul Barker
PATCH v3 1/1 overlayfs: Fix the QA skip for ignored mounts
From: Vyacheslav Yurkov <[email protected]>
The supressing of QA check for mounts should happens twice, at parsing stage and at rootfs postprocessing. If the mount point is configured to be skipped, but it is still present in the configuration (machine or distro), then the parsing would complain.
--- meta/lib/oe/overlayfs.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/lib/oe/overlayfs.py +2 -1
@@ -36,7 +36,8 @@ def unitFileList(d): # check that we have required mount points set first requiredMountPoints = d.getVarFlags('OVERLAYFS_WRITABLE_PATHS') for mountPoint in requiredMountPoints: - if mountPoint not in overlayMountPoints: + qaSkip = (d.getVarFlag("OVERLAYFS_QA_SKIP", mountPoint) or "").split() + if mountPoint not in overlayMountPoints and not "mount-configured" in qaSkip: bb.fatal("Missing required mount point for OVERLAYFS_MOUNT_POINT[%s] in your MACHINE configuration" % mountPoint) for mountPoint in overlayMountPoints:
Re: [OE-core] [PATCH v3] overlayfs: Fix the QA skip for ignored mounts
Hi,
You have sent multiple versions of some patches (v3 for 1/4, v2 for the rest, I think): it's not straightforward for maintainers to follow which patch version to apply.
I also see Paul has made some comment about the naming order of the patch titles.
Can you take Paul's comments into account, and can you please re-send the series with the proper versions of the patches? All patches should have the same version.
Thanks!
Antonin
Signature · 3 lines
Antonin Godard, Bootlin Embedded Linux and Kernel engineering https://bootlin.com