overlayfs-qa: common functions for overlayfs
PATCH v3 1/8 overlayfs-qa: common functions for overlayfs
This class provides commom functions for overlayfs and its QA check, which is performed in ROOTFS_POSTPROCESS_COMMAND
--- meta/classes/overlayfs-qa.bbclass | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 meta/classes/overlayfs-qa.bbclass
meta/classes/overlayfs-qa.bbclass new file +14 -0
@@ -0,0 +1,14 @@ +# This class contains common functions for overlayfs and its QA check, +# which is performed in ROOTFS_POSTPROCESS_COMMAND + +# this function is based on https://github.com/systemd/systemd/blob/main/src/basic/unit-name.c +def escapeSystemdUnitName(path): + escapeMap = { + '/': '-', + '-': "\\x2d", + '\\': "\\x5d" + } + return "".join([escapeMap.get(c, c) for c in path.strip('/')]) + +def mountUnitName(unit): + return escapeSystemdUnitName(unit) + '.mount'
Signature · 1 line
2.28.0
PATCH v3 2/8 overlayfs.bbclass: generate overlayfs mount units
It's often desired in Embedded System design to have a read-only rootfs. But a lot of different applications might want to have a read-write access to some parts of a filesystem. It can be especially useful when your update mechanism overwrites the whole rootfs, but you want your application data to be preserved between updates. This class provides a way to achieve that by means of overlayfs and at the same time keeping the base rootfs read-only.
--- meta/classes/overlayfs.bbclass | 133 +++++++++++++++++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 meta/classes/overlayfs.bbclass
meta/classes/overlayfs.bbclass new file +133 -0
@@ -0,0 +1,133 @@ +# Class for generation of overlayfs mount units +# +# It's often desired in Embedded System design to have a read-only rootfs. +# But a lot of different applications might want to have a read-write access to +# some parts of a filesystem. It can be especially useful when your update mechanism +# overwrites the whole rootfs, but you want your application data to be preserved +# between updates. This class provides a way to achieve that by means +# of overlayfs and at the same time keeping the base rootfs read-only. +# +# Usage example. +# +# Set a mount point for a partition overlayfs is going to use as upper layer +# in your machine configuration. Underlying file system can be anything that +# is supported by overlayfs. This has to be done in your machine configuration. +# QA check fails to catch file existence if you redefine this variable in your recipe! +# +# OVERLAYFS_MOUNT_POINT[data] ?= "/data" +# +# The class assumes you have a data.mount systemd unit defined in your +# systemd-machine-units recipe and installed to the image. +# +# Then you can specify writable directories on a recipe base +# +# OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application" +# +# To support several mount points you can use a different variable flag. Assume we +# want to have a writable location on the file system, but not interested where the data +# survive a reboot. Then we could have a mnt-overlay.mount unit for a tmpfs file system: +# +# OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay" +# OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application" +# +# Note: the class does not support /etc directory itself, because systemd depends on it + +REQUIRED_DISTRO_FEATURES += "systemd overlayfs" + +inherit systemd features_check overlayfs-qa + +def strForBash(s): + return s.replace('\\', '\\\\') + +def unitFileList(d): + fileList = [] + overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT") + + if not overlayMountPoints: + bb.fatal("A recipe uses overlayfs class but there is no OVERLAYFS_MOUNT_POINT set in your MACHINE configuration") + + # check that we have required mount points set first + requiredMountPoints = d.getVarFlags('OVERLAYFS_WRITABLE_PATHS') + for mountPoint in requiredMountPoints: + if mountPoint not in overlayMountPoints: + bb.fatal("Missing required mount point for OVERLAYFS_MOUNT_POINT[%s] in your MACHINE configuration" % mountPoint) + + for mountPoint in overlayMountPoints: + for path in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split(): + fileList.append(mountUnitName(path)) + fileList.append(helperUnitName(path)) + + return fileList + +def helperUnitName(unit): + return escapeSystemdUnitName(unit) + '-create-upper-dir.service' + +python do_create_overlayfs_units() { + CreateDirsUnitTemplate = """[Unit] +Description=Overlayfs directories setup +Requires={DATA_MOUNT_UNIT} +After={DATA_MOUNT_UNIT} +DefaultDependencies=no + +[Service] +Type=oneshot +ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR} && mkdir -p {DATA_MOUNT_POINT}/upper{LOWERDIR} +RemainAfterExit=true +StandardOutput=journal + +[Install] +WantedBy=multi-user.target +""" + MountUnitTemplate = """[Unit] +Description=Overlayfs mount unit +Requires={CREATE_DIRS_SERVICE} +After={CREATE_DIRS_SERVICE} + +[Mount] +What=overlay +Where={LOWERDIR} +Type=overlay +Options=lowerdir={LOWERDIR},upperdir={DATA_MOUNT_POINT}/upper{LOWERDIR},workdir={DATA_MOUNT_POINT}/workdir{LOWERDIR} + +[Install] +WantedBy=multi-user.target +""" + + def prepareUnits(data, lower): + args = { + 'DATA_MOUNT_POINT': data, + 'DATA_MOUNT_UNIT': mountUnitName(data), + 'CREATE_DIRS_SERVICE': helperUnitName(lower), + 'LOWERDIR': lower, + } + + with open(os.path.join(d.getVar('WORKDIR'), mountUnitName(lower)), 'w') as f: + f.write(MountUnitTemplate.format(**args)) + + with open(os.path.join(d.getVar('WORKDIR'), helperUnitName(lower)), 'w') as f: + f.write(CreateDirsUnitTemplate.format(**args)) + + overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT") + for mountPoint in overlayMountPoints: + for lower in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split(): + prepareUnits(d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint), lower) +} + +# we need to generate file names early during parsing stage +python () { + unitList = unitFileList(d) + for unit in unitList: + d.appendVar('SYSTEMD_SERVICE_' + d.getVar('PN'), ' ' + unit); + d.appendVar('FILES_' + d.getVar('PN'), ' ' + strForBash(unit)) + + d.setVar('OVERLAYFS_UNIT_LIST', ' '.join([strForBash(s) for s in unitList])) +} + +do_install_append() { + install -d ${D}${systemd_system_unitdir} + for unit in ${OVERLAYFS_UNIT_LIST}; do + install -m 0444 ${WORKDIR}/${unit} ${D}${systemd_system_unitdir} + done +} + +addtask create_overlayfs_units before do_install
Signature · 1 line
2.28.0
Re: [OE-core] [PATCH v3 2/8] overlayfs.bbclass: generate overlayfs mount units
Quoted 72 lines · Vyacheslav Yurkov
It's often desired in Embedded System design to have a read-only rootfs. But a lot of different applications might want to have a read-write access to some parts of a filesystem. It can be especially useful when your update mechanism overwrites the whole rootfs, but you want your application data to be preserved between updates. This class provides a way to achieve that by means of overlayfs and at the same time keeping the base rootfs read-only.
Signed-off-by Vyacheslav Yurkov <[email protected]>---
meta/classes/overlayfs.bbclass | 133 +++++++++++++++++++++++++++++++++1 file changed, 133 insertions(+)create mode 100644 meta/classes/overlayfs.bbclass
meta/classes/overlayfs.bbclass new file +53
-0@@ -0,0 +1,133 @@ +# Class for generation of overlayfs mount units +# +# It's often desired in Embedded System design to have a read-only rootfs. +# But a lot of different applications might want to have a read-write access to +# some parts of a filesystem. It can be especially useful when your update mechanism +# overwrites the whole rootfs, but you want your application data to be preserved +# between updates. This class provides a way to achieve that by means +# of overlayfs and at the same time keeping the base rootfs read-only. +# +# Usage example. +# +# Set a mount point for a partition overlayfs is going to use as upper layer +# in your machine configuration. Underlying file system can be anything that +# is supported by overlayfs. This has to be done in your machine configuration. +# QA check fails to catch file existence if you redefine this variable in your recipe! +# +# OVERLAYFS_MOUNT_POINT[data] ?= "/data" +# +# The class assumes you have a data.mount systemd unit defined in your +# systemd-machine-units recipe and installed to the image. +# +# Then you can specify writable directories on a recipe base +# +# OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application" +# +# To support several mount points you can use a different variable flag. Assume we +# want to have a writable location on the file system, but not interested where the data +# survive a reboot. Then we could have a mnt-overlay.mount unit for a tmpfs file system: +# +# OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay" +# OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application" +# +# Note: the class does not support /etc directory itself, because systemd depends on it + +REQUIRED_DISTRO_FEATURES += "systemd overlayfs" + +inherit systemd features_check overlayfs-qa + +def strForBash(s): + return s.replace('\\', '\\\\') + +def unitFileList(d): + fileList = [] + overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT") + + if not overlayMountPoints: + bb.fatal("A recipe uses overlayfs class but there is no OVERLAYFS_MOUNT_POINT set in your MACHINE configuration") + + # check that we have required mount points set first + requiredMountPoints = d.getVarFlags('OVERLAYFS_WRITABLE_PATHS') + for mountPoint in requiredMountPoints: + if mountPoint not in overlayMountPoints: + bb.fatal("Missing required mount point for OVERLAYFS_MOUNT_POINT[%s] in your MACHINE configuration" % mountPoint)
meta/classes/overlayfs.bbclass line 63
Quoted 10 lines
meta/classes/overlayfs.bbclass new file +10
-0+ + for mountPoint in overlayMountPoints: + for path in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split(): + fileList.append(mountUnitName(path)) + fileList.append(helperUnitName(path)) + + return fileList + +def helperUnitName(unit): + return escapeSystemdUnitName(unit) + '-create-upper-dir.service'
Again, I'm a little torn on some of the functions. Would some of these be better as part of a function library in meta/lib/oe/ ?
The downside is that variable dependencies are no longer automatic so the dependency on OVERLAYFS_WRITABLE_PATHS may be missed, I don't think it would see OVERLAYFS_MOUNT_POINT as I'm not sure the magic parsing code understand getVarFlags (compared to getVarFlag which it does).
Cheers,
Richard
Re: [OE-core] [PATCH v3 2/8] overlayfs.bbclass: generate overlayfs mount units
Quoted 86 lines · Richard Purdie
Quoted 72 lines · Vyacheslav Yurkov
It's often desired in Embedded System design to have a read-only rootfs. But a lot of different applications might want to have a read-write access to some parts of a filesystem. It can be especially useful when your update mechanism overwrites the whole rootfs, but you want your application data to be preserved between updates. This class provides a way to achieve that by means of overlayfs and at the same time keeping the base rootfs read-only.
Signed-off-by Vyacheslav Yurkov <[email protected]>--- meta/classes/overlayfs.bbclass | 133 +++++++++++++++++++++++++++++++++ 1 file changed, 133 insertions(+)create mode 100644 meta/classes/overlayfs.bbclass
meta/classes/overlayfs.bbclass new file +53
-0@@ -0,0 +1,133 @@ +# Class for generation of overlayfs mount units +# +# It's often desired in Embedded System design to have a read-only rootfs. +# But a lot of different applications might want to have a read-write access to +# some parts of a filesystem. It can be especially useful when your update mechanism +# overwrites the whole rootfs, but you want your application data to be preserved +# between updates. This class provides a way to achieve that by means +# of overlayfs and at the same time keeping the base rootfs read-only. +# +# Usage example. +# +# Set a mount point for a partition overlayfs is going to use as upper layer +# in your machine configuration. Underlying file system can be anything that +# is supported by overlayfs. This has to be done in your machine configuration. +# QA check fails to catch file existence if you redefine this variable in your recipe! +# +# OVERLAYFS_MOUNT_POINT[data] ?= "/data" +# +# The class assumes you have a data.mount systemd unit defined in your +# systemd-machine-units recipe and installed to the image. +# +# Then you can specify writable directories on a recipe base +# +# OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application" +# +# To support several mount points you can use a different variable flag. Assume we +# want to have a writable location on the file system, but not interested where the data +# survive a reboot. Then we could have a mnt-overlay.mount unit for a tmpfs file system: +# +# OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay" +# OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application" +# +# Note: the class does not support /etc directory itself, because systemd depends on it + +REQUIRED_DISTRO_FEATURES += "systemd overlayfs" + +inherit systemd features_check overlayfs-qa + +def strForBash(s): + return s.replace('\\', '\\\\') + +def unitFileList(d): + fileList = [] + overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT") + + if not overlayMountPoints: + bb.fatal("A recipe uses overlayfs class but there is no OVERLAYFS_MOUNT_POINT set in your MACHINE configuration") + + # check that we have required mount points set first + requiredMountPoints = d.getVarFlags('OVERLAYFS_WRITABLE_PATHS') + for mountPoint in requiredMountPoints: + if mountPoint not in overlayMountPoints: + bb.fatal("Missing required mount point for OVERLAYFS_MOUNT_POINT[%s] in your MACHINE configuration" % mountPoint)meta/classes/overlayfs.bbclass line 63
Quoted 10 lines
meta/classes/overlayfs.bbclass new file +10
-0+ + for mountPoint in overlayMountPoints: + for path in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split(): + fileList.append(mountUnitName(path)) + fileList.append(helperUnitName(path)) + + return fileList + +def helperUnitName(unit): + return escapeSystemdUnitName(unit) + '-create-upper-dir.service'Again, I'm a little torn on some of the functions. Would some of these be better as part of a function library in meta/lib/oe/ ?
Quoted 8 lines
The downside is that variable dependencies are no longer automatic so the dependency on
OVERLAYFS_WRITABLE_PATHSmay be missed, I don't think it would seeOVERLAYFS_MOUNT_POINTas I'm not sure the magic parsing code understand getVarFlags (compared to getVarFlag which it does).Cheers,
Richard
Sorry for the late response. Do you mean to move only helper functions to meta/lib/oe, right? By variable dependencies you mean that bitbake would not pick up meta data changes automatically?
I will try to move common parts to meta/lib/oe then and send a v4
Thanks,
Vyacheslav
Re: [OE-core] [PATCH v3 2/8] overlayfs.bbclass: generate overlayfs mount units
Quoted 2 lines · Vyacheslav Yurkov
Sorry for the late response. Do you mean to move only helper functions to meta/lib/oe, right?
Correct, yes. The things being defined as "def <function>".
Quoted 2 lines
By variable dependencies you mean that bitbake would not pick up meta data changes automatically?
I mean that if a function is something like:
def function(d):
d.getVar("VAR")
it will no longer magically depend on the variable VAR.
Quoted 1 line
I will try to move common parts to meta/lib/oe then and send a v4
Thanks!
Cheers,
Richard
PATCH v3 3/8 maintainers.inc: overlayfs bbclass maintainer
In reply to Vyacheslav Yurkov--- meta/conf/distro/include/maintainers.inc | 2 ++ 1 file changed, 2 insertions(+)
meta/conf/distro/include/maintainers.inc +2 -0
@@ -808,3 +808,5 @@ RECIPE_MAINTAINER_pn-xz = "Denys Dmytriyenko <[email protected]>" RECIPE_MAINTAINER_pn-zip = "Denys Dmytriyenko <[email protected]>" RECIPE_MAINTAINER_pn-zlib = "Denys Dmytriyenko <[email protected]>" RECIPE_MAINTAINER_pn-zstd = "Alexander Kanavin <[email protected]>" +BBCLASS_MAINTAINER_overlayfs = "Vyacheslav Yurkov <[email protected]>" +BBCLASS_MAINTAINER_overlayfs-qa = "Vyacheslav Yurkov <[email protected]>"
Signature · 1 line
2.28.0
Re: [OE-core] [PATCH v3 3/8] maintainers.inc: overlayfs bbclass maintainer
Quoted 13 lines · Vyacheslav Yurkov
Signed-off-by Vyacheslav Yurkov <[email protected]>---
meta/conf/distro/include/maintainers.inc | 2 ++1 file changed, 2 insertions(+)meta/conf/distro/include/maintainers.inc +0
-0@@ -808,3 +808,5 @@ RECIPE_MAINTAINER_pn-xz = "Denys Dmytriyenko <[email protected]>"RECIPE_MAINTAINER_pn-zip = "Denys Dmytriyenko <[email protected]>" RECIPE_MAINTAINER_pn-zlib = "Denys Dmytriyenko <[email protected]>" RECIPE_MAINTAINER_pn-zstd = "Alexander Kanavin <[email protected]>"
meta/conf/distro/include/maintainers.inc RECIPE_MAINTAINER_pn-xz = "Denys Dmytriyenko <[email protected]>" · line 812
Quoted 2 lines
meta/conf/distro/include/maintainers.inc +2
-0+BBCLASS_MAINTAINER_overlayfs = "Vyacheslav Yurkov <[email protected]>" +BBCLASS_MAINTAINER_overlayfs-qa = "Vyacheslav Yurkov <[email protected]>"
This variable doesn't exist.
I've been trying to figure out how to better establish maintainers and there is a MAINTAINERS.md patch queued to create a maintainers file. I think this entry could be better off in there? Could you create a follow on patch for that file instead please?
Cheers,
Richard
PATCH v3 4/8 rootfs-postcommands: add QA check for overlayfs
In reply to Vyacheslav YurkovThe check is conditional and only enabled when overlayfs is set in DISTRO_FEATURES
--- meta/classes/rootfs-postcommands.bbclass | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
meta/classes/rootfs-postcommands.bbclass +24 -0
@@ -39,7 +39,10 @@ ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("DISTRO_FEATURES", "systemd" ROOTFS_POSTPROCESS_COMMAND += 'empty_var_volatile;' +ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("DISTRO_FEATURES", "overlayfs", "overlayfs_qa_check;", "", d)}' + inherit image-artifact-names +inherit ${@bb.utils.contains("DISTRO_FEATURES", "overlayfs", "overlayfs-qa", "", d)} # Sort the user and group entries in /etc by ID in order to make the content # deterministic. Package installs are not deterministic, causing the ordering
@@ -373,3 +376,24 @@ rootfs_reproducible () { fi fi } + +python overlayfs_qa_check() { + # this is a dumb check for unit existence, not its validity + overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT") + imagepath = d.getVar("IMAGE_ROOTFS") + searchpaths = [oe.path.join(imagepath, d.getVar("sysconfdir"), "systemd", "system"), + oe.path.join(imagepath, d.getVar("systemd_system_unitdir"))] + + allUnitExist = True; + for mountPoint in overlayMountPoints: + path = d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint) + unit = mountUnitName(path) + + if not any(os.path.isfile(oe.path.join(dirpath, unit)) + for dirpath in searchpaths): + bb.warn('Unit name %s not found in systemd unit directories' % unit) + allUnitExist = False; + + if not allUnitExist: + bb.fatal('Not all mount units are installed by the BSP') +}
Signature · 1 line
2.28.0
PATCH v3 5/8 systemd-machine-units: add bbappend for meta-selftest
In reply to Vyacheslav Yurkov--- .../systemd-machine-units/systemd-machine-units_%.bbappend | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 meta-selftest/recipes-test/systemd-machine-units/systemd-machine-units_%.bbappend
meta-selftest/recipes-test/systemd-machine-units/systemd-machine-units_%.bbappend new file +2 -0
@@ -0,0 +1,2 @@ +# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests. +include test_recipe.inc
Signature · 1 line
2.28.0
PATCH v3 6/8 overlayfs: meta-selftest recipe
In reply to Vyacheslav YurkovThe recipe demonstrates example usage of overlayfs bbclass
--- .../overlayfs-user/overlayfs-user.bb | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 meta-selftest/recipes-test/overlayfs-user/overlayfs-user.bb
meta-selftest/recipes-test/overlayfs-user/overlayfs-user.bb new file +17 -0
@@ -0,0 +1,17 @@ +SUMMARY = "Overlayfs class unit test" +DESCRIPTION = "Contains an overlayfs configuration" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" +EXCLUDE_FROM_WORLD = "1" + +inherit ${@bb.utils.contains("DISTRO_FEATURES", "overlayfs", "overlayfs", "", d)} +include test_recipe.inc + +OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/my-application" + +do_install() { + install -d ${D}/usr/share/my-application +} + +FILES_${PN} += "/usr"
Signature · 1 line
2.28.0
PATCH v3 7/8 oeqa/selftest: overlayfs unit tests
In reply to Vyacheslav YurkovUnit tests for overlayfs.bbclass
--- meta/lib/oeqa/selftest/cases/overlayfs.py | 171 ++++++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 meta/lib/oeqa/selftest/cases/overlayfs.py
meta/lib/oeqa/selftest/cases/overlayfs.py new file +171 -0
@@ -0,0 +1,171 @@ +# +# SPDX-License-Identifier: MIT +# + +from oeqa.selftest.case import OESelftestTestCase +from oeqa.utils.commands import runCmd, bitbake, get_bb_var, runqemu + +class OverlayFSTests(OESelftestTestCase): + """Overlayfs class usage tests""" + + def getline(self, res, line): + for l in res.output.split('\n'): + if line in l: + return l + + def add_overlay_conf_to_machine(self): + machine_inc = """ +OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay" +""" + self.set_machine_config(machine_inc) + + def test_distro_features_missing(self): + """ + Summary: Check that required DISTRO_FEATURES are set + Expected: Fail when either systemd or overlayfs are not in DISTRO_FEATURES + Author: Vyacheslav Yurkov <[email protected]> + """ + + config = """ +IMAGE_INSTALL_append = " overlayfs-user" +""" + overlayfs_recipe_append = """ +inherit overlayfs +""" + self.write_config(config) + self.add_overlay_conf_to_machine() + self.write_recipeinc('overlayfs-user', overlayfs_recipe_append) + + res = bitbake('core-image-minimal', ignore_status=True) + line = self.getline(res, "overlayfs-user was skipped: missing required distro features") + self.assertTrue("overlayfs" in res.output, msg=res.output) + self.assertTrue("systemd" in res.output, msg=res.output) + self.assertTrue("ERROR: Required build target 'core-image-minimal' has no buildable providers." in res.output, msg=res.output) + + def test_not_all_units_installed(self): + """ + Summary: Test QA check that we have required mount units in the image + Expected: Fail because mount unit for overlay partition is not installed + Author: Vyacheslav Yurkov <[email protected]> + """ + + config = """ +IMAGE_INSTALL_append = " overlayfs-user" +DISTRO_FEATURES += "systemd overlayfs" +""" + + self.write_config(config) + self.add_overlay_conf_to_machine() + + res = bitbake('core-image-minimal', ignore_status=True) + line = self.getline(res, "Unit name mnt-overlay.mount not found in systemd unit directories") + self.assertTrue(line and line.startswith("WARNING:"), msg=res.output) + line = self.getline(res, "Not all mount units are installed by the BSP") + self.assertTrue(line and line.startswith("ERROR:"), msg=res.output) + + def test_mount_unit_not_set(self): + """ + Summary: Test whether mount unit was set properly + Expected: Fail because mount unit was not set + Author: Vyacheslav Yurkov <[email protected]> + """ + + config = """ +IMAGE_INSTALL_append = " overlayfs-user" +DISTRO_FEATURES += "systemd overlayfs" +""" + + self.write_config(config) + + res = bitbake('core-image-minimal', ignore_status=True) + line = self.getline(res, "A recipe uses overlayfs class but there is no OVERLAYFS_MOUNT_POINT set in your MACHINE configuration") + self.assertTrue(line and line.startswith("Parsing recipes...ERROR:"), msg=res.output) + + def test_wrong_mount_unit_set(self): + """ + Summary: Test whether mount unit was set properly + Expected: Fail because not the correct flag used for mount unit + Author: Vyacheslav Yurkov <[email protected]> + """ + + config = """ +IMAGE_INSTALL_append = " overlayfs-user" +DISTRO_FEATURES += "systemd overlayfs" +""" + + wrong_machine_config = """ +OVERLAYFS_MOUNT_POINT[usr-share-overlay] = "/usr/share/overlay" +""" + + self.write_config(config) + self.set_machine_config(wrong_machine_config) + + res = bitbake('core-image-minimal', ignore_status=True) + line = self.getline(res, "Missing required mount point for OVERLAYFS_MOUNT_POINT[mnt-overlay] in your MACHINE configuration") + self.assertTrue(line and line.startswith("Parsing recipes...ERROR:"), msg=res.output) + + def test_correct_image(self): + """ + Summary: Check that we can create an image when all parameters are + set correctly + Expected: Image is created successfully + Author: Vyacheslav Yurkov <[email protected]> + """ + + config = """ +IMAGE_INSTALL_append = " overlayfs-user systemd-machine-units" +DISTRO_FEATURES += "systemd overlayfs" + +# Use systemd as init manager +VIRTUAL-RUNTIME_init_manager = "systemd" + +# enable overlayfs in the kernel +KERNEL_EXTRA_FEATURES_append = " features/overlayfs/overlayfs.scc" +""" + + systemd_machine_unit_append = """ +SYSTEMD_SERVICE_${PN} += " \ + mnt-overlay.mount \ +" + +do_install_append() { + install -d ${D}${systemd_system_unitdir} + cat <<EOT > ${D}${systemd_system_unitdir}/mnt-overlay.mount +[Unit] +Description=Tmpfs directory +DefaultDependencies=no + +[Mount] +What=tmpfs +Where=/mnt/overlay +Type=tmpfs +Options=mode=1777,strictatime,nosuid,nodev + +[Install] +WantedBy=multi-user.target +EOT +} + +""" + + self.write_config(config) + self.add_overlay_conf_to_machine() + self.write_recipeinc('systemd-machine-units', systemd_machine_unit_append) + + bitbake('core-image-minimal') + + def getline_qemu(out, line): + for l in out.split('\n'): + if line in l: + return l + + with runqemu('core-image-minimal') as qemu: + # Check that we have /mnt/overlay fs mounted as tmpfs and + # /usr/share/my-application as an overlay (see overlayfs-user recipe) + status, output = qemu.run_serial("/bin/mount -t tmpfs,overlay") + + line = getline_qemu(output, "on /mnt/overlay") + self.assertTrue(line and line.startswith("tmpfs"), msg=output) + + line = getline_qemu(output, "upperdir=/mnt/overlay/upper/usr/share/my-application") + self.assertTrue(line and line.startswith("overlay"), msg=output)
Signature · 1 line
2.28.0
PATCH v3 8/8 docs: add overlayfs class
In reply to Vyacheslav Yurkov--- documentation/ref-manual/classes.rst | 46 ++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+)
documentation/ref-manual/classes.rst +46 -0
@@ -1708,6 +1708,52 @@ one such example. However, being aware of this class can reduce the proliferation of different versions of similar classes across multiple layers. +.. _ref-classes-overlayfs: + +``overlayfs.bbclass`` +======================= + + +It's often desired in Embedded System design to have a read-only rootfs. +But a lot of different applications might want to have a read-write access to +some parts of a filesystem. It can be especially useful when your update mechanism +overwrites the whole rootfs, but you want your application data to be preserved +between updates. The ``overlayfs`` class provides a way to achieve that by means +of overlayfs and at the same time keeping the base rootfs read-only. + +To use this class, set a mount point for a partition overlayfs is going to use as upper +layer in your machine configuration. Underlying file system can be anything that +is supported by overlayfs. This has to be done in your machine configuration:: + + OVERLAYFS_MOUNT_POINT[data] = "/data" + +.. note:: + + QA check fails to catch file existence if you redefine this variable in your recipe! + +The class assumes you have a data.mount systemd unit defined elsewhere in your BSP +(e.g. in systemd-machine-units recipe) and it's installed to the image. + +Then you can specify writable directories on a recipe basis:: + + OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application" + +To support several mount points you can use a different variable flag. Assume we +want to have a writable location on the file system, but not interested where the data +survive a reboot. Then we could have a mnt-overlay.mount unit for a tmpfs file system. + +In your machine configuration:: + + OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay" + +and then in your recipe:: + + OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application" + +.. note:: + + The class does not support /etc directory itself, because systemd depends on it + .. _ref-classes-own-mirrors: ``own-mirrors.bbclass``
Signature · 1 line
2.28.0
Hi,
Quoted 19 lines · Vyacheslav Yurkov
This class provides commom functions for overlayfs and its QA check, which is performed in
ROOTFS_POSTPROCESS_COMMANDSigned-off-by Vyacheslav Yurkov <[email protected]>---
meta/classes/overlayfs-qa.bbclass | 14 ++++++++++++++1 file changed, 14 insertions(+)create mode 100644 meta/classes/overlayfs-qa.bbclass
meta/classes/overlayfs-qa.bbclass new file +4
-0@@ -0,0 +1,14 @@ +# This class contains common functions for overlayfs and its QA check, +# which is performed in ROOTFS_POSTPROCESS_COMMAND + +# this function is based on https://github.com/systemd/systemd/blob/main/src/basic/unit-name.c
meta/classes/overlayfs-qa.bbclass line 14
Quoted 10 lines
meta/classes/overlayfs-qa.bbclass new file +10
-0+def escapeSystemdUnitName(path): + escapeMap = { + '/': '-', + '-': "\\x2d", + '\\': "\\x5d" + } + return "".join([escapeMap.get(c, c) for c in path.strip('/')]) + +def mountUnitName(unit): + return escapeSystemdUnitName(unit) + '.mount'
Thanks for working on this, it is passing our automated tests but I had some review comments. I think the above would make much more sense as code in meta/lib/oe/ somewhere as it is basically just function library code. There is a general intent to move the more generic python functions there and try to establish some better APIs so if you could move this there as new code, that would be great!
Cheers,
Richard