ref-manual: add overlayfs class
PATCH 1/1 ref-manual: add overlayfs class
--- documentation/ref-manual/classes.rst | 46 ++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+)
documentation/ref-manual/classes.rst +46 -0
@@ -1710,6 +1710,52 @@ one such example. However, being aware of this class can reduce the proliferation of different versions of similar classes across multiple layers. +.. _ref-classes-overlayfs: + +``overlayfs.bbclass`` +======================= + + +It's often desired in Embedded System design to have a read-only rootfs. +But a lot of different applications might want to have a read-write access to +some parts of a filesystem. It can be especially useful when your update mechanism +overwrites the whole rootfs, but you want your application data to be preserved +between updates. The ``overlayfs`` class provides a way to achieve that by means +of overlayfs and at the same time keeping the base rootfs read-only. + +To use this class, set a mount point for a partition overlayfs is going to use as upper +layer in your machine configuration. Underlying file system can be anything that +is supported by overlayfs. This has to be done in your machine configuration:: + + OVERLAYFS_MOUNT_POINT[data] = "/data" + +.. note:: + + QA check fails to catch file existence if you redefine this variable in your recipe! + +The class assumes you have a data.mount systemd unit defined elsewhere in your BSP +(e.g. in systemd-machine-units recipe) and it's installed to the image. + +Then you can specify writable directories on a recipe basis:: + + OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application" + +To support several mount points you can use a different variable flag. Assume we +want to have a writable location on the file system, but not interested where the data +survive a reboot. Then we could have a mnt-overlay.mount unit for a tmpfs file system. + +In your machine configuration:: + + OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay" + +and then in your recipe:: + + OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application" + +.. note:: + + The class does not support /etc directory itself, because systemd depends on it + .. _ref-classes-own-mirrors: ``own-mirrors.bbclass``
Signature · 1 line
2.28.0
Hi Vyacheslav,
Thanks for the patch!
Quoted 10 lines · Vyacheslav Yurkov
Signed-off-by Vyacheslav Yurkov <[email protected]>--- documentation/ref-manual/classes.rst | 46 ++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+)documentation/ref-manual/classes.rst +0
-0
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1718
Quoted 9 lines
documentation/ref-manual/classes.rst +6
-0@@ -1710,6 +1710,52 @@ one such example. However, being aware of this class can reduce the proliferation of different versions of similar classes across multiple layers. +.. _ref-classes-overlayfs: + +``overlayfs.bbclass`` +======================= + +
One newline is enough :)
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1723
Quoted 5 lines
documentation/ref-manual/classes.rst +5
-0+It's often desired in Embedded System design to have a read-only rootfs. +But a lot of different applications might want to have a read-write access to +some parts of a filesystem. It can be especially useful when your update mechanism +overwrites the whole rootfs, but you want your application data to be preserved +between updates. The ``overlayfs`` class provides a way to achieve that by means
s/``overlayfs``/:ref:overlayfs <ref-classes-overlayfs>/
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1725
Quoted 2 lines
documentation/ref-manual/classes.rst +2
-0+of overlayfs and at the same time keeping the base rootfs read-only. +
I would highlight overlayfs with tick quotes (``overlayfs``).
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1729
Quoted 4 lines
documentation/ref-manual/classes.rst +4
-0+To use this class, set a mount point for a partition overlayfs is going to use as upper +layer in your machine configuration. Underlying file system can be anything that +is supported by overlayfs. This has to be done in your machine configuration:: +
How do I know what's supported by overlayfs? Where can I find this important piece of information?
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1735
Quoted 6 lines
documentation/ref-manual/classes.rst +6
-0+ OVERLAYFS_MOUNT_POINT[data] = "/data" + +.. note:: + + QA check fails to catch file existence if you redefine this variable in your recipe! +
What about:
This variable must only be set from a configuration file as QA check will fail to catch file existence if this variable is also set in a recipe.
Also, I'm not sure to understand what "catch file existence" means? Could you explain to me please :)
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1736
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+The class assumes you have a data.mount systemd unit defined elsewhere in your BSP
s/data.mount/``data.mount``/
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1738
Quoted 2 lines
documentation/ref-manual/classes.rst +2
-0+(e.g. in systemd-machine-units recipe) and it's installed to the image. +
s/systemd-machine-units/``systemd-machine-units``/
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1740
Quoted 2 lines
documentation/ref-manual/classes.rst +2
-0+Then you can specify writable directories on a recipe basis:: +
If package recipes are supported, does it actually make sense to have such variable in there instead of in image recipes?
Just trying to know if we should highlight whether it's a bad idea for users to do so.
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1743
Quoted 3 lines
documentation/ref-manual/classes.rst +3
-0+ OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application" + +To support several mount points you can use a different variable flag. Assume we
s/Assume/Assuming/ ?
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1746
Quoted 3 lines
documentation/ref-manual/classes.rst +3
-0+want to have a writable location on the file system, but not interested where the data +survive a reboot. Then we could have a mnt-overlay.mount unit for a tmpfs file system. +
s/mnt-overlay.mount/``mnt-overlay.mount``/
s/tmpfs/``tmpfs``/ ?
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1758
Quoted 12 lines
documentation/ref-manual/classes.rst +12
-0+In your machine configuration:: + + OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay" + +and then in your recipe:: + + OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application" + +.. note:: + + The class does not support /etc directory itself, because systemd depends on it +
s/\/etc/``\/etc``/
s/systemd/``systemd``/
My nitpicks excluded, looks good :) ~
Cheers,
Quentin
Hi Quentin,
Thanks for the review! I'll address the notes in v2. A few answers to your questions are inlined below.
Quoted 65 lines · On Mon, Aug 16, 2021, 13:14 Quentin Schulz <…
Hi Vyacheslav,
Thanks for the patch!
Review
Quoted 6 lines · Vyacheslav Yurkov
Signed-off-by Vyacheslav Yurkov <[email protected]>--- documentation/ref-manual/classes.rst | 46 ++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+)(quoted diff) +0
-0
b/documentation/ref-manual/classes.rstdocumentation/ref-manual/classes.rst one such example. However, being aware of this
Quoted 4 lines
index
a98a64c43..5d33e81a1100644documentation/ref-manual/classes.rst +0
-0@@ -1710,6 +1710,52 @@ one such example. However, being aware of thisclass can reduce the
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1718
Quoted 9 lines
documentation/ref-manual/classes.rst +6
-0proliferation of different versions of similar classes across multiple layers. +.. _ref-classes-overlayfs: + +``overlayfs.bbclass`` +======================= + +One newline is enough :)
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1720
Quoted 2 lines
documentation/ref-manual/classes.rst +2
-0+It's often desired in Embedded System design to have a read-only rootfs. +But a lot of different applications might want to have a read-writeaccess to
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1721
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+some parts of a filesystem. It can be especially useful when yourupdate mechanism
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1722
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+overwrites the whole rootfs, but you want your application data to bepreserved
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1723
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+between updates. The ``overlayfs`` class provides a way to achieve thatby means
s/``overlayfs``/:ref:
overlayfs <ref-classes-overlayfs>/documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1725
Quoted 2 lines
documentation/ref-manual/classes.rst +2
-0+of overlayfs and at the same time keeping the base rootfs read-only. +I would highlight overlayfs with tick quotes (``overlayfs``).
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1726
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+To use this class, set a mount point for a partition overlayfs is goingto use as upper
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1727
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+layer in your machine configuration. Underlying file system can beanything that
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1728
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+is supported by overlayfs. This has to be done in your machineconfiguration::
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1729
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+How do I know what's supported by overlayfs? Where can I find this important piece of information?
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the
Quoted 5 lines
documentation/ref-manual/classes.rst +5
-0+ OVERLAYFS_MOUNT_POINT[data] = "/data" + +.. note:: + + QA check fails to catch file existence if you redefine this variablein your recipe!
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1735
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+
Quoted 8 lines
What about:
This variable must only be set from a configuration file as QA check will fail
to catch file existence if this variable is also set in a recipe.Also, I'm not sure to understand what "catch file existence" means? Could you explain to me please :)
That means I check only that file exists in QA check, I do not check whether it's a valid systemd mount unit (to avoid additional dependency on systemd-native recipe).
Quoted 10 lines
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1736
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+The class assumes you have a data.mount systemd unit defined elsewherein your BSP
s/data.mount/``data.mount``/
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1738
Quoted 2 lines
documentation/ref-manual/classes.rst +2
-0+(e.g. in systemd-machine-units recipe) and it's installed to the image. +s/systemd-machine-units/``systemd-machine-units``/
Quoted 9 lines
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1740
Quoted 2 lines
documentation/ref-manual/classes.rst +2
-0+Then you can specify writable directories on a recipe basis:: +If package recipes are supported, does it actually make sense to have such variable in there instead of in image recipes?
Just trying to know if we should highlight whether it's a bad idea for users to do so.
There are actually two pitfalls here.
Semantically directory belongs to a recipe it's used in. If you redefine OVERLAYFS_WRITABLE_PATHS in another recipe, do_rootfs would fail when two recipes install the same file.
Another trade-off I had to deal with is if you redefine OVERLAYFS_MOUNT_POINT in a recipe, then I can't catch it in my QA check. This is due to a bitbake limitation, in particular I can't forbid further variable overrides. That's why I mentioned above that this variable has to be set in machine configuration.
I hope that makes sense :)
Regards,
Vyacheslav
Quoted 40 lines
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the
Quoted 3 lines
documentation/ref-manual/classes.rst +3
-0+ OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application" + +To support several mount points you can use a different variable flag.Assume we
s/Assume/Assuming/ ?
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1744
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+want to have a writable location on the file system, but not interestedwhere the data
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1745
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+survive a reboot. Then we could have a mnt-overlay.mount unit for atmpfs file system.
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1746
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+s/mnt-overlay.mount/``mnt-overlay.mount``/
s/tmpfs/``tmpfs``/ ?
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the
Quoted 7 lines
documentation/ref-manual/classes.rst +7
-0+In your machine configuration:: + + OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay" + +and then in your recipe:: + + OVERLAYFS_WRITABLE_PATHS[mnt-overlay] ="
/usr/share/another-application"documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the
Quoted 4 lines
documentation/ref-manual/classes.rst +4
-0+ +.. note:: + + The class does not support /etc directory itself, because systemddepends on it
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1758
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+s/\/etc/``\/etc``/
s/systemd/``systemd``/
My nitpicks excluded, looks good :) ~
Cheers,
Quentin
Hi Vyacheslav,
Quoted 77 lines · Vyacheslav Yurkov
Hi Quentin,
Thanks for the review! I'll address the notes in v2. A few answers to your questions are inlined below.Quoted 70 lines · On Mon, Aug 16, 2021, 13:14 Quentin Schulz <…
Hi Vyacheslav,
Thanks for the patch!
Review
Quoted 6 lines · Vyacheslav Yurkov
Signed-off-by Vyacheslav Yurkov <[email protected]>--- documentation/ref-manual/classes.rst | 46 ++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+)(quoted diff) +0
-0
b/documentation/ref-manual/classes.rstdocumentation/ref-manual/classes.rst one such example. However, being aware of this
Quoted 4 lines
index
a98a64c43..5d33e81a1100644documentation/ref-manual/classes.rst +0
-0@@ -1710,6 +1710,52 @@ one such example. However, being aware of thisclass can reduce the
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1718
Quoted 9 lines
documentation/ref-manual/classes.rst +6
-0proliferation of different versions of similar classes across multiple layers. +.. _ref-classes-overlayfs: + +``overlayfs.bbclass`` +======================= + +One newline is enough :)
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1720
Quoted 2 lines
documentation/ref-manual/classes.rst +2
-0+It's often desired in Embedded System design to have a read-only rootfs. +But a lot of different applications might want to have a read-writeaccess to
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1721
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+some parts of a filesystem. It can be especially useful when yourupdate mechanism
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1722
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+overwrites the whole rootfs, but you want your application data to bepreserved
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1723
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+between updates. The ``overlayfs`` class provides a way to achieve thatby means
s/``overlayfs``/:ref:
overlayfs <ref-classes-overlayfs>/documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1725
Quoted 2 lines
documentation/ref-manual/classes.rst +2
-0+of overlayfs and at the same time keeping the base rootfs read-only. +I would highlight overlayfs with tick quotes (``overlayfs``).
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1726
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+To use this class, set a mount point for a partition overlayfs is goingto use as upper
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1727
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+layer in your machine configuration. Underlying file system can beanything that
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1728
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+is supported by overlayfs. This has to be done in your machineconfiguration::
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1729
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+How do I know what's supported by overlayfs? Where can I find this important piece of information?
documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the
Quoted 5 lines
documentation/ref-manual/classes.rst +5
-0+ OVERLAYFS_MOUNT_POINT[data] = "/data" + +.. note:: + + QA check fails to catch file existence if you redefine this variablein your recipe!
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1735
Quoted 1 line
documentation/ref-manual/classes.rst +1
-0+What about:
This variable must only be set from a configuration file as QA check will fail
to catch file existence if this variable is also set in a recipe.
Quoted 8 lines
Quoted 3 lines
Also, I'm not sure to understand what "catch file existence" means? Could you explain to me please :)
That means I check only that file exists in QA check, I do not check whether it's a valid systemd mount unit (to avoid additional dependency on systemd-native recipe).
I think what I was missing is that the QA check is actually part of the ROOTFS_POSTPROCESS_COMMAND variable. Therefore, only the image recipe will do the QA check.
I think we should highlight that this variable should only be set in configuration files OR image recipes. (or basically, say that it shouldn't be set in package recipes).
So I'd actually add two notes:
1) Variables to be defined in conf files or image recipes (or say the
opposite: not in package recipes)
2) Only existence of the systemd mount unit file is checked, not its content.
Quoted 27 lines
Quoted 11 lines
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1736
Quoted 2 lines
documentation/ref-manual/classes.rst +1
-0+The class assumes you have a data.mount systemd unit defined elsewherein your BSP
s/data.mount/``data.mount``/
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1738
Quoted 2 lines
documentation/ref-manual/classes.rst +2
-0+(e.g. in systemd-machine-units recipe) and it's installed to the image. +s/systemd-machine-units/``systemd-machine-units``/
Quoted 9 lines
documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1740
Quoted 2 lines
documentation/ref-manual/classes.rst +2
-0+Then you can specify writable directories on a recipe basis:: +If package recipes are supported, does it actually make sense to have such variable in there instead of in image recipes?
Just trying to know if we should highlight whether it's a bad idea for users to do so.
There are actually two pitfalls here.
Semantically directory belongs to a recipe it's used in. If you redefine
OVERLAYFS_WRITABLE_PATHSin another recipe,do_rootfswould fail when two recipes install the same file.
Quoted 6 lines
Another trade-off I had to deal with is if you redefine
OVERLAYFS_MOUNT_POINTin a recipe, then I can't catch it in my QA check. This is due to a bitbake limitation, in particular I can't forbid further variable overrides. That's why I mentioned above that this variable has to be set in machine configuration.
I feel like OVERLAYFS configuration makes sense only when defined globally (conf files) or in image recipes.
Just thinking out loud right now but couldn't we add the following in overlayfs.bbclass anonymous python code?
if not bb.data.inherits_class('image', d):
bb.fatal('Only image recipes can inherit overlayfs.bbclass. %s
is a package recipe.' % d.getVar('${PN}'))or something like that?
What are your thoughts on my remarks?
Lemme know!
Regards,
Quentin
Hi Quentin
Quoted 8 lines
Quoted 4 lines
That means I check only that file exists in QA check, I do not check whether it's a valid systemd mount unit (to avoid additional dependency on systemd-native recipe).
I think what I was missing is that the QA check is actually part of the
ROOTFS_POSTPROCESS_COMMANDvariable. Therefore, only the image recipe will do the QA check.
Quoted 8 lines
I think we should highlight that this variable should only be set in configuration files OR image recipes. (or basically, say that it shouldn't be set in package recipes).
So I'd actually add two notes:
1) Variables to be defined in conf files or image recipes (or say the
opposite: not in package recipes)
Not entirely correct, see my other answer below.
Quoted 2 lines
2) Only existence of the systemd mount unit file is checked, not its content.
Agreed.
Quoted 6 lines
I feel like OVERLAYFS configuration makes sense only when defined globally (conf files) or in image recipes.
Just thinking out loud right now but couldn't we add the following in overlayfs.bbclass anonymous python code?
Quoted 7 lines
if not
bb.data.inherits_class('image', d):bb.fatal('Only image recipes can inherit overlayfs.bbclass. %s is a package recipe.' % d.getVar('${PN}'))or something like that?
What are your thoughts on my remarks?
To sum it up:
- OVERLAYFS_MOUNT_POINT should be defined globally and _not_ redefined afterwards.
- OVERLAYFS_WRITABLE_PATHS can be used in any recipe (I guess not a package group though). It doesn't have to be image recipe, because a directory specified there can be application / recipe specific. For example, if we have a recipe my-app.bb and an application there wants to have a writable path in /usr/share/my-app, then it makes sense to set it in the my-app recipe, not in the image recipe, right?
- QA check is part of ROOTFS_POSTPROCESS_COMMAND as you noted before and is done as one of the last step during image creation.
Quoted 4 lines
Lemme know!
Regards,
Quentin
Cheers,
Vyacheslav