Skip to contentSkip to conversation

ref-manual: add overlayfs class

PATCH Superseded Vyacheslav Yurkov 2 participants 5 messages Aug 14, 2021 – Aug 17, 2021

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ref-manual: add overlayfs class

---
 documentation/ref-manual/classes.rst | 46 ++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
documentation/ref-manual/classes.rst +46 -0
index a98a64c43..5d33e81a1 100644
@@ -1710,6 +1710,52 @@ one such example. However, being aware of this class can reduce the
 proliferation of different versions of similar classes across multiple
 layers.
 
+.. _ref-classes-overlayfs:
+
+``overlayfs.bbclass``
+=======================
+
+
+It's often desired in Embedded System design to have a read-only rootfs.
+But a lot of different applications might want to have a read-write access to
+some parts of a filesystem. It can be especially useful when your update mechanism
+overwrites the whole rootfs, but you want your application data to be preserved
+between updates. The ``overlayfs`` class provides a way to achieve that by means
+of overlayfs and at the same time keeping the base rootfs read-only.
+
+To use this class, set a mount point for a partition overlayfs is going to use as upper
+layer in your machine configuration. Underlying file system can be anything that
+is supported by overlayfs. This has to be done in your machine configuration::
+
+  OVERLAYFS_MOUNT_POINT[data] = "/data"
+
+.. note::
+
+  QA check fails to catch file existence if you redefine this variable in your recipe!
+
+The class assumes you have a data.mount systemd unit defined elsewhere in your BSP
+(e.g. in systemd-machine-units recipe) and it's installed to the image.
+
+Then you can specify writable directories on a recipe basis::
+
+  OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
+
+To support several mount points you can use a different variable flag. Assume we
+want to have a writable location on the file system, but not interested where the data
+survive a reboot. Then we could have a mnt-overlay.mount unit for a tmpfs file system.
+
+In your machine configuration::
+
+  OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
+
+and then in your recipe::
+
+  OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
+
+.. note::
+
+   The class does not support /etc directory itself, because systemd depends on it
+
 .. _ref-classes-own-mirrors:
 
 ``own-mirrors.bbclass``
Signature · 1 line
2.28.0
#1 lore raw

Hi Vyacheslav,

Thanks for the patch!

Quoted 10 lines · Vyacheslav Yurkov
---
 documentation/ref-manual/classes.rst | 46 ++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
documentation/ref-manual/classes.rst +0 -0
index a98a64c43..5d33e81a1 100644

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1718

Quoted 9 lines
documentation/ref-manual/classes.rst +6 -0
@@ -1710,6 +1710,52 @@ one such example. However, being aware of this class can reduce the
 proliferation of different versions of similar classes across multiple
 layers.
 
+.. _ref-classes-overlayfs:
+
+``overlayfs.bbclass``
+=======================
+
+

One newline is enough :)

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1723

Quoted 5 lines
documentation/ref-manual/classes.rst +5 -0
+It's often desired in Embedded System design to have a read-only rootfs.
+But a lot of different applications might want to have a read-write access to
+some parts of a filesystem. It can be especially useful when your update mechanism
+overwrites the whole rootfs, but you want your application data to be preserved
+between updates. The ``overlayfs`` class provides a way to achieve that by means

s/``overlayfs``/:ref:overlayfs <ref-classes-overlayfs>/

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1725

Quoted 2 lines
documentation/ref-manual/classes.rst +2 -0
+of overlayfs and at the same time keeping the base rootfs read-only.
+

I would highlight overlayfs with tick quotes (``overlayfs``).

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1729

Quoted 4 lines
documentation/ref-manual/classes.rst +4 -0
+To use this class, set a mount point for a partition overlayfs is going to use as upper
+layer in your machine configuration. Underlying file system can be anything that
+is supported by overlayfs. This has to be done in your machine configuration::
+

How do I know what's supported by overlayfs? Where can I find this important piece of information?

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1735

Quoted 6 lines
documentation/ref-manual/classes.rst +6 -0
+  OVERLAYFS_MOUNT_POINT[data] = "/data"
+
+.. note::
+
+  QA check fails to catch file existence if you redefine this variable in your recipe!
+

What about:
This variable must only be set from a configuration file as QA check will fail to catch file existence if this variable is also set in a recipe.

Also, I'm not sure to understand what "catch file existence" means? Could you explain to me please :)

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1736

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+The class assumes you have a data.mount systemd unit defined elsewhere in your BSP

s/data.mount/``data.mount``/

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1738

Quoted 2 lines
documentation/ref-manual/classes.rst +2 -0
+(e.g. in systemd-machine-units recipe) and it's installed to the image.
+

s/systemd-machine-units/``systemd-machine-units``/

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1740

Quoted 2 lines
documentation/ref-manual/classes.rst +2 -0
+Then you can specify writable directories on a recipe basis::
+

If package recipes are supported, does it actually make sense to have such variable in there instead of in image recipes?

Just trying to know if we should highlight whether it's a bad idea for users to do so.

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1743

Quoted 3 lines
documentation/ref-manual/classes.rst +3 -0
+  OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
+
+To support several mount points you can use a different variable flag. Assume we

s/Assume/Assuming/ ?

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1746

Quoted 3 lines
documentation/ref-manual/classes.rst +3 -0
+want to have a writable location on the file system, but not interested where the data
+survive a reboot. Then we could have a mnt-overlay.mount unit for a tmpfs file system.
+

s/mnt-overlay.mount/``mnt-overlay.mount``/

s/tmpfs/``tmpfs``/ ?

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1758

Quoted 12 lines
documentation/ref-manual/classes.rst +12 -0
+In your machine configuration::
+
+  OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
+
+and then in your recipe::
+
+  OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
+
+.. note::
+
+   The class does not support /etc directory itself, because systemd depends on it
+

s/\/etc/``\/etc``/

s/systemd/``systemd``/

My nitpicks excluded, looks good :) ~

Cheers,
Quentin

#2 lore raw

Hi Quentin,
Thanks for the review! I'll address the notes in v2. A few answers to your questions are inlined below.

Quoted 65 lines · On Mon, Aug 16, 2021, 13:14 Quentin Schulz <…

Hi Vyacheslav,

Thanks for the patch!

Review

Quoted 6 lines · Vyacheslav Yurkov
---
 documentation/ref-manual/classes.rst | 46 ++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
(quoted diff) +0 -0

b/documentation/ref-manual/classes.rst

documentation/ref-manual/classes.rst one such example. However, being aware of this

Quoted 4 lines

index a98a64c43..5d33e81a1 100644

documentation/ref-manual/classes.rst +0 -0
@@ -1710,6 +1710,52 @@ one such example. However, being aware of this

class can reduce the

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1718

Quoted 9 lines
documentation/ref-manual/classes.rst +6 -0
 proliferation of different versions of similar classes across multiple
 layers.
 
+.. _ref-classes-overlayfs:
+
+``overlayfs.bbclass``
+=======================
+
+

One newline is enough :)

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1720

Quoted 2 lines
documentation/ref-manual/classes.rst +2 -0
+It's often desired in Embedded System design to have a read-only rootfs.
+But a lot of different applications might want to have a read-write

access to

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1721

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+some parts of a filesystem. It can be especially useful when your

update mechanism

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1722

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+overwrites the whole rootfs, but you want your application data to be

preserved

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1723

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+between updates. The ``overlayfs`` class provides a way to achieve that

by means

s/``overlayfs``/:ref:overlayfs <ref-classes-overlayfs>/

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1725

Quoted 2 lines
documentation/ref-manual/classes.rst +2 -0
+of overlayfs and at the same time keeping the base rootfs read-only.
+

I would highlight overlayfs with tick quotes (``overlayfs``).

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1726

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+To use this class, set a mount point for a partition overlayfs is going

to use as upper

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1727

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+layer in your machine configuration. Underlying file system can be

anything that

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1728

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+is supported by overlayfs. This has to be done in your machine

configuration::

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1729

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+

How do I know what's supported by overlayfs? Where can I find this important piece of information?

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the

Quoted 5 lines
documentation/ref-manual/classes.rst +5 -0
+  OVERLAYFS_MOUNT_POINT[data] = "/data"
+
+.. note::
+
+  QA check fails to catch file existence if you redefine this variable

in your recipe!

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1735

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+
Quoted 8 lines

What about:
This variable must only be set from a configuration file as QA check will fail
to catch file existence if this variable is also set in a recipe.

Also, I'm not sure to understand what "catch file existence" means? Could you explain to me please :)

That means I check only that file exists in QA check, I do not check whether it's a valid systemd mount unit (to avoid additional dependency on systemd-native recipe).

Quoted 10 lines

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1736

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+The class assumes you have a data.mount systemd unit defined elsewhere

in your BSP

s/data.mount/``data.mount``/

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1738

Quoted 2 lines
documentation/ref-manual/classes.rst +2 -0
+(e.g. in systemd-machine-units recipe) and it's installed to the image.
+

s/systemd-machine-units/``systemd-machine-units``/

Quoted 9 lines

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1740

Quoted 2 lines
documentation/ref-manual/classes.rst +2 -0
+Then you can specify writable directories on a recipe basis::
+

If package recipes are supported, does it actually make sense to have such variable in there instead of in image recipes?

Just trying to know if we should highlight whether it's a bad idea for users to do so.

There are actually two pitfalls here.

Semantically directory belongs to a recipe it's used in. If you redefine OVERLAYFS_WRITABLE_PATHS in another recipe, do_rootfs would fail when two recipes install the same file.

Another trade-off I had to deal with is if you redefine OVERLAYFS_MOUNT_POINT in a recipe, then I can't catch it in my QA check. This is due to a bitbake limitation, in particular I can't forbid further variable overrides. That's why I mentioned above that this variable has to be set in machine configuration.

I hope that makes sense :)

Regards,
Vyacheslav

Quoted 40 lines

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the

Quoted 3 lines
documentation/ref-manual/classes.rst +3 -0
+  OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
+
+To support several mount points you can use a different variable flag.

Assume we

s/Assume/Assuming/ ?

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1744

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+want to have a writable location on the file system, but not interested

where the data

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1745

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+survive a reboot. Then we could have a mnt-overlay.mount unit for a

tmpfs file system.

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1746

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+

s/mnt-overlay.mount/``mnt-overlay.mount``/

s/tmpfs/``tmpfs``/ ?

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the

Quoted 7 lines
documentation/ref-manual/classes.rst +7 -0
+In your machine configuration::
+
+  OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
+
+and then in your recipe::
+
+  OVERLAYFS_WRITABLE_PATHS[mnt-overlay] =

"/usr/share/another-application"

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the

Quoted 4 lines
documentation/ref-manual/classes.rst +4 -0
+
+.. note::
+
+   The class does not support /etc directory itself, because systemd

depends on it

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1758

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+

s/\/etc/``\/etc``/

s/systemd/``systemd``/

My nitpicks excluded, looks good :) ~

Cheers,
Quentin

#3 lore raw

Hi Vyacheslav,

Quoted 77 lines · Vyacheslav Yurkov

Hi Quentin,
Thanks for the review! I'll address the notes in v2. A few answers to your questions are inlined below.

Quoted 70 lines · On Mon, Aug 16, 2021, 13:14 Quentin Schulz <…

Hi Vyacheslav,

Thanks for the patch!

Review

Quoted 6 lines · Vyacheslav Yurkov
---
 documentation/ref-manual/classes.rst | 46 ++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
(quoted diff) +0 -0

b/documentation/ref-manual/classes.rst

documentation/ref-manual/classes.rst one such example. However, being aware of this

Quoted 4 lines

index a98a64c43..5d33e81a1 100644

documentation/ref-manual/classes.rst +0 -0
@@ -1710,6 +1710,52 @@ one such example. However, being aware of this

class can reduce the

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the · line 1718

Quoted 9 lines
documentation/ref-manual/classes.rst +6 -0
 proliferation of different versions of similar classes across multiple
 layers.
 
+.. _ref-classes-overlayfs:
+
+``overlayfs.bbclass``
+=======================
+
+

One newline is enough :)

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1720

Quoted 2 lines
documentation/ref-manual/classes.rst +2 -0
+It's often desired in Embedded System design to have a read-only rootfs.
+But a lot of different applications might want to have a read-write

access to

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1721

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+some parts of a filesystem. It can be especially useful when your

update mechanism

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1722

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+overwrites the whole rootfs, but you want your application data to be

preserved

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1723

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+between updates. The ``overlayfs`` class provides a way to achieve that

by means

s/``overlayfs``/:ref:overlayfs <ref-classes-overlayfs>/

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1725

Quoted 2 lines
documentation/ref-manual/classes.rst +2 -0
+of overlayfs and at the same time keeping the base rootfs read-only.
+

I would highlight overlayfs with tick quotes (``overlayfs``).

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1726

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+To use this class, set a mount point for a partition overlayfs is going

to use as upper

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1727

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+layer in your machine configuration. Underlying file system can be

anything that

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1728

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+is supported by overlayfs. This has to be done in your machine

configuration::

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1729

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+

How do I know what's supported by overlayfs? Where can I find this important piece of information?

documentation/ref-manual/classes.rst one such example. However, being aware of this class can reduce the

Quoted 5 lines
documentation/ref-manual/classes.rst +5 -0
+  OVERLAYFS_MOUNT_POINT[data] = "/data"
+
+.. note::
+
+  QA check fails to catch file existence if you redefine this variable

in your recipe!

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1735

Quoted 1 line
documentation/ref-manual/classes.rst +1 -0
+

What about:
This variable must only be set from a configuration file as QA check will fail
to catch file existence if this variable is also set in a recipe.

Quoted 8 lines
Quoted 3 lines

Also, I'm not sure to understand what "catch file existence" means? Could you explain to me please :)

That means I check only that file exists in QA check, I do not check whether it's a valid systemd mount unit (to avoid additional dependency on systemd-native recipe).

I think what I was missing is that the QA check is actually part of the ROOTFS_POSTPROCESS_COMMAND variable. Therefore, only the image recipe will do the QA check.

I think we should highlight that this variable should only be set in configuration files OR image recipes. (or basically, say that it shouldn't be set in package recipes).

So I'd actually add two notes:

1) Variables to be defined in conf files or image recipes (or say the
opposite: not in package recipes)
2) Only existence of the systemd mount unit file is checked, not its content.

Quoted 27 lines
Quoted 11 lines

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1736

Quoted 2 lines
documentation/ref-manual/classes.rst +1 -0
+The class assumes you have a data.mount systemd unit defined elsewhere

in your BSP

s/data.mount/``data.mount``/

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1738

Quoted 2 lines
documentation/ref-manual/classes.rst +2 -0
+(e.g. in systemd-machine-units recipe) and it's installed to the image.
+

s/systemd-machine-units/``systemd-machine-units``/

Quoted 9 lines

documentation/ref-manual/classes.rst one such example. However, being aware of this · line 1740

Quoted 2 lines
documentation/ref-manual/classes.rst +2 -0
+Then you can specify writable directories on a recipe basis::
+

If package recipes are supported, does it actually make sense to have such variable in there instead of in image recipes?

Just trying to know if we should highlight whether it's a bad idea for users to do so.

There are actually two pitfalls here.

Semantically directory belongs to a recipe it's used in. If you redefine OVERLAYFS_WRITABLE_PATHS in another recipe, do_rootfs would fail when two recipes install the same file.

Quoted 6 lines

Another trade-off I had to deal with is if you redefine OVERLAYFS_MOUNT_POINT in a recipe, then I can't catch it in my QA check. This is due to a bitbake limitation, in particular I can't forbid further variable overrides. That's why I mentioned above that this variable has to be set in machine configuration.

I feel like OVERLAYFS configuration makes sense only when defined globally (conf files) or in image recipes.

Just thinking out loud right now but couldn't we add the following in overlayfs.bbclass anonymous python code?

if not bb.data.inherits_class('image', d):

	bb.fatal('Only image recipes can inherit overlayfs.bbclass. %s
		 is a package recipe.' % d.getVar('${PN}'))

or something like that?

What are your thoughts on my remarks?

Lemme know!

Regards,
Quentin

#4 lore raw

Hi Quentin

Quoted 8 lines
Quoted 4 lines

That means I check only that file exists in QA check, I do not check whether it's a valid systemd mount unit (to avoid additional dependency on systemd-native recipe).

I think what I was missing is that the QA check is actually part of the ROOTFS_POSTPROCESS_COMMAND variable. Therefore, only the image recipe will do the QA check.

Quoted 8 lines

I think we should highlight that this variable should only be set in configuration files OR image recipes. (or basically, say that it shouldn't be set in package recipes).

So I'd actually add two notes:

1) Variables to be defined in conf files or image recipes (or say the
opposite: not in package recipes)

Not entirely correct, see my other answer below.

Quoted 2 lines

2) Only existence of the systemd mount unit file is checked, not its content.

Agreed.

Quoted 6 lines

I feel like OVERLAYFS configuration makes sense only when defined globally (conf files) or in image recipes.

Just thinking out loud right now but couldn't we add the following in overlayfs.bbclass anonymous python code?

Quoted 7 lines

if not bb.data.inherits_class('image', d):

	bb.fatal('Only image recipes can inherit overlayfs.bbclass. %s
		 is a package recipe.' % d.getVar('${PN}'))

or something like that?

What are your thoughts on my remarks?

To sum it up:
- OVERLAYFS_MOUNT_POINT should be defined globally and _not_ redefined afterwards.
- OVERLAYFS_WRITABLE_PATHS can be used in any recipe (I guess not a package group though). It doesn't have to be image recipe, because a directory specified there can be application / recipe specific. For example, if we have a recipe my-app.bb and an application there wants to have a writable path in /usr/share/my-app, then it makes sense to set it in the my-app recipe, not in the image recipe, right?
- QA check is part of ROOTFS_POSTPROCESS_COMMAND as you noted before and is done as one of the last step during image creation.

Quoted 4 lines

Lemme know!

Regards,
Quentin

Cheers,
Vyacheslav