Alternative to the filesystem overlay
In OpenBMC we've been using an overlay to make the read-only filesystem (or part of it) writable. It hasn't come without its issues, like missing directories (openbmc/openbmc#3564) and various flavors of corruption (openbmc/openbmc#3578 and issues when adding files to /etc in the build causing files that had been previously modified on the bmc to become corrupted).
An alternative would be to use a bind mount and manage the file merging by an OpenBMC app. Doing some experimentation, seems this solves the issues seen above. The way that it'd work would be:
- The init script or initramfs would do an rsync from /etc to the bind mount destination directory in the read-write filesystem, skipping the contents of a list which would contain the files that have been modified in the BMC and should be preserved.
- A new OpenBMC app would monitor /etc for changes via inotify for example, and if a file is modified, it'd add it to the "list".
Considerations:
- The /etc dir mainly being a configuration destination does not get modified often so the app that is watching the directory would not be triggered often.
- The bind mount would duplicate the contents of etc although it's not much. If space is a concern, some exploration could be done to have the bind mount be a tmpfs and have the monitor app copy the modified files to the read-write filesystem instead of adding them to a list.
Thoughts?
Quoted 7 lines · Adriana Kobylak
In OpenBMC we've been using an overlay to make the read-only filesystem (or part of it) writable. It hasn't come without its issues, like missing directories (openbmc/openbmc#3564) and various flavors of corruption (openbmc/openbmc#3578 and issues when adding files to /etc in the build causing files that had been previously modified on the bmc to become corrupted).
The overlay sounds ideal (when it works) because we don't need all of the special code and logic you mention below. Any chance we can just fix whatever is causing the corruption?
Quoted 10 lines
An alternative would be to use a bind mount and manage the file merging by an OpenBMC app. Doing some experimentation, seems this solves the issues seen above. The way that it'd work would be:
- The init script or initramfs would do an rsync from /etc to the bind mount destination directory in the read-write filesystem, skipping the contents of a list which would contain the files that have been modified in the BMC and should be preserved.
- A new OpenBMC app would monitor /etc for changes via inotify for example, and if a file is modified, it'd add it to the "list".
We always have race condition here though? Someone updates the BMC network info and before the app can detect it they reboot the BMC and we lose the settings.
Any idea what other flash based distros do out there? Seems like this would be a common problem for a lot of projects.
Quoted 12 lines
Considerations:
- The /etc dir mainly being a configuration destination does not get modified often so the app that is watching the directory would not be triggered often.
- The bind mount would duplicate the contents of etc although it's not much. If space is a concern, some exploration could be done to have the bind mount be a tmpfs and have the monitor app copy the modified files to the read-write filesystem instead of adding them to a list.Thoughts?
Quoted 6 lines · Adriana Kobylak
In OpenBMC we've been using an overlay to make the read-only filesystem (or part of it) writable. It hasn't come without its issues, like missing directories (openbmc/openbmc#3564) and various flavors of corruption (openbmc/openbmc#3578 and issues when adding files to /etc in the build causing files that had been previously modified on the bmc to become corrupted).
There is one more issue for the same type - openbmc/openbmc#3468. And it can't be solved by the way you propose.
Quoted 20 lines
An alternative would be to use a bind mount and manage the file merging by an OpenBMC app. Doing some experimentation, seems this solves the issues seen above. The way that it'd work would be:
- The init script or initramfs would do an rsync from /etc to the bind mount destination directory in the read-write filesystem, skipping the contents of a list which would contain the files that have been modified in the BMC and should be preserved.
- A new OpenBMC app would monitor /etc for changes via inotify for example, and if a file is modified, it'd add it to the "list".Considerations:
- The /etc dir mainly being a configuration destination does not get modified often so the app that is watching the directory would not be triggered often.
- The bind mount would duplicate the contents of etc although it's not much. If space is a concern, some exploration could be done to have the bind mount be a tmpfs and have the monitor app copy the modified files to the read-write filesystem instead of adding them to a list.
Quoted 2 lines
Thoughts?
Most of these issues can be solved by the simple shell scripts. We could modify the initramfs/init scenario to exec these scripts at the moment when ROFS and RWFS have mounted but are not combined to OVERLAY yet.
Of course, this method doesn't prevent an occurrence of any new issues of this type but brings an easy way to solve them.
--
Regards,
Alexander
Sounds like this is just a data schema update type of issue, where you just need a service to run at first update boot and upgrade the files as expected by the change.
We could probably do this with systemd services that are ConditionNeedsUpdate= and
https://www.freedesktop.org/software/systemd/man/systemd-update-done.service.html
I'd prefer to see us move to a mostly immutable filesystem with symlinks that map into a rwfs for specific files / directories we expect to mutate at runtime. I don't think we should do any smart types of writeback caching at the filesystem level as that will just lead to confusion about persistence that is hard to debug. Not to mention tools like rsync don't guarantee atomic snapshotting during the copy for file consistency. Our applications should be designed in such a way that they are always consistent on disk. Ex. writing text files should be done to a temporary, then moved over the old one. That way the update is atomic and we don't have partially written files.
Quoted 45 lines · Alexander A. Filippov
Quoted 6 lines · Adriana Kobylak
In OpenBMC we've been using an overlay to make the read-only filesystem (or part of it) writable. It hasn't come without its issues, like missing directories (openbmc/openbmc#3564) and various flavors of corruption (openbmc/openbmc#3578 and issues when adding files to /etc in the build causing files that had been previously modified on the bmc to become corrupted).
There is one more issue for the same type - openbmc/openbmc#3468. And it can't be solved by the way you propose.
Quoted 20 lines
An alternative would be to use a bind mount and manage the file merging by an OpenBMC app. Doing some experimentation, seems this solves the issues seen above. The way that it'd work would be:
- The init script or initramfs would do an rsync from /etc to the bind mount destination directory in the read-write filesystem, skipping the contents of a list which would contain the files that have been modified in the BMC and should be preserved.
- A new OpenBMC app would monitor /etc for changes via inotify for example, and if a file is modified, it'd add it to the "list".Considerations:
- The /etc dir mainly being a configuration destination does not get modified often so the app that is watching the directory would not be triggered often.
- The bind mount would duplicate the contents of etc although it's not much. If space is a concern, some exploration could be done to have the bind mount be a tmpfs and have the monitor app copy the modified files to the read-write filesystem instead of adding them to a list.Quoted 2 lines
Thoughts?
Most of these issues can be solved by the simple shell scripts. We could modify the initramfs/init scenario to exec these scripts at the moment when ROFS and RWFS have mounted but are not combined to OVERLAY yet.
Of course, this method doesn't prevent an occurrence of any new issues of this type but brings an easy way to solve them.
--
Regards,
Alexander
Quoted 11 lines · Alexander A. Filippov
Most of these issues can be solved by the simple shell scripts. We could modify the initramfs/init scenario to exec these scripts at the moment when ROFS and RWFS have mounted but are not combined to OVERLAY yet.
Of course, this method doesn't prevent an occurrence of any new issues of this type but brings an easy way to solve them.
--
Regards,
Alexander
Possibly the overlay FS is not expecting and does not support the ROFS to change as it does when a new build adds a file to an existing directory. When this occurs an overlayed file in the same directory could cause corruption in the directory. Are you suggesting scripts that detect a change to the underlying FS and attempt to patch the overlay?
Thanks,
Terry
Quoted 14 lines · Terry Duncan
Quoted 11 lines · Alexander A. Filippov
Most of these issues can be solved by the simple shell scripts. We could modify the initramfs/init scenario to exec these scripts at the moment when ROFS and RWFS have mounted but are not combined to OVERLAY yet.
Of course, this method doesn't prevent an occurrence of any new issues of this type but brings an easy way to solve them.
--
Regards,
Alexander
Quoted 8 lines
Possibly the overlay FS is not expecting and does not support the ROFS to change as it does when a new build adds a file to an existing directory. When this occurs an overlayed file in the same directory could cause corruption in the directory. Are you suggesting scripts that detect a change to the underlying FS and attempt to patch the overlay?
Thanks,
Terry
That's not entirely true. I'm suggesting to keep the scripts to solve each particular issue. But if we can detect and resolve such issues automatically that would be nice.
Regards,
Alexander
Quoted 8 lines · William Kennington
Sounds like this is just a data schema update type of issue, where you just need a service to run at first update boot and upgrade the files as expected by the change.
We could probably do this with systemd services that are ConditionNeedsUpdate= and
https://www.freedesktop.org/software/systemd/man/systemd-update-done.service.html
Yes, it looks like a good solution. I will try it. Thanks for the tip.
Quoted 11 lines
I'd prefer to see us move to a mostly immutable filesystem with symlinks that map into a rwfs for specific files / directories we expect to mutate at runtime. I don't think we should do any smart types of writeback caching at the filesystem level as that will just lead to confusion about persistence that is hard to debug. Not to mention tools like rsync don't guarantee atomic snapshotting during the copy for file consistency. Our applications should be designed in such a way that they are always consistent on disk. Ex. writing text files should be done to a temporary, then moved over the old one. That way the update is atomic and we don't have partially written files.
Regards,
Alexander