docs/manual: document filesystems
PATCH 1/1 docs/manual: document filesystems
We do expect that users be able to implement new filesystems and submit them, or have local, custom and private filesystems in their br2-external trees.
So, add documentation for filesystems, based on the format we have for packages.
---
Changes v1 -> v2:
- credit Carlos for prompting this addition to the manual. :-)
- fix the name of the _CMD macro (no trailing 'S')
--- docs/manual/adding-filesystems.txt | 119 ++++++++++++++++++++++++++++++++++ docs/manual/adding-packages-hooks.txt | 7 +- docs/manual/adding-packages.txt | 2 + 3 files changed, 127 insertions(+), 1 deletion(-) create mode 100644 docs/manual/adding-filesystems.txt
docs/manual/adding-filesystems.txt new file +119 -0
@@ -0,0 +1,119 @@ +// -*- mode:doc; -*- +// vim: set syntax=asciidoc: + +=== Infrastructure for filesystems + +As for packages, Buildroot provides an infrastructure to build +filesystems. + +The filesystem infrastructure follows a carefully-crafted sequence: + +1. Assemble an intermediate, internal tarball from the +target/+ + directory: + * create users and groups, + * set the correct rights on files, + * create device nodes, + * create the intermediate tarball; +2. Then for each filesystem: + * extract the intermediate tarball to a filesystem-specific + +$(TARGET_DIR)+, + * execute the filesystem pre-gen hooks (see below), + * optionally, touch everything to a specific date, for reproducibility, + * execute the filesystem commands, + * execute the filesystem post-gen hooks (see below), + * remove the filesystem-specific +$(TARGET_DIR)+; +3. Remove the intermediate tarball. + +[[filesystem-tutorial]] +==== +filesystem+ tutorial + +Let's look at an example filesytem. First and foremost, like for a +package, create a directory named from your filesystem, e.g. +foo+. + +Then, create a +Config.in+ file describing the menuconfig entries +for the filesystem. Be sure to have that +Config.in+ included from ++fs/Config.in+ (or from the +Config.in+ of your *br2-external* tree). +For example: + +---- +01: config BR2_TARGET_ROOTFS_FOO +02: bool "foo root filesystem" +03: help +04: Build a foo root filesystem. +05: +06: if BR2_TARGET_ROOTFS_FOO +07: +08: config BR2_TARGET_ROOTFS_FOO_FORMAT_BAR +09: bool "bar format" +10: help +11: Enable the bar format. +12: +13: endif +---- + +Then, in the same directory, create a +foo.mk+ file describing how +to actually assemble the filesystem image from the content of ++$(TARGET_DIR)+. Be sure that this +.mk+ file is included in the +Makefile code (automatic for built-in filesystems, not for filesystems +in a *br2-external* tree.) For example: + +---- +01: ################################################################################ +02: # +03: # foo filesystem +04: # +05: ################################################################################ +06: +07: ROOTFS_FOO_DEPENDENCIES = host-foo-utils +08: +09: ifeq ($(BR2_TARGET_ROOTFS_FOO_FORMAT_BAR),y) +10: ROOTFS_FOO_OPTS = --format=bar +11: else +12: ROOTFS_FOO_OPTS = --format=foo +13: endif +14: +15: define ROOTFS_FOO_CMD +16: $(HOST_DIR)/bin/mkfs.foo \ +17: --root-dir=$(TARGET_DIR) \ +18: --option-foo --option-bar \ +19: $(ROOTFS_FOO_OPTS) \ +20: --output-file=$(@) +21: endef +22: +23: $(eval $(rootfs)) +---- + +On line 7, we specify the dependencies on packages needed to build the +filesystem. Usually, those dependencies are one or more host-packages +that provide the filesystem generator. + +On lines 9 to 13, we define a conditional set of options, depending on +the user's choice. + +On lines 15 to 21, we define the actual command that will generate the +filesystem image. + +Finally, on line 23, we invoke the +rootfs+ macro that generates all the +Makefile rules that actually allow the filesystem to be built. + +[[filesystem-reference]] +==== +filesystem+ reference + +If one compares a package with a filesystem, one may notice that the +variables for a filesystem start with the prefix +ROOTFS_+, when the +variables for packages have no prefix. + +There is a single macro that a filesystem may set: + +* +ROOTFS_FOO_CMD+ contains the commands to execute to actually + generate the filesystem iamge from the content of the +$(TARGET_DIR)+ + directory. + +Additionally, like for packages, there are a set of hooks; see +xref:hooks[] for details. + +And here is a list of variables that may be used (but may *not* be set): + +* +$(TARGET_DIR)+ is the directory containing the +target/+ directory. + +* +$(@)+ is the filename where to store the filesystem image into.
docs/manual/adding-packages-hooks.txt +6 -1
@@ -11,7 +11,7 @@ Most hooks aren't really useful for generic packages, since the +.mk+ file already has full control over the actions performed in each step of the package construction. -The following hook points are available: +The following hook points are available for packages: * +LIBFOO_PRE_DOWNLOAD_HOOKS+ * +LIBFOO_POST_DOWNLOAD_HOOKS+
@@ -46,6 +46,11 @@ The following hook points are available: * +LIBFOO_PRE_LEGAL_INFO_HOOKS+ * +LIBFOO_POST_LEGAL_INFO_HOOKS+ +The following hook points are available for filesystems: + +* +ROOTFS_FOO_PRE_GEN_HOOKS+ +* +ROOTFS_FOO_POST_GEN_HOOKS+ + These variables are 'lists' of variable names containing actions to be performed at this hook point. This allows several hooks to be registered at a given hook point. Here is an example:
docs/manual/adding-packages.txt +2 -0
@@ -47,6 +47,8 @@ include::adding-packages-asciidoc.txt[] include::adding-packages-linux-kernel-spec-infra.txt[] +include::adding-filesystems.txt[] + include::adding-packages-hooks.txt[] include::adding-packages-gettext.txt[]
Signature · 1 line
2.14.1
PATCH 1/1 docs/manual: document filesystems
Quoted 6 lines
From: "Yann Morin" <[email protected]>
To: "buildroot" <[email protected]>
Cc: "Yann Morin" <[email protected]>, "Thomas De Schampheleire" <[email protected]>, "DATACOM" <[email protected]>, "Thomas Petazzoni" <[email protected]>
Sent: Saturday, June 9, 2018 6:46:47 AM
Subject: [PATCH] docs/manual: document filesystems
[...]
docs/manual/adding-filesystems.txt line 4
Quoted 12 lines
docs/manual/adding-filesystems.txt new file +4
-0@@ -0,0 +1,119 @@ +// -*- mode:doc; -*- +// vim: set syntax=asciidoc: + +=== Infrastructure for filesystems
Shouldn't this be a separate chapter instead of a section of packages? Filesystems are different from packages, regardless the syntactical similarities in the configurations and recipe files. See also the comment below about the hooks section.
Quoted 101 lines
docs/manual/adding-filesystems.txt new file +57
-0+ +As for packages, Buildroot provides an infrastructure to build +filesystems. + +The filesystem infrastructure follows a carefully-crafted sequence: + +1. Assemble an intermediate, internal tarball from the +target/+ + directory: + * create users and groups, + * set the correct rights on files, + * create device nodes, + * create the intermediate tarball; +2. Then for each filesystem: + * extract the intermediate tarball to a filesystem-specific + +$(TARGET_DIR)+, + * execute the filesystem pre-gen hooks (see below), + * optionally, touch everything to a specific date, for reproducibility, + * execute the filesystem commands, + * execute the filesystem post-gen hooks (see below), + * remove the filesystem-specific +$(TARGET_DIR)+; +3. Remove the intermediate tarball. + +[[filesystem-tutorial]] +==== +filesystem+ tutorial + +Let's look at an example filesytem. First and foremost, like for a +package, create a directory named from your filesystem, e.g. +foo+. + +Then, create a +Config.in+ file describing the menuconfig entries +for the filesystem. Be sure to have that +Config.in+ included from ++fs/Config.in+ (or from the +Config.in+ of your *br2-external* tree). +For example: + +---- +01: config BR2_TARGET_ROOTFS_FOO +02: bool "foo root filesystem" +03: help +04: Build a foo root filesystem. +05: +06: if BR2_TARGET_ROOTFS_FOO +07: +08: config BR2_TARGET_ROOTFS_FOO_FORMAT_BAR +09: bool "bar format" +10: help +11: Enable the bar format. +12: +13: endif +---- + +Then, in the same directory, create a +foo.mk+ file describing how +to actually assemble the filesystem image from the content of ++$(TARGET_DIR)+. Be sure that this +.mk+ file is included in the +Makefile code (automatic for built-in filesystems, not for filesystems +in a *br2-external* tree.) For example: + +---- +01:################################################################################
+02: #
+03: # foo filesystem
+04: #
+05:
################################################################################
+06:
+07:ROOTFS_FOO_DEPENDENCIES= host-foo-utils
+08:
+09: ifeq ($(BR2_TARGET_ROOTFS_FOO_FORMAT_BAR),y)
+10:ROOTFS_FOO_OPTS= --format=bar
+11: else
+12:ROOTFS_FOO_OPTS= --format=foo
+13: endif
+14:
+15: defineROOTFS_FOO_CMD
+16: $(HOST_DIR)/bin/mkfs.foo \
+17: --root-dir=$(TARGET_DIR) \
+18: --option-foo --option-bar \
+19: $(ROOTFS_FOO_OPTS) \
+20: --output-file=$(@)
+21: endef
+22:
+23: $(eval $(rootfs))
+----docs/manual/adding-filesystems.txt new file +19
-0+ +On line 7, we specify the dependencies on packages needed to build the +filesystem. Usually, those dependencies are one or more host-packages +that provide the filesystem generator. + +On lines 9 to 13, we define a conditional set of options, depending on +the user's choice. + +On lines 15 to 21, we define the actual command that will generate the +filesystem image. + +Finally, on line 23, we invoke the +rootfs+ macro that generates all the +Makefile rules that actually allow the filesystem to be built. + +[[filesystem-reference]] +==== +filesystem+ reference + +If one compares a package with a filesystem, one may notice that the +variables for a filesystem start with the prefix +ROOTFS_+, when the
docs/manual/adding-filesystems.txt line 113
Quoted 10 lines
docs/manual/adding-filesystems.txt new file +10
-0+variables for packages have no prefix. + +There is a single macro that a filesystem may set: + +* +ROOTFS_FOO_CMD+ contains the commands to execute to actually + generate the filesystem iamge from the content of the +$(TARGET_DIR)+ + directory. + +Additionally, like for packages, there are a set of hooks; see +xref:hooks[] for details.
There are two fundamental differences between packages and filesystem images:
Filesystem images are always rebuilt and installed while packages are built/installed once and rebuilt/re-installed only on explicit calls to "make pkg-name-{reconfigure,rebuild,reinstall}".
Packages are supposed to install, remove or modify files in the global $(TARGET_DIR) hierarchy. Genarating filesystem images do not modify the contents of the global $(TARGET_DIR) and user-provided rootfs extensions must not do this.
[Side note: we could add an optional "paranoid mode" config which would trigger a checksum generation/checking right before/after the creation of the rootfs images.]
Quoted 1 line
+And here is a list of variables that may be used (but may *not* be set):
"must" not be set.
docs/manual/adding-packages-hooks.txt
Quoted 9 lines
docs/manual/adding-filesystems.txt new file +4
-0+ +* +$(TARGET_DIR)+ is the directory containing the +target/+ directory. + +* +$(@)+ is the filename where to store the filesystem image into.docs/manual/adding-packages-hooks.txt +0
-0
Consider moving the hooks section to a separate chapter if you put the filesystem text in a separate chapter too.
[...]
Signature · 3 lines
Carlos Santos (Casantos) - DATACOM, P&D ?Marched towards the enemy, spear upright, armed with the certainty that only the ignorant can have.? ? Epitaph of a volunteer
PATCH 1/1 docs/manual: document filesystems
Carlos, All,
Thanks for the review! :-)
On 2018-06-09 19:20 -0300, Carlos Santos spake thusly:
Quoted 1 line
Quoted 1 line
From: "Yann Morin" <[email protected]>
[--SNIP--]
Quoted 8 lines
Quoted 8 lines
docs/manual/adding-filesystems.txt new file +1
-0@@ -0,0 +1,119 @@ +// -*- mode:doc; -*-
Quoted 8 lines
docs/manual/adding-filesystems.txt line 4
Quoted 3 lines
docs/manual/adding-filesystems.txt new file +3
-0+// vim: set syntax=asciidoc: + +=== Infrastructure for filesystemsShouldn't this be a separate chapter instead of a section of packages? Filesystems are different from packages, regardless the syntactical similarities in the configurations and recipe files. See also the comment below about the hooks section.
I also wondered about that.
But I was a bit wary of creating another chapter just for filesystems.
And notice that in the package chapter, we also talk about the 'asciidoc-document' infra, which is already not a package infra.
I'd like input from others. Thomas, Peter, Arnout?
[--SNIP--]
Quoted 6 lines
There are two fundamental differences between packages and filesystem images:
Filesystem images are always rebuilt and installed while packages are built/installed once and rebuilt/re-installed only on explicit calls to "make pkg-name-{reconfigure,rebuild,reinstall}".
ACK, good point. I'll add this to the first section, that explains the infra.
Quoted 2 lines
Packages are supposed to install, remove or modify files in the global $(TARGET_DIR) hierarchy.
Well, in the first part of this patch, I added a description of the sequence, making it obvious (or so I hoped) that indeed filesystems were acting on a transient copy:
2. Then for each filesystem:
* extract the intermediate tarball to a filesystem-specific $(TARGET_DIR), [...]
* remove the filesystem-specific $(TARGET_DIR);
Quoted 3 lines
Genarating filesystem images do not modify the
contents of the global $(TARGET_DIR) and user-provided rootfs extensions must not do this.
Why do you insist on modifying the global target/ directory from your filesystem? If you do that, and build two or more filesystems [*], they might each try to modify target/ is incompatible ways, and this is not even safe, concurrency-wise (e.g. ony may try to archive the content of /var while another is doing a rm -rf on it.)
The filesystem infra has been changed to provide each filesystem a copy of the target/ directory, so that each filesystem can do their own tweaks without impacting the others, without the need to be idempotent, and allowing filesystems to be run in parallel.
I have still not seen a *compelling* reason to let filesystems muck in the original target/ directory.
[*] Buildroot terminology, like BR2_TARGET_ROOTFS_SQUASHFS=y and BR2_TARGET_ROOTFS_ISO9660=y at the same time.
Quoted 3 lines
[Side note: we could add an optional "paranoid mode" config which would trigger a checksum generation/checking right before/after the creation of the rootfs images.]
Not needed, since there is no reason to let them do.
Quoted 2 lines
docs/manual/adding-filesystems.txt line 115
Quoted 1 line
docs/manual/adding-filesystems.txt new file +1
-0+And here is a list of variables that may be used (but may *not* be set):"must" not be set.
https://www.merriam-webster.com/dictionary/may
4. : SHALL, MUST ? used in law where the sense, purpose, or policy requires this interpretation
(yes, I tend to use a pedant or/and archaic english. ;-])
Quoted 11 lines
docs/manual/adding-packages-hooks.txt
Quoted 8 lines
docs/manual/adding-filesystems.txt new file +3
-0+* +$(TARGET_DIR)+ is the directory containing the +target/+ directory. + +* +$(@)+ is the filename where to store the filesystem image into.docs/manual/adding-packages-hooks.txt +0
-0Consider moving the hooks section to a separate chapter if you put the filesystem text in a separate chapter too.
If others agreee that the filesystem infra requires its own chapter, then I will, yes.
Thanks again for the review! :-)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 docs/manual: document filesystems
Carlos, Arnout, All,
On 2018-06-10 10:12 +0200, Yann E. MORIN spake thusly:
Quoted 1 line
On 2018-06-09 19:20 -0300, Carlos Santos spake thusly:
[--SNIP--]
Quoted 2 lines
Quoted 2 lines
Packages are supposed to install, remove or modify files in the global $(TARGET_DIR) hierarchy.
I forgot to say that this will no longer be true when we (eventually) introduce the top-level parallel build, because, as Arnout already said, each package will install in its own, private copy of TARGET_DIR, not only that, but also its private STAGING_DIR and HOST_DIR as well.
One thing that Arnout suggested, is to (mostly) get rid of TARGET_DIR, in favour of a package-prefix FOO_TARGET_DIR (amd similarly for filesystems).
However I still don't think that is a good idea, because that would allow packages to easily find and write into another package's TARGET_DIR. Using a single TARGET_DIR, instead, will make it harder (but not impossible, agreed).
Ditto filesystems.
So I believe we still want to keep a context-specific TARGET_DIR. It is easier to use, it is more stable, we can update it at will, and it makes it harder for packages and filesystems to do weird stuff.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 docs/manual: document filesystems
Quoted 13 lines · Yann E. MORIN
Carlos, Arnout, All,
On 2018-06-10 10:12 +0200, Yann E. MORIN spake thusly:
Quoted 1 line
On 2018-06-09 19:20 -0300, Carlos Santos spake thusly:
[--SNIP--]
Quoted 2 lines
Quoted 2 lines
Packages are supposed to install, remove or modify files in the global $(TARGET_DIR) hierarchy.
I forgot to say that this will no longer be true when we (eventually) introduce the top-level parallel build, because, as Arnout already said, each package will install in its own, private copy of
TARGET_DIR, not only that, but also its privateSTAGING_DIRandHOST_DIRas well.
Quoted 8 lines
One thing that Arnout suggested, is to (mostly) get rid of
TARGET_DIR, in favour of a package-prefixFOO_TARGET_DIR(amd similarly for filesystems).However I still don't think that is a good idea, because that would allow packages to easily find and write into another package's
TARGET_DIR. Using a singleTARGET_DIR, instead, will make it harder (but not impossible, agreed).
How does the definition of TARGET_DIR have any influence on how fs/foo/foo.mk
uses ROOTFS_BAR_TARGET_DIR?
Quoted 4 lines
Ditto filesystems.
So I believe we still want to keep a context-specific
TARGET_DIR. It is easier to use,
TARGET_DIR is only marginally easier to use than FOO_TARGET_DIR...
Quoted 1 line
it is more stable,
???
You actually may have a point here, but I don't understand it.
Quoted 1 line
we can update it at will
We can also update the definition of FOO_TARGET_DIR at will.
, and it makes it harder for packages and filesystems to do weird stuff.
No it doesn't.
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH 1/1 docs/manual: document filesystems
In reply to Yann E. MORINHello,
Quoted 8 lines · Yann E. MORIN
I also wondered about that.
But I was a bit wary of creating another chapter just for filesystems.
And notice that in the package chapter, we also talk about the 'asciidoc-document' infra, which is already not a package infra.
I'd like input from others. Thomas, Peter, Arnout?
The section about packages is already very long, so I wouldn't mind seeing other sections being created outside of it.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/1 docs/manual: document filesystems
In reply to Yann E. MORINQuoted 11 lines
From: "Yann Morin" <[email protected]>
To: "DATACOM" <[email protected]>
Cc: "buildroot" <[email protected]>, "Thomas De Schampheleire" <[email protected]>, "Thomas Petazzoni" <[email protected]>, "Peter Korsgaard" <[email protected]>, "Arnout Vandecappelle" <[email protected]>
Sent: Sunday, June 10, 2018 5:12:17 AM
Subject: Re: [PATCH] docs/manual: document filesystemsCarlos, All,
Thanks for the review! :-)
[...]
Quoted 9 lines
Quoted 3 lines
Genarating filesystem images do not modify the
contents of the global $(TARGET_DIR) and user-provided rootfs extensions must not do this.Why do you insist on modifying the global target/ directory from your filesystem? If you do that, and build two or more filesystems [*], they might each try to modify target/ is incompatible ways, and this is not even safe, concurrency-wise (e.g. ony may try to archive the content of /var while another is doing a rm -rf on it.)
I never intended to modify the global $(TARGET_DIR).
Signature · 3 lines
Carlos Santos (Casantos) - DATACOM, P&D ?Marched towards the enemy, spear upright, armed with the certainty that only the ignorant can have.? ? Epitaph of a volunteer
PATCH 1/1 docs/manual: document filesystems
Carlos, All,
On 2018-06-10 10:38 -0300, Carlos Santos spake thusly:
Quoted 4 lines
Quoted 4 lines
From: "Yann Morin" <[email protected]>
Quoted 3 lines
Genarating filesystem images do not modify the
contents of the global $(TARGET_DIR) and user-provided rootfs extensions must not do this.
[--SNIP--]
Quoted 7 lines
Quoted 5 lines
Why do you insist on modifying the global target/ directory from your filesystem? If you do that, and build two or more filesystems [*], they might each try to modify target/ is incompatible ways, and this is not even safe, concurrency-wise (e.g. ony may try to archive the content of /var while another is doing a rm -rf on it.)
I never intended to modify the global $(TARGET_DIR).
Sorry, I mis-interpreted your comment above, as "user-provided rootfs [] must not do this" to which I appended in my head "but that is exactly what I want to do in my case."
I'm sorry, I should be more careful in my interpretations in the future.
So, I'll take that as a hint to further improve the section that explains how filesystems are generated, to more carefully describe and explain the sequence.
Thanks again for the feedback! :-)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'