Skip to contentSkip to conversation

Pull request for branch yem/genimages

GIT PULL New Yann E. MORIN 3 participants 10 messages Dec 20, 2013 – Dec 21, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

From: "Yann E. MORIN" <[email protected]>

Hello All!

During the last developpers' day in Edimburgh, we discussed my Buildroot.config project, and especially the 'genimages' infrastructure I've done in there.

I've came up with 'genimages' after I came to the conclusion that the current images generation in Buildroot is limited to generating a single filesystem images, but most device do require a more complex setup.

For example. the Rapsberry Pi needs an SDcard (mmcblk0) formatted as thus:

    mmcblk0
    |- mmcblk0p1  (FAT32)
    |  |- bootcode.bin
    |  |- fixup.dat
    |  |- start.elf
    |  |- config.txt
    |  |- cmdline.txt
    |  `- kernel.bin
    `- mmcblk0p2  (whatever, eg. ext3, btrfs...)
       `- all of /

With the current output of Buildroot, that requires a lot of fiddling with the partitioning of the SDcard (may require root), formatting the filesystems (may require root), extract rootfs.tar (may require root to mount, requires root to extract), copy select files from output/images/rpi-firmware/ to the first partition (may require root to mount).

Each user would either have to manually perform those actions (error prone), or write a small shell script to automate this (not trivial).

Also, some other devices require special, but different formatting as well.

Rather than have every users all suffer in silence (and sometimes not in silence) by re-inventing such a script over and over again, for each device that need such a setup, I came up with the 'genimages' idea: a description of the layout of the storage devices, partitioning schemes, and partitions content.

So, here is the 'genimages' from Buildroot.config, that I updated (in fact, trimmed) to include in Buildroot itself.

The basis is that we bundle some trivial partition description for the boards we have in board/ and reference it in the corresponding defconfigs.

How does 'genimages' work?

First, genimages extract the generated rootfs.tar. Hence, rootfs.tar is always generated if a partition description is configured.

Second, genimages generates one filesystem image for each partition listed in the description. Once the filesystem is generated, its mountpoint is emptied, so it does not appear in the upper filesystem.

Third, genimages aggregates all the partition of a device into a single image file

Finally, those device images can be flashed directly on the raw device of the storage, without manually fiddling with fdisk, mount, tar and so on...

How does 'genimages' work internally?

The partition layout is decribed in a .ini-like file.

genimages is a relatively simple shell script that parses that .ini file. Based on each device/partition description, it calls to 'handlers' (or hooks), for each kind of content we support.

For example, we have two types of partitioning scheme supported for
device: mbr and gpt. Each is implemented in its own shell-fragment. Ditto for partitions: we support different filesystems, ext, squashfs, and vfat; they are each implemented in their own shell-fragment.

Each handler has to define the 'do_image' shell function. That functions should expect two arguments: the base dir of the filesystem, and the filename of the image to generate, and it has access to the configuration from the .ini file.

How to use 'genimages'?

It's all explained in the manual. A pre-rendered version is available for reading, and easy review on-line there:
http://ymorin.is-a-geek.org/download/tmp/buildroot/manual/manual.html

The new section is "3.3.6 Customizing the generated filesystem images" and there is a detailed description of the partition table layout in appendix "12.3. Partition table layout description syntax".

Changes v2 -> v3:
- misc typoes in the manual

Changes v1 -> v2:
- drop rpi-firmware install into /boot (Thomas, Ryan)
- drop move of rpi-firmware in bootloader sub-menu (Thomas)
- add possibility to generate an unmounted filesystem (Thomas, Ryan)
- add ability to specify files to add in a filesystem instead of a sub-dir of target/ (Ryan, during a hacking session)
- properly depend on the host-tools if they are selected
- add GPT support
- add a third filesystem, squashfs (in its own patch, as an example of how easy it is to add one)

Regards,
Yann E. MORIN.

The following changes since commit 2746158497cdaf80a5b85584e3972857ac2e7e82:

util-linux: Add config switches for some more binaries (2013-12-20 22:45:31 +0100)

are available in the git repository at:

git://gitorious.org/buildroot/buildroot.git yem/genimages

for you to fetch changes up to 81062cef5d4d51b6f196fb7c3d9c171c7c362555:

fs/custom: add support for squashfs (2013-12-20 23:12:22 +0100)

----------------------------------------------------------------
Yann E. MORIN (4):
fs/custom: generate complete, partition-based device images board/raspberrypi: provide partition description for the new genimages package/squashfs: add selection for the host variant fs/custom: add support for squashfs

 board/raspberrypi/partitions          |  36 ++++
 configs/raspberrypi_defconfig         |   8 +
 docs/manual/appendix.txt              |   1 +
 docs/manual/customize-filesystems.txt |  35 ++++
 docs/manual/customize.txt             |   2 +
 docs/manual/partition-layout.txt      | 323 ++++++++++++++++++++++++++++++++++
 fs/Config.in                          |   1 +
 fs/custom/Config.in                   |  16 ++
 fs/custom/boot/gpt                    | 126 +++++++++++++
 fs/custom/boot/mbr                    |  57 ++++++
 fs/custom/boot/pre-post               |   8 +
 fs/custom/custom.mk                   |  41 +++++
 fs/custom/fs/ext                      |  22 +++
 fs/custom/fs/pre-post                 |  67 +++++++
 fs/custom/fs/squashfs                 |  15 ++
 fs/custom/fs/vfat                     |  17 ++
 fs/custom/functions                   |  47 +++++
 fs/custom/genimages                   | 242 +++++++++++++++++++++++++
 package/Config.in.host                |   1 +
 package/squashfs/Config.in.host       |   6 +
 20 files changed, 1071 insertions(+)
 create mode 100644 board/raspberrypi/partitions
 create mode 100644 docs/manual/customize-filesystems.txt
 create mode 100644 docs/manual/partition-layout.txt
 create mode 100644 fs/custom/Config.in
 create mode 100644 fs/custom/boot/gpt
 create mode 100644 fs/custom/boot/mbr
 create mode 100644 fs/custom/boot/pre-post
 create mode 100644 fs/custom/custom.mk
 create mode 100644 fs/custom/fs/ext
 create mode 100644 fs/custom/fs/pre-post
 create mode 100644 fs/custom/fs/squashfs
 create mode 100644 fs/custom/fs/vfat
 create mode 100644 fs/custom/functions
 create mode 100755 fs/custom/genimages
 create mode 100644 package/squashfs/Config.in.host
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH 1/4 fs/custom: generate complete, partition-based device images

From: "Yann E. MORIN" <[email protected]>

Contrary to the existing fs/ schemes, which each generate only a single filesystem image for the root filesystem, this new scheme allows the user to generate more complex images.

The basis behind this is a .ini-like description of the layout of the final target storage:
- the list of device(s)
- per-device, the list of partition(s)
- per-partition, the content

It is possible to create MBR- or GPT-based partitoining schemes. Adding new ones should be relatively easy (but would need adequate host tools).

For now, the only content possible for partitions is a filesystem. It should be pretty easy to add new types (eg. un-formated, or raw blob).

Also, only two filesystems are supported: ext{2,3,4} and vfat. Adding more will be relatively easy, provided we have the necessary host packages to deal with those filesystems.

The existing Buildroot filesystem generators are re-used as much as possible when it makes sense; when it does not (eg. for vfat), a specific generator is used.

Cc Arnout Vandecappelle <[email protected]>
Cc Ryan Barnett <[email protected]>
---
 docs/manual/appendix.txt              |   1 +
 docs/manual/customize-filesystems.txt |  35 ++++
 docs/manual/customize.txt             |   2 +
 docs/manual/partition-layout.txt      | 314 ++++++++++++++++++++++++++++++++++
 fs/Config.in                          |   1 +
 fs/custom/Config.in                   |  16 ++
 fs/custom/boot/gpt                    | 126 ++++++++++++++
 fs/custom/boot/mbr                    |  57 ++++++
 fs/custom/boot/pre-post               |   8 +
 fs/custom/custom.mk                   |  38 ++++
 fs/custom/fs/ext                      |  22 +++
 fs/custom/fs/pre-post                 |  67 ++++++++
 fs/custom/fs/vfat                     |  17 ++
 fs/custom/functions                   |  47 +++++
 fs/custom/genimages                   | 242 ++++++++++++++++++++++++++
 15 files changed, 993 insertions(+)
 create mode 100644 docs/manual/customize-filesystems.txt
 create mode 100644 docs/manual/partition-layout.txt
 create mode 100644 fs/custom/Config.in
 create mode 100644 fs/custom/boot/gpt
 create mode 100644 fs/custom/boot/mbr
 create mode 100644 fs/custom/boot/pre-post
 create mode 100644 fs/custom/custom.mk
 create mode 100644 fs/custom/fs/ext
 create mode 100644 fs/custom/fs/pre-post
 create mode 100644 fs/custom/fs/vfat
 create mode 100644 fs/custom/functions
 create mode 100755 fs/custom/genimages
docs/manual/appendix.txt +1 -0
index 74ee8fd..53f4205 100644
@@ -6,6 +6,7 @@ Appendix
 
 include::makedev-syntax.txt[]
 include::makeusers-syntax.txt[]
+include::partition-layout.txt[]
 
 
 // Automatically generated lists:
docs/manual/customize-filesystems.txt new file +35 -0
new file mode 100644
index 0000000..cddee42
@@ -0,0 +1,35 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[filesystem-custom]]
+Customizing the generated filesystem images
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
++Buildroot+ knows by default how to generate a few different kind of
+filesystems, such as +squashfs+, +ext2/3/4+, +cramfs+... But those
+filesystems are all generated to contain the complete target directory
+hierarchy in a single filesystem, mounted as the root filesystem +/+.
+That is, even if you select both an +ext2+ and a +squashfs+ filesystems,
+the content of the two generated images will be the exact same, only the
+types of the filesystems will be different.
+
+Most devices require a more complex setup, with different parts of the
+directory structure split across different filesystems, each stored on
+different partitions of one or more storage devices.
+
++Buildroot+ can generate such complex setups, using a +partition table layout
+description+. This is a simple text file, not unlike the +.ini+ style of
+configuration files, that describes how the target directory hierarchy has
+to be split across the target storage devices. It is a bit like a flattened
+tree of the storage layout.
+
+Set the variable +BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE+ to the path of
+the file containing your +partition table layout description+.
+
+See xref:part-layout-desc-syntax[] for the complete documentation of the
++partition table layout description+ syntax.
+
+[underline]*Note:* Although more versatile than the single filesystem image
+mechanism, the +partition table layout description+ might be unable to
+describe very complex setups. For example, it is not capable of handling
++initramfs+ based systems, or NFS-mounted filesystems.
docs/manual/customize.txt +2 -0
index 7e46fd8..6d062ea 100644
@@ -14,6 +14,8 @@ include::customize-kernel-config.txt[]
 
 include::customize-toolchain.txt[]
 
+include::customize-filesystems.txt[]
+
 include::customize-store.txt[]
 
 include::customize-packages.txt[]
docs/manual/partition-layout.txt new file +314 -0

314 lines not on this page · raw message

fs/Config.in +1 -0
index da4c5ff..44e04f7 100644
@@ -3,6 +3,7 @@ menu "Filesystem images"
 source "fs/cloop/Config.in"
 source "fs/cpio/Config.in"
 source "fs/cramfs/Config.in"
+source "fs/custom/Config.in"
 source "fs/ext2/Config.in"
 source "fs/initramfs/Config.in"
 source "fs/iso9660/Config.in"
fs/custom/Config.in new file +16 -0
new file mode 100644
index 0000000..e5a8ee7
@@ -0,0 +1,16 @@
+config BR2_TARGET_ROOTFS_CUSTOM
+	bool "Custom partition table layout"
+	select BR2_TARGET_ROOTFS_TAR
+
+config BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE
+	string "path to the custom partition table layout description"
+	depends on BR2_TARGET_ROOTFS_CUSTOM
+	help
+	  Enter the path to a partition-table for your device.
+
+	  This will allow Buildroot to generate a more complex target
+	  image, which may consist of more than one filesystem on more
+	  than one partition.
+
+	  See docs/manual/bla-bla on how to construct such a partition
+	  table.
fs/custom/boot/gpt new file +126 -0

126 lines not on this page · raw message

fs/custom/boot/mbr new file +57 -0

57 lines not on this page · raw message

fs/custom/boot/pre-post new file +8 -0
new file mode 100644
index 0000000..507d17e
@@ -0,0 +1,8 @@
+do_image_pre() {
+    :
+}
+do_image_post() {
+    :
+}
+
+#vim: set ft=sh
fs/custom/custom.mk new file +38 -0
new file mode 100644
index 0000000..63c1f23
@@ -0,0 +1,38 @@
+################################################################################
+#
+# custom partitioning
+#
+################################################################################
+
+define ROOTFS_CUSTOM_CMD
+	BUILD_DIR=$(BUILD_DIR) fs/custom/genimages \
+		'$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'
+endef
+
+# rootfs-custom uses rootfs.tar as the source to generate
+# the resulting image(s), so we need to build it first.
+ROOTFS_CUSTOM_DEPENDENCIES += rootfs-tar
+
+# All of the following filesystem generators, or partition managers, are
+# optional, but if they are selected, we may need them, so we need to
+# depend on them
+ifeq ($(BR2_PACKAGE_HOST_DOSFSTOOLS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-dosfstools
+endif
+ifeq ($(BR2_PACKAGE_HOST_E2FSPROGS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-e2fsprogs
+endif
+ifeq ($(BR2_PACKAGE_HOST_GENEXT2FS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-genext2fs
+endif
+ifeq ($(BR2_PACKAGE_HOST_GENPART),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-genpart
+endif
+ifeq ($(BR2_PACKAGE_HOST_MTOOLS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-mtools
+endif
+ifeq ($(BR2_PACKAGE_HOST_PARTED),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-parted
+endif
+
+$(eval $(call ROOTFS_TARGET,custom))
fs/custom/fs/ext new file +22 -0
new file mode 100644
index 0000000..5f9b4e7
@@ -0,0 +1,22 @@
+# Create an extended file system
+
+do_image() {
+    local root_dir="${1}"
+    local img="${2}"
+    local -a fs_opts
+    local gen rev
+
+    fs_opts+=( -z )
+    fs_opts+=( -d "${root_dir}" )
+    [ -z "${size}"    ] || fs_opts+=( -b $((size/1024)) )
+    [ -n "${ext_gen}" ] || ext_gen=2
+    [ -n "${ext_rev}" ] || ext_rev=1
+
+    # Remember, we're running from Buildroot's TOP_DIR
+    GEN=${ext_gen} REV=${ext_rev}                   \
+    ./fs/ext2/genext2fs.sh "${fs_opts[@]}" "${img}" >/dev/null
+
+    [ -z "${fs_label}" ] || tune2fs -L "${fs_label}" "${img}" >/dev/null
+}
+
+# vim: ft=sh
fs/custom/fs/pre-post new file +67 -0

67 lines not on this page · raw message

fs/custom/fs/vfat new file +17 -0
new file mode 100644
index 0000000..aa5545f
@@ -0,0 +1,17 @@
+# Create a VFAT file system
+
+do_image() {
+    local root_dir="${1}"
+    local img="${2}"
+    local -a fs_opts
+
+    dd if=/dev/zero of="${img}" bs=${size} count=0 seek=1 2>/dev/null
+
+    [ -z "${vfat_size}" ] || fs_opts+=( -F ${vfat_size} )
+    [ -z "${fs_label}"  ] || fs_opts+=( -n "${fs_label}" )
+    mkfs.vfat "${fs_opts[@]}" "${img}" >/dev/null
+
+    mcopy -i "${img}" "${root_dir}/"* '::'
+}
+
+# vim: ft=sh
fs/custom/functions new file +47 -0

47 lines not on this page · raw message

fs/custom/genimages new file +242 -0

242 lines not on this page · raw message

Signature · 1 line
1.8.1.2
#2 lore raw

PATCH 1/4 fs/custom: generate complete, partition-based device images

Dear Yann E. MORIN,

Quoted 5 lines · Yann E. MORIN
docs/manual/partition-layout.txt new file +5 -0
+Requirements on host packages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Depending on what types of partitioning and filesystems are used by your
++partition table layout description+, you may have to enable some

docs/manual/partition-layout.txt line 214

Quoted 10 lines
docs/manual/partition-layout.txt new file +10 -0
+host-packages in the +Host utilities+ sub-menu.
+
+Since the content of a +partition table layout description+ is very
+specific to a board and/or a project, there is no way for Buildroot to
+automatically select those host packages, and thus it is your
+responsibility to select the appropriate ones.
+
+For example, for an MBR partitioning, you will have to enable the +host
+genpart+ package. For FAT filesystems, you will have to enable both of
++host dosfstools+ and +host mtools+.

I'll have other comments about the genimages stuff later on, but I got an idea this night about this specific problem. I don't really like the fact that the user has to manually enable in menuconfig the relevant host tools, depending on his genimage .ini configuration file.

What about doing something like:

ROOTFS_CUSTOM_DEPENDENCIES = $(shell fs/custom/genimages --show-depends \
				'$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'

The --show-depends option of genimages would, instead of doing the normal genimages work, simply parse the .ini file, and return a space separated list of dependencies needed by the ROOTFS_CUSTOM stuff. Not only this would make the dependencies automatic for the user, but also it would allow you to remove the following piece of code from the custom.mk file:

+# All of the following filesystem generators, or partition managers, are +# optional, but if they are selected, we may need them, so we need to +# depend on them
+ifeq ($(BR2_PACKAGE_HOST_DOSFSTOOLS),y) +ROOTFS_CUSTOM_DEPENDENCIES += host-dosfstools
+endif
+ifeq ($(BR2_PACKAGE_HOST_E2FSPROGS),y) +ROOTFS_CUSTOM_DEPENDENCIES += host-e2fsprogs
+endif
+ifeq ($(BR2_PACKAGE_HOST_GENEXT2FS),y) +ROOTFS_CUSTOM_DEPENDENCIES += host-genext2fs
+endif
+ifeq ($(BR2_PACKAGE_HOST_GENPART),y) +ROOTFS_CUSTOM_DEPENDENCIES += host-genpart
+endif
+ifeq ($(BR2_PACKAGE_HOST_MTOOLS),y) +ROOTFS_CUSTOM_DEPENDENCIES += host-mtools
+endif
+ifeq ($(BR2_PACKAGE_HOST_PARTED),y) +ROOTFS_CUSTOM_DEPENDENCIES += host-parted
+endif

What do you think?

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#3 lore raw

PATCH 1/4 fs/custom: generate complete, partition-based device images

Quoted 31 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Thomas" == Thomas Petazzoni <[email protected]> writes:

Dear Yann E. MORIN,

Quoted 5 lines · Yann E. MORIN
docs/manual/partition-layout.txt new file +5 -0
+Requirements on host packages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Depending on what types of partitioning and filesystems are used by your
++partition table layout description+, you may have to enable some

docs/manual/partition-layout.txt line 214

Quoted 10 lines
docs/manual/partition-layout.txt new file +10 -0
+host-packages in the +Host utilities+ sub-menu.
+
+Since the content of a +partition table layout description+ is very
+specific to a board and/or a project, there is no way for Buildroot to
+automatically select those host packages, and thus it is your
+responsibility to select the appropriate ones.
+
+For example, for an MBR partitioning, you will have to enable the +host
+genpart+ package. For FAT filesystems, you will have to enable both of
++host dosfstools+ and +host mtools+.

I'll have other comments about the genimages stuff later on, but I got an idea this night about this specific problem. I don't really like the fact that the user has to manually enable in menuconfig the relevant host tools, depending on his genimage .ini configuration file.

What about doing something like:

ROOTFS_CUSTOM_DEPENDENCIES = $(shell fs/custom/genimages --show-depends \
				'$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'
Quoted 6 lines

The --show-depends option of genimages would, instead of doing the normal genimages work, simply parse the .ini file, and return a space separated list of dependencies needed by the ROOTFS_CUSTOM stuff. Not only this would make the dependencies automatic for the user, but also it would allow you to remove the following piece of code from the custom.mk file:

Sounds good to me. They won't actumatically get selected in kconfig though, but that might not be such a big deal (do they still even need to be visible in kconfig?)

Signature · 1 line
Bye, Peter Korsgaard
#4 lore raw

PATCH 1/4 fs/custom: generate complete, partition-based device images

Thomas, Peter, All,

On 2013-12-21 17:40 +0100, Peter Korsgaard spake thusly:

Quoted 31 lines
Quoted 30 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Thomas" == Thomas Petazzoni <[email protected]> writes:

Dear Yann E. MORIN,

Quoted 5 lines · Yann E. MORIN
docs/manual/partition-layout.txt new file +5 -0
+Requirements on host packages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Depending on what types of partitioning and filesystems are used by your
++partition table layout description+, you may have to enable some

docs/manual/partition-layout.txt line 214

Quoted 10 lines
docs/manual/partition-layout.txt new file +10 -0
+host-packages in the +Host utilities+ sub-menu.
+
+Since the content of a +partition table layout description+ is very
+specific to a board and/or a project, there is no way for Buildroot to
+automatically select those host packages, and thus it is your
+responsibility to select the appropriate ones.
+
+For example, for an MBR partitioning, you will have to enable the +host
+genpart+ package. For FAT filesystems, you will have to enable both of
++host dosfstools+ and +host mtools+.

I'll have other comments about the genimages stuff later on, but I got an idea this night about this specific problem. I don't really like the fact that the user has to manually enable in menuconfig the relevant host tools, depending on his genimage .ini configuration file.

What about doing something like:

ROOTFS_CUSTOM_DEPENDENCIES = $(shell fs/custom/genimages --show-depends \
				'$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'
Quoted 8 lines
Quoted 6 lines

The --show-depends option of genimages would, instead of doing the normal genimages work, simply parse the .ini file, and return a space separated list of dependencies needed by the ROOTFS_CUSTOM stuff. Not only this would make the dependencies automatic for the user, but also it would allow you to remove the following piece of code from the custom.mk file:

Sounds good to me.

Yep, I've implemented that now, and it seems to work pretty well.

Quoted 3 lines

They won't actumatically get selected in kconfig
though, but that might not be such a big deal (do they still even need to be visible in kconfig?)

Yes, in case someone wants to generate filesystem images with a post-build script, and not with genimages.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#5 lore raw

PATCH 1/4 fs/custom: generate complete, partition-based device images

In reply to Peter Korsgaard

Dear Peter Korsgaard,

Quoted 10 lines · Peter Korsgaard
Quoted 6 lines

The --show-depends option of genimages would, instead of doing the normal genimages work, simply parse the .ini file, and return a space separated list of dependencies needed by the ROOTFS_CUSTOM stuff. Not only this would make the dependencies automatic for the user, but also it would allow you to remove the following piece of code from the custom.mk file:

Sounds good to me. They won't actumatically get selected in kconfig though, but that might not be such a big deal (do they still even need to be visible in kconfig?)

I believe we have existing occurrences of host packages being listed in some <pkg>_DEPENDENCIES, but not necessarily selected at the Kconfig level. While we have enforced this not to happen for target packages, we've never enforced that for host packages.

The rationale is that host packages should normally be completely transparent to the user, and therefore not visible in Kconfig. The host package sub-menu is here only to make it easier for the user to enable *some* of the host packages that are actually relevant for the user (and not merely build dependencies of other packages).

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#6 lore raw

PATCH 1/4 fs/custom: generate complete, partition-based device images

In reply to Thomas Petazzoni

Thomas, All,

On 2013-12-21 09:49 +0100, Thomas Petazzoni spake thusly:

Quoted 11 lines · Yann E. MORIN
Quoted 10 lines
docs/manual/partition-layout.txt new file +9 -0
+Requirements on host packages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Depending on what types of partitioning and filesystems are used by your
++partition table layout description+, you may have to enable some
+host-packages in the +Host utilities+ sub-menu.
+
+Since the content of a +partition table layout description+ is very
+specific to a board and/or a project, there is no way for Buildroot to
Quoted 8 lines

docs/manual/partition-layout.txt line 214

Quoted 6 lines
docs/manual/partition-layout.txt new file +6 -0
+automatically select those host packages, and thus it is your
+responsibility to select the appropriate ones.
+
+For example, for an MBR partitioning, you will have to enable the +host
+genpart+ package. For FAT filesystems, you will have to enable both of
++host dosfstools+ and +host mtools+.

I'll have other comments about the genimages stuff later on,

Hey! You can't let us all waiting like that! Speak up, now! ;-)

Quoted 1 line

but I got an idea this night

I don't know what you usually do at night, but I, for one, sleep. ;-)

Quoted 8 lines

about this specific problem. I don't really like the
fact that the user has to manually enable in menuconfig the relevant host tools, depending on his genimage .ini configuration file.

What about doing something like:

ROOTFS_CUSTOM_DEPENDENCIES = $(shell fs/custom/genimages --show-depends \
				'$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'

[--SNIP--]

Quoted 1 line

What do you think?

I was initially thinking about something like it, but found it a bit dubious. But since you suggested it (and now Peter concurred), I have now (re-)implemented it, and it works nicely.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#7 lore raw

PATCH 2/4 board/raspberrypi: provide partition description for the new genimages

In reply to Yann E. MORIN

From: "Yann E. MORIN" <[email protected]>

Now we can generate a complete target storage image with the genimages infra, add a partition table layout description for the Raspberry Pi as an example for how to use genimages.

---
 board/raspberrypi/partitions  | 36 ++++++++++++++++++++++++++++++++++++
 configs/raspberrypi_defconfig |  8 ++++++++
 2 files changed, 44 insertions(+)
 create mode 100644 board/raspberrypi/partitions
board/raspberrypi/partitions new file +36 -0
new file mode 100644
index 0000000..c5e8de5
@@ -0,0 +1,36 @@
+[global]
+extract=tar
+devices=mmcblk0
+
+[mmcblk0]
+type=boot
+boot_type=mbr
+partitions=boot,root
+partalign=$((1048576))
+
+[root]
+type=fs
+size=$((32*1048576))
+mbr_type=$((0x83))
+fs_type=ext
+fs_vfstype=ext4
+fs_mntops=discard
+fs_root=/
+fs_label=ROOT
+ext_gen=4
+ext_rev=1
+
+[boot]
+type=fs
+# FAT32 is not possible under 9MiB
+size=$((9*1048576))
+mbr_type=$((0xc))
+fs_type=vfat
+fs_files_0=rpi-firmware/bootcode.bin
+fs_files_1=rpi-firmware/start.elf
+fs_files_2=rpi-firmware/fixup.dat
+fs_files_3=rpi-firmware/config.txt
+fs_files_4=rpi-firmware/cmdline.txt
+fs_files_5=zImage
+fs_label=BOOT
+vfat_size=16
configs/raspberrypi_defconfig +8 -0
index 4df0f4b..c571817 100644
@@ -21,3 +21,11 @@ BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION="5ebf1db80e2c03e92d570605b53f91a06112574f"
 BR2_LINUX_KERNEL_USE_DEFCONFIG=y
 BR2_LINUX_KERNEL_DEFCONFIG="bcmrpi_quick"
 BR2_LINUX_KERNEL_ZIMAGE=y
+
+BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE="$(TOPDIR)/board/raspberrypi/partitions"
+
+BR2_PACKAGE_HOST_DOSFSTOOLS=y
+BR2_PACKAGE_HOST_E2FSPROGS=y
+BR2_PACKAGE_HOST_GENEXT2FS=y
+BR2_PACKAGE_HOST_GENPART=y
+BR2_PACKAGE_HOST_MTOOLS=y
Signature · 1 line
1.8.1.2
#8 lore raw

PATCH 3/4 package/squashfs: add selection for the host variant

In reply to Yann E. MORIN

From: "Yann E. MORIN" <[email protected]>

If any of the post-image scripts, or genimages, wants to handle squashfs filesystems, we need to expose an option for squashfs-tools to be user-selectable.

---
 package/Config.in.host          | 1 +
 package/squashfs/Config.in.host | 6 ++++++
 2 files changed, 7 insertions(+)
 create mode 100644 package/squashfs/Config.in.host
package/Config.in.host +1 -0
index 34e84bf..71597bf 100644
@@ -12,6 +12,7 @@ source "package/omap-u-boot-utils/Config.in.host"
 source "package/openocd/Config.in.host"
 source "package/parted/Config.in.host"
 source "package/sam-ba/Config.in.host"
+source "package/squashfs/Config.in.host"
 source "package/sunxi-tools/Config.in.host"
 source "package/uboot-tools/Config.in.host"
 
package/squashfs/Config.in.host new file +6 -0
new file mode 100644
index 0000000..b353374
@@ -0,0 +1,6 @@
+config BR2_PACKAGE_HOST_SQUASHFS
+	bool "host squashfs"
+	help
+	  Tools to generate SquashFS filesystems.
+
+	  http://squashfs.sourceforge.net/
Signature · 1 line
1.8.1.2
#9 lore raw

PATCH 4/4 fs/custom: add support for squashfs

In reply to Yann E. MORIN

From: "Yann E. MORIN" <[email protected]>

---
 docs/manual/partition-layout.txt | 11 ++++++++++-
 fs/custom/custom.mk              |  3 +++
 fs/custom/fs/squashfs            | 15 +++++++++++++++
 3 files changed, 28 insertions(+), 1 deletion(-)
 create mode 100644 fs/custom/fs/squashfs
docs/manual/partition-layout.txt +10 -1
index 553f862..7b64a45 100644
@@ -133,7 +133,8 @@ Properties for +type=fs+
 
 * +fs_type+ (mandatory): the type of filesystem to generate
 ** +ext+: generate an extended filesystem (ext2, ext3, ext4)
-** +vfat+: generate a VFAT filesystem (FAT16, FAT32)
+** +squashfs+: generate a squashfs (version 4) filesystem
+** +vfat+: generate a VFAT filesystem (FAT12, FAT16 or FAT32)
 
 * +fs_label+ (optional): the label to assign to this filesystem, if that
   filesystem supports a label
@@ -177,6 +178,14 @@ Properties for +fs_type=ext+
 ** +1+ (mandatory for ext3 or ext4): generate a revision 1 extended
    filesystem
 
+Properties for +fs_type=squashfs+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +squashfs_comp+ (mandatory): the compression type to use
+** +gzip+, +xz+, +lzo+: gzip, xz or lzo compression
+
+* +squashfs_block+ (optional): the size of blocks, in bytes
+
 Properties for +fs_type=vfat+
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
fs/custom/custom.mk +3 -0
index 63c1f23..158e886 100644
@@ -34,5 +34,8 @@ endif
 ifeq ($(BR2_PACKAGE_HOST_PARTED),y)
 ROOTFS_CUSTOM_DEPENDENCIES += host-parted
 endif
+ifeq ($(BR2_PACKAGE_HOST_SQUASHFS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-squashfs
+endif
 
 $(eval $(call ROOTFS_TARGET,custom))
fs/custom/fs/squashfs new file +15 -0
new file mode 100644
index 0000000..ba3646e
@@ -0,0 +1,15 @@
+# Create a squashfs filesystem
+
+do_image() {
+    local root_dir="${1}"
+    local img="${2}"
+    local -a fs_opts
+
+    fs_opts+=( -no-progress -noappend )
+    [ -z "${squashfs_comp}"  ] || fs_opts+=( -comp "${squashfs_comp}" )
+    [ -z "${squashfs_block}" ] || fs_opts+=( -b ${squahsfs_block} )
+
+    mksquashfs "${root_dir}" "${img}" "${fs_opts[@]}" >/dev/null
+}
+
+# vim: ft=sh
Signature · 1 line
1.8.1.2