SUBJECT

From: "Yann E. MORIN" <[email protected]>

If any of the post-image scripts wants to handle squashfs filesystems, we need to expose an option for squashfs-tools to be user-selectable.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Ryan Barnett <[email protected]>

---
 package/Config.in.host          | 1 +
 package/squashfs/Config.in.host | 6 ++++++
 2 files changed, 7 insertions(+)
 create mode 100644 package/squashfs/Config.in.host

package/Config.in.host +1 -0

index 34e84bf..71597bf 100644
@@ -12,6 +12,7 @@ source "package/omap-u-boot-utils/Config.in.host"
 source "package/openocd/Config.in.host"
 source "package/parted/Config.in.host"
 source "package/sam-ba/Config.in.host"
+source "package/squashfs/Config.in.host"
 source "package/sunxi-tools/Config.in.host"
 source "package/uboot-tools/Config.in.host"
 

package/squashfs/Config.in.host new file +0 -0

new file mode 100644
index 0000000..b353374

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

From: "Yann E. MORIN" <[email protected]>
If any of the post-image scripts wants to handle squashfs filesystems, we need to expose an option for squashfs-tools to be user-selectable.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Ryan Barnett <[email protected]>

From: "Yann E. MORIN" <[email protected]>

Contrary to the existing fs/ schemes, which each generate only a single filesystem image for the root filesystem, this new scheme allows the user to generate more complex images.

The basis behind this is a .ini-like description of the layout of the final target storage:
- the list of device(s)
- per-device, the list of partition(s)
- per-partition, the content

It is possible to create MBR- or GPT-based partitoining schemes. Adding new ones should be relatively easy (but would need adequate host tools).

For now, the only content possible for partitions is a filesystem. It should be pretty easy to add new types (eg. un-formated, or raw blob).

Also, only two filesystems are supported: ext{2,3,4} and vfat. Adding more will be relatively easy, provided we have the necessary host packages to deal with those filesystems.

The existing Buildroot filesystem generators are re-used as much as possible when it makes sense; when it does not (eg. for vfat), a specific generator is used.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Arnout Vandecappelle <[email protected]>

Cc Ryan Barnett <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc "J?r?me Oufella" <[email protected]>

---
 docs/manual/appendix.txt              |   1 +
 docs/manual/customize-filesystems.txt |  36 ++++
 docs/manual/customize.txt             |   2 +
 docs/manual/partition-layout.txt      | 298 +++++++++++++++++++++++++++++
 fs/Config.in                          |   1 +
 fs/custom/Config.in                   |  16 ++
 fs/custom/boot/gpt                    | 129 +++++++++++++
 fs/custom/boot/mbr                    |  64 +++++++
 fs/custom/boot/pre-post               |  14 ++
 fs/custom/custom.mk                   |  25 +++
 fs/custom/fs/ext                      |  26 +++
 fs/custom/fs/pre-post                 |  72 +++++++
 fs/custom/fs/vfat                     |  21 +++
 fs/custom/genimages                   | 342 ++++++++++++++++++++++++++++++++++
 14 files changed, 1047 insertions(+)
 create mode 100644 docs/manual/customize-filesystems.txt
 create mode 100644 docs/manual/partition-layout.txt
 create mode 100644 fs/custom/Config.in
 create mode 100644 fs/custom/boot/gpt
 create mode 100644 fs/custom/boot/mbr
 create mode 100644 fs/custom/boot/pre-post
 create mode 100644 fs/custom/custom.mk
 create mode 100644 fs/custom/fs/ext
 create mode 100644 fs/custom/fs/pre-post
 create mode 100644 fs/custom/fs/vfat
 create mode 100755 fs/custom/genimages

docs/manual/appendix.txt +1 -0

index 74ee8fd..53f4205 100644
@@ -6,6 +6,7 @@ Appendix
 
 include::makedev-syntax.txt[]
 include::makeusers-syntax.txt[]
+include::partition-layout.txt[]
 
 
 // Automatically generated lists:

docs/manual/customize-filesystems.txt new file +36 -0

new file mode 100644
index 0000000..fd65c97
@@ -0,0 +1,36 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[filesystem-custom]]
+Customizing the generated filesystem images
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
++Buildroot+ knows by default how to generate a few different kind of
+filesystems, such as +squashfs+, +ext2/3/4+, +cramfs+... But those
+filesystems are all generated to contain the complete target directory
+hierarchy in a single filesystem, mounted as the root filesystem +/+.
+That is, even if you select both an +ext2+ and a +squashfs+ filesystems,
+the content of the two generated images will be the exact same, only the
+types of the filesystems will be different.
+
+Most devices require a more complex setup, with different parts of the
+directory structure split across different filesystems, each stored on
+different partitions of one or more storage devices.
+
++Buildroot+ can generate such complex setups, using a +partition table layout
+description+. This is a simple text file, not unlike the +.ini+ style of
+configuration files, that describes how the target directory hierarchy has
+to be split across the target storage devices. It is a bit like a flattened
+tree of the storage layout.
+
+Set the variable +BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE+ to the path of
+the file containing your +partition table layout description+.
+
+See xref:part-layout-desc-syntax[] for the complete documentation of the
++partition table layout description+ syntax.
+
+[underline]*Note:* Although more versatile than the single filesystem image
+mechanism, the +partition table layout description+ might be unable to
+describe very complex setups. For example, it is not capable of handling
+NFS-mounted filesystems, and +initramfs+ setups is not trivial (although
+possible in most cases).

docs/manual/customize.txt +2 -0

index 7e46fd8..6d062ea 100644
@@ -14,6 +14,8 @@ include::customize-kernel-config.txt[]
 
 include::customize-toolchain.txt[]
 
+include::customize-filesystems.txt[]
+
 include::customize-store.txt[]
 
 include::customize-packages.txt[]

docs/manual/partition-layout.txt new file +288 -0

new file mode 100644
index 0000000..092ae1b
@@ -0,0 +1,298 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[part-layout-desc-syntax]]
+
+Partition table layout description syntax
+-----------------------------------------
+
+The +partition table layout description+ syntax is not unlike the standard
+https://en.wikipedia.org/wiki/.ini[+.ini+] syntax. There are two types of
+entries: +sections+, that may each contain zero or more +properties+.
+
++Sections+ are specified between square brackets +[]+, _eg._: +[name]+.
+
++Properties+ are specified as key-value pairs, _eg._: +key=value+, and
+are documented as:
+
+* +key-name+ (optional or mandatory): description
+** +value1+: description
+** +value2+: description
+** ...
+
+[underline]*Note:* Unlike the standard +.ini+ syntax, the +partition table
+layout description+ _is_ case-sensitive.
+
+The order of +sections+ is irrelevant. However, for readability, we recomend
+the +partition table layout description+ starts with the +global+ section.
+
+The global section
+~~~~~~~~~~~~~~~~~~
+
+The +[global]+ section defines some global settings, and the list of devices.
+
+Properties for the global section
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +extract+ (mandatory): the type of base image to extract
+** +tar+: extract the +rootfs.tar+ base image generated by +Buildroot+
+
+* +devices+ (mandatory): the comma-separated list of storage devices to
+  use on the device. Each device is the filename of the device node
+  present in +/dev+
+
+* +keep_partitions+ (optional): also copy the individual partition images
+  of all devices to +$(BINARIES_DIR)+. Settings from the device sections
+  or the partition sections take precedence over this one.
+** +yes+: copy the individual partition images
+** +no+ (the default): do not copy individual partition images
+
+The devices and partitions sections
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The devices and partitions sections define, for each device or partition,
+the content of that device or partition.
+
+For each device listed in the +[global]+ section, there must be a
+corresponding section named after that device.
+
+For each partition listed in a device section, there must be a corresponding
+section named after that partition.
+
+Properties for the device section
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +type+ (mandatory): the type of content for that device or partition
+** +boot+: the device contains one or more partitions, and
+   _may_ serve as a boot device
+
+* +keep_partitions+ (optional): also copy the individual partition images
+  for this device to +$(BINARIES_DIR)+. Settings from the partition
+  sections take precedence over this one.
+** +yes+: copy the individual partition images
+** +no+ (the default): do not copy individual partition images
+
+Properties for the partition section
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +type+ (mandatory): the type of content for that device or partition
+** +fs+: the partition contains a filesystem
+
+* +size+: the size of that partition, in bytes
+
+* +keep+ (optional): copy this partition image to +$(BINARIES_DIR)+
+** +yes+: copy this partition image
+** +no+ (the default): do not copy this partition image
+
+Properties for +type=boot+
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +boot_type+ (mandatory): the partitioning scheme to use on this device
+** +mbr+: use an https://en.wikipedia.org/wiki/Master_boot_record[MBR]
+   partitioning scheme
+** +gpt+: use a https://en.wikipedia.org/wiki/GUID_Partition_Table[GPT]
+   partitioning scheme
+
+* +partitions+ (mandatory): the comma-separated list of partition(s) on
+  this device; no two partitions may have the same name, even if they
+  reside on different devices; partitions names shall match this regexp:
+  `^[[:alpha:]][[:alnum:]-_]*$` (_ie._ starts with a letter, followed by
+  zero or more alpha-numeric character or a dash or an underscore)
+
+* +partalign+ (optional): the alignment of partitions, in bytes; defaults
+  to an alignment of one, which means no alignment; depending on the
+  +boot_type+, some restrictions may apply, and are documented for each
+  +boot_type+
+
+Properties for +boot_type=mbr+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +mbr_bootcode+ (optional): the bootcode to use, as a path to the file
+  containing the bootcode image, relative to the +$(BINARIES_DIR)+
+  directory; defaults to no bootcode (eg. filled with zeroes)
+
+* +partalign+: must be a multiple of 512
+
+Properties for +boot_type=gpt+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +partalign+: must be a multiple of 512
+
+********
+Currently, only 512-byte sectors are supported. 4k sectors are not.
+********
+
+Properties for partitions whose containing device is +boot_type=mbr+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +mbr_type+ (mandatory): the partition
+  https://en.wikipedia.org/wiki/Partition_type#List_of_partition_IDs[type]
+
+Properties for +type=fs+
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +fs_type+ (mandatory): the type of filesystem to generate
+** +ext+: generate an extended filesystem (ext2, ext3, ext4)
+** +vfat+: generate a VFAT filesystem (FAT16, FAT32)
+
+* +fs_label+ (optional): the label to assign to this filesystem, if that
+  filesystem supports a label
+
+* +fs_files_0+, +fs_files_1+, +fs_files_N+ (optional): the list of files,
+  relative to $(BINARIES_DIR), to store in the filesystem. These entries
+  must be indexed starting from 0, and must be sequential: the first
+  missing entry ends the list
+
+* +fs_root+ (optional): the mountpoint of the filesystem
+
+* +fs_vfstype+ (optional): the type of filesystem to use when calling
+  +mount+, if different from +fs_type+
+
+* +fs_mntopts+ (optional): the mount options; defaults to +defaults+
+
+_Note_: valid use-cases for +fs_root+ and +fs_files_N+:
+
+* if only +fs_root+ is specified (and no +fs_files_N+): the filesystem
+  content is made exclusively from +$(TARGET_DIR)/$(fs_root)+, and the
+  filesystem is mounted at runtime
+
+* if both +fs_root+ and at least one +fs_files_N+ are specified: the
+  filesystem content is made exclusively from the +fs_files_N+ entries,
+  and mounted at runtime. +$(TARGET_DIR)/$(fs_root)+ must be empty
+
+* if at least one +fs_files_N+ is specified, and +fs_root+ is not: the
+  filesystem content is made exclusively from the +fs_files_N+ entries,
+  and the filesystem is not mounted at runtime
+
+* if neither +fs_root+ nor +fs_files_N+ is specified: this is an error
+
+Properties for +fs_type=ext+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +ext_gen+ (mandatory): the generation of extended filesystem to generate
+** +2+, +3+, +4+: for an ext2, ext3 or ext4 filesystem
+
+* +ext_rev+ (mandatory): the revision of the extended filesystem
+** +0+ (ext2 only): generate a revision 0 extended filesystem filesystem
+** +1+ (mandatory for ext3 or ext4): generate a revision 1 extended
+   filesystem
+
+Properties for +fs_type=vfat+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +vfat_size+ (optional): the VFAT-size of the filesystem
+** +12+, +16+, +32+: generate a FAT12, FAT16, or FAT32
+
+Generation of the filesystems
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The filesystems are generated in an order such that a filesystem that is
+mounted as a sub-directory of another filesystem is generated first.
+
+A filesystem is filled with the content of the directory corresponding to
+its mountpoint, and then that directory is emptied before continuing to the
+next filesystem. That way, the mountpoints are empty before the filesystem
+that contains them are generated.
+
+Finally, an entry in added in +/etc/fstab+ for each generated filesystem, so
+they are mounted at boot time.
+
+Examples
+~~~~~~~~
+
+.Simplest partition table layout description
+====
+----
+[global]
+extract=tar
+devices=sda
+
+[sda]
+type=boot
+boot_type=mbr
+partitions=root
+partalign=1048576
+
+[root]
+type=fs
+fs_type=ext
+fs_vfstype=ext4
+fs_root=/
+ext_gen=4
+ext_rev=1
+----
+
+The +partition table layout description+ above defines a single device
++sda+. That device contains a single partition, +root+, with an ext4
+filesystem, which is filled with the whole content of the +rootfs.tar+,
+and is mounted on +/+.
+====
+
+.More complex table layout description
+====
+----
+[global]
+extract=tar
+devices=mmcblk0,sda
+
+[mmcblk0]
+type=boot
+boot_type=mbr
+partitions=boot,root
+partalign=$((1024*1024))
+
+[sda]
+type=boot
+boot_type=mbr
+partitions=data
+partalign=4096
+
+[boot]
+type=fs
+mbr_type=$((0xC))
+size=$((16*1048576))
+fs_type=vfat
+fs_mntopts=ro
+fs_label=BOOT
+fs_root=/boot
+vfat_size=32
+
+[root]
+type=fs
+mbr_type=$((0x83))
+size=268435456
+fs_type=ext
+fs_vfstype=ext4
+fs_mntopts=discard,delalloc
+fs_root=/
+fs_label=ROOT
+ext_gen=4
+ext_rev=1
+
+[data]
+type=fs
+mbr_type=$((0x83))
+size=$((4*1024*1048576))
+fs_type=ext
+fs_vfstype=ext2
+fs_root=/data
+fs_label=DATA
+ext_gen=2
+ext_rev=1
+----
+====
+
+The example above defines two devices, +mmcblk0+ and +sda+.
+
+The +mmcblk0+ device contains two partitions, +boot+ and +root+; partitions
+are aligned on a 1MiB boundary. The +sda+ device contains a single partition,

fs/Config.in +1 -0

index da4c5ff..44e04f7 100644
@@ -3,6 +3,7 @@ menu "Filesystem images"
 source "fs/cloop/Config.in"
 source "fs/cpio/Config.in"
 source "fs/cramfs/Config.in"
+source "fs/custom/Config.in"
 source "fs/ext2/Config.in"
 source "fs/initramfs/Config.in"
 source "fs/iso9660/Config.in"

fs/custom/Config.in new file +16 -0

new file mode 100644
index 0000000..e5a8ee7
@@ -0,0 +1,16 @@
+config BR2_TARGET_ROOTFS_CUSTOM
+       bool "Custom partition table layout"
+       select BR2_TARGET_ROOTFS_TAR
+
+config BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE
+       string "path to the custom partition table layout description"
+       depends on BR2_TARGET_ROOTFS_CUSTOM
+       help
+         Enter the path to a partition-table for your device.
+
+         This will allow Buildroot to generate a more complex target
+         image, which may consist of more than one filesystem on more
+         than one partition.
+
+         See docs/manual/bla-bla on how to construct such a partition
+         table.

fs/custom/boot/gpt new file +21 -0

new file mode 100644
index 0000000..f978524
@@ -0,0 +1,129 @@
+# Build a complete GPT-based image
+
+#-----------------------------------------------------------------------------
+DEPENDS+=( parted )
+
+#-----------------------------------------------------------------------------
+# For a GPT-based partitionning, we need to compute the complete
+# image size before we can attempt to generate the partition table.
+# Then, we need to add the size of the GPT itself, plus that of its
+# backup copy, plus the protective MBR.
+# The size of the GPT itself depends on the sector size, and the
+# number of partitions in the GPT. Sectors can be either 512-byte
+# or 4096-byte large; The numbers of partitions is unlimited, but
+# it is suggested there is space to store at least 128 of them; a
+# partition description is 128-byte large.
+#
+# https://en.wikipedia.org/wiki/GUID_Partition_Table
+#
+# So, here's what we do:
+#   - consider 512-byte sectors (since GPT on 4k sectors is not well
+#     documented)

fs/custom/boot/gpt new file +15 -0

+# add this later on.
+do_image() {
+    # ${1} is fs_root, irrelevant here
+    local img="${2}"
+
+    # How many partitions do we have?
+    nb_parts=0
+    for part in ${partitions[${dev}]//,/ }; do
+        nb_parts=$((nb_parts+1))
+    done
+
+    # How many partition entries do we need?
+    nb_entries=$((4*((nb_parts+3)/4)))
+    nb_entries=$((nb_entries<128?128:nb_entries))
+

fs/custom/boot/gpt new file +63 -0

+
+    # Compute the space required to store all partitions
+    # and store them in the image file
+    size_parts=0
+    _offset=${begin}
+    i=1
+    debug "adding partions descriptions\n"
+    for part in ${partitions[${dev}]//,/ }; do
+        debug "  %s\n" "${part}"
+        _part_img="${tmp_dir}/${dev}.${part}.img"
+        _size=$( align_val $( stat -c '%s' "${_part_img}" ) 512 )
+        part_offset+=( ${_offset} )
+        _attr="${values["${part}:gpt_attr"]}"
+        _label="${values["${part}:gpt_label"]}"
+
+        # If the partition has no label, use the filesystem label
+        if [ -z "${_label}" ]; then
+            _label="${values["${part}:fs_label"]}"
+        fi
+        if [ -z "${_label}" ]; then
+            _label="data"
+        fi
+
+        debug "    start=%s\n" "${_offset}"
+        debug "    size =%s\n" "${_size}"
+        debug "    end  =%s\n" "$((_offset+_size-1))"
+
+        dd if="${_part_img}" of="${img}"    \
+           bs=512 seek=$((_offset/512))     \
+           conv=notrunc,sparse              2>/dev/null
+
+        parted_script+=( mkpart "${_label}"    \
+                                ${_offset}              \
+                                $((_offset+_size-1))    \
+                       )
+        if [ -n "${_attr}" ]; then
+            for attr in "${_attr//,/ }"; do
+                parted_script+=( set ${i} ${attr} on )
+            done
+        fi
+
+        size_parts=$((size_parts+_size))
+        _offset=$((_offset+_size))
+        i=$((i+1))
+    done
+
+    # Terminate our image file
+    img_size=$(align_val $(( begin + size_parts + gpt_size )) 512)
+    debug "begin   =%s\n" ${begin}
+    debug "nb_entry=%s\n" ${nb_entries}
+    debug "gpt_size=%s\n" ${gpt_size}
+    debug "img_size=%s\n" ${img_size}
+    dd if=/dev/zero of="${img}" \
+       bs=1 seek=${img_size}    \
+       count=0 conv=sparse      2>/dev/null
+
+    for i in parted -s "${img}" mklabel gpt unit B "${parted_script[@]}"; do
+        debug "--> '%s'\n" "${i}"
+    done
+    parted -s "${img}" mklabel gpt unit B "${parted_script[@]}"
+}
+
+# vim: ft=sh

fs/custom/boot/mbr new file +64 -0

new file mode 100644
index 0000000..667feed
@@ -0,0 +1,64 @@
+# Build a complete MBR-based image
+
+#-----------------------------------------------------------------------------
+DEPENDS+=( genpart )
+
+#-----------------------------------------------------------------------------
+do_image() {
+    # ${1} is fs_root, irrelevant here
+    local img="${2}"
+    local i begin part part_img size type _begin _size
+    local -a part_offset part_file bootcode
+
+    # Fill-in the boot record
+    bootcode="${values["${dev}:mbr_bootcode"]}"
+    if [ -n "${bootcode}" ]; then
+        bootcode="${BINARIES_DIR}/${bootcode}"
+    else
+        bootcode="/dev/zero"
+    fi
+    debug "adding bootcode '%s'\n" "${bootcode}"
+    dd if="${bootcode}" of="${img}" bs=$((0x1be)) count=1 2>/dev/null
+
+    # Generate partition entries
+    i=0
+    begin=${partalign}
+    debug "adding partitions descriptors\n"
+    for part in ${partitions[${dev}]//,/ }; do
+        debug "  %s\n" "${part}"
+        part_offset+=( ${begin} )
+        part_img="${tmp_dir}/${dev}.${part}.img"
+        part_file+=( "${part_img}" )
+        size=$( align_val $( stat -c '%s' "${part_img}" ) 512 )
+        type="${values["${part}:mbr_type"]}"
+        # LBA is exressed in a number of 512-byte blocks
+        # and genparts only deals with LBA
+        _begin=$((begin/512))   # begin is already 512-byte aligned
+        _size=$((size/512))     # size is already 512-byte aligned
+        debug "    start=%s (LBA %s)\n" "${begin}" "${_begin}"
+        debug "    size =%s (LBA %s)\n" "${size}"  "${_size}"
+        debug "    type =%s\n"          "${type}"
+        genpart -b ${_begin} -s ${_size} -t ${type} >>"${img}"
+        begin=$( align_val $((begin+size)) ${partalign} )
+        i=$((i+1))
+    done
+    nb_parts=${i}
+    # Generate entries for empty partitions
+    for(( ; i<4; i++ )); do
+        debug "  (empty)\n"
+        genpart -t 0 >>"${img}"
+    done
+    # Dump the boot signature
+    printf "\x55\xaa" >>"${img}"
+
+    for(( i=0; i<nb_parts; i++ )); do
+        part_img="${part_file[${i}]}"
+        offset=${part_offset[${i}]}
+        _offset=$(( offset/512 ))  # offset is already 512-byte aligned
+        dd if="${part_img}" of="${img}" \
+           bs=512 seek=${_offset}       \
+           conv=notrunc,sparse          2>/dev/null
+    done
+}
+
+# vim: ft=sh

fs/custom/boot/pre-post new file +14 -0

new file mode 100644
index 0000000..af4bcf5
@@ -0,0 +1,14 @@
+#-----------------------------------------------------------------------------
+# No dependencies
+
+#-----------------------------------------------------------------------------
+do_image_pre() {
+    :
+}
+
+#-----------------------------------------------------------------------------
+do_image_post() {
+    :
+}
+
+#vim: set ft=sh

fs/custom/custom.mk new file +25 -0

new file mode 100644
index 0000000..ca53e0b
@@ -0,0 +1,25 @@
+################################################################################
+#
+# custom partitioning
+#
+################################################################################
+
+# rootfs-custom uses rootfs.tar as the source to generate
+# the resulting image(s), so we need to build it first.
+ROOTFS_CUSTOM_DEPENDENCIES += rootfs-tar
+
+# If we are not selected, we won't have a partition table, so genimages
+# will complain (both on stdout and return code), so it will faill, so
+# we should not get our dependencies
+ifeq ($(BR2_TARGET_ROOTFS_CUSTOM),y)
+ROOTFS_CUSTOM_DEPENDENCIES += \
+       $(patsubst %,host-%,$(shell $(USER_HOOKS_EXTRA_ENV) fs/custom/genimages --show-depends \
+                                   '$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'))
+endif
+
+define ROOTFS_CUSTOM_CMD
+       $(USER_HOOKS_EXTRA_ENV) fs/custom/genimages \
+               '$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'
+endef
+
+$(eval $(call ROOTFS_TARGET,custom))

fs/custom/fs/ext new file +26 -0

new file mode 100644
index 0000000..a184969
@@ -0,0 +1,26 @@
+# Create an extended file system
+
+#-----------------------------------------------------------------------------
+DEPENDS+=( e2fsprogs genext2fs )
+
+#-----------------------------------------------------------------------------
+do_image() {
+    local root_dir="${1}"
+    local img="${2}"
+    local -a fs_opts
+    local gen rev
+
+    fs_opts+=( -z )
+    fs_opts+=( -d "${root_dir}" )
+    [ -z "${size}"    ] || fs_opts+=( -b $((size/1024)) )
+    [ -n "${ext_gen}" ] || ext_gen=2
+    [ -n "${ext_rev}" ] || ext_rev=1
+
+    # Remember, we're running from Buildroot's TOP_DIR
+    GEN=${ext_gen} REV=${ext_rev}                   \
+    ./fs/ext2/genext2fs.sh "${fs_opts[@]}" "${img}" >/dev/null
+
+    [ -z "${fs_label}" ] || tune2fs -L "${fs_label}" "${img}" >/dev/null
+}
+
+# vim: ft=sh

fs/custom/fs/pre-post new file +72 -0

new file mode 100644
index 0000000..d931437
@@ -0,0 +1,72 @@
+#-----------------------------------------------------------------------------
+# No additional dependencies
+
+#-----------------------------------------------------------------------------
+do_image_pre() {
+    local i file
+
+    # if fs_root_dir is not specified, we have to create one
+    # It *does* override the caller's fs_root_dir value, but
+    # that's on purpose
+    # If fs_root_dir is specified, and we have at least fs_files_0,
+    # then fs_root_dir/ must be enpty
+    if [ -z "${fs_root_dir}" ]; then
+        if [ -n "${values["${part}:fs_files_0"]}" ]; then
+            error "%s: no fs_root specified, and no fs_files_0\n" "${part}"
+        fi
+        fs_root_dir="$( mktemp -d "${tmp_dir}/XXXXXX" )"
+    else
+        if [    -n "${values["${part}:fs_files_0"]}"    \
+             -a $( ls -1A "${fs_root_dir}" 2>/dev/null  \
+                   |wc -l                               \
+                 ) -ne 0                                ]; then
+            error "%s: %s is not empty, but fs_files_0 is specified\n"  \
+                  "${part}" "${fs_root_dir#${fs_root}}"
+        fi
+    fi
+
+    i=0
+    while true; do
+        file="${values["${part}:fs_files_${i}"]}"
+        [ -n "${file}" ] || break
+        debug "%s: adding fs_files_%d %s\n" "${part}" ${i} "${file}"
+        install -D "${BINARIES_DIR}/${file}" "${fs_root_dir}/${file##*/}"
+        i=$((i+1))
+    done
+}
+
+#-----------------------------------------------------------------------------
+do_image_post() {
+    local rootfs_dir="${1}"
+    local fs_root="${2}"
+    local img_file="${3}"
+    local part="${4}"
+    local dev mntops vfstype fs_root_esc
+
+    subname+="[post-image]"
+
+    # Empty the partition's mountpoint
+    find "${fs_root_dir}" -maxdepth 1 \! -path "${fs_root_dir}" -exec rm -rf {} +
+
+    # Add entry in fstab, but not if this is '/'
+    # Don't add either if rootfs was not extracted
+    if [    "${fs_root}" = "/" -o -z "${fs_root}" \
+         -o -z "${values["global:extract"]}"      ]; then
+        return 0
+    fi
+    fs_root_esc="$( sed -r -e 's:/:\\/:g;' <<<"${fs_root}" )"
+    sed -r -i -e "/[^[:space:]]+[[:space:]]+${fs_root_esc}[[:space:]]/d"    \
+                 "${rootfs_dir}/etc/fstab"
+    dev="$( get_part_dev_node "${part}" )"
+    vfstype="${fs_vfstype:-${fs_type}}"
+    mntops="${fs_mntops:-defaults}"
+    printf "/dev/%s %s %s %s 0 0\n"     \
+           "${dev}" "${fs_root}"        \
+           "${vfstype}" "${mntops}"     \
+           >>"${rootfs_dir}/etc/fstab"
+
+    subname="${subname%\[post-image\]}"
+}
+
+#-----------------------------------------------------------------------------
+# vim: ft=sh

fs/custom/fs/vfat new file +21 -0

new file mode 100644
index 0000000..5fc243d
@@ -0,0 +1,21 @@
+# Create a VFAT file system
+
+#-----------------------------------------------------------------------------
+DEPENDS+=( dosfstools mtools )
+
+#-----------------------------------------------------------------------------
+do_image() {
+    local root_dir="${1}"
+    local img="${2}"
+    local -a fs_opts
+
+    dd if=/dev/zero of="${img}" bs=${size} count=0 seek=1 2>/dev/null
+
+    [ -z "${vfat_size}" ] || fs_opts+=( -F ${vfat_size} )
+    [ -z "${fs_label}"  ] || fs_opts+=( -n "${fs_label}" )
+    mkfs.vfat "${fs_opts[@]}" "${img}" >/dev/null
+
+    mcopy -i "${img}" "${root_dir}/"* '::'
+}
+
+# vim: ft=sh

fs/custom/genimages new file +198 -0

new file mode 100755
index 0000000..204f13b
@@ -0,0 +1,342 @@
+#!/bin/bash
+
+#-----------------------------------------------------------------------------
+main() {
+    local part_table="${1}"
+    local tmp_dir
+    local rootfs_dir
+    local -a devices
+    local extract
+    local cur_section
+    local -a sections devices partitions
+    local -A variables values partdevs
+    local sec dev part var val
+    local secs devs parts vars vals
+    local has_global_section
+
+    # We need bash 4 or above for associative arrays
+    if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
+        error "bash 4 or above is needed\n"
+    fi
+
+    if [ "${part_table}" = "--show-depends" ]; then
+        SHOW_DEPENDS=1
+        part_table="${2}"
+        trace() { :; } # Be silent, we just want the dependencies...
+    else
+        SHOW_DEPENDS=0
+    fi
+
+    if [ ! -f "${part_table}" ]; then
+        error "'%s': no such file\n" "${part_table}"
+        exit 1
+    fi
+
+    export PATH="${HOST_DIR}/usr/bin:${HOST_DIR}/usr/sbin:${PATH}"
+
+    # Parse all the sections in one go, we'll sort
+    # all the mess afterwards...
+    debug "parsing partitions descriptions file '%s'\n" \
+          "${part_table}"
+    parse_ini "${part_table}"
+
+    # The 'global' section is mandatory
+    has_global_section=0
+    for s in "${sections[@]}"; do
+        if [ "${s}" = "global" ]; then
+            has_global_section=1
+            break
+        fi
+    done
+    if [ ${has_global_section} -eq 0 ]; then
+        error "no global section defined\n"
+    fi
+
+    # Create lists of devices, partitions, and partition:device pairs.
+    debug "creating intermediate lists\n"
+    devices=( ${values["global:devices"]//,/ } )
+    for dev in "${devices[@]}"; do
+        # Sanity check first: all devices must have a corresponding
+        # section, which means they should have a type
+        if [ -z "${values["${dev}:type"]}" ]; then
+            error "device '%s' has no type (no section?)\n" "${dev}"
+        fi
+        partitions+=( ${values["${dev}:partitions"]//,/ } )
+        for part in ${values["${dev}:partitions"]//,/ }; do
+            # Sanity check first: all partitions must have a corresponding
+            # section, which means they should have a type
+            if [ -z "${values["${part}:type"]}" ]; then
+                error "partition '%s:%s' has no type (no section?)\n"   \
+                      "${dev}" "${part}"
+            fi
+            partdevs+=( ["${part}"]="${dev}" )
+        done
+    done
+
+    # Now, we must order the partitions so that their mountpoint
+    # is empty by the time we build the upper-level partition.
+    # For example, given this layout of mountpoints:
+    #   /
+    #   /usr
+    #   /usr/var
+    # We must ensure /usr/var is empty at the time we create the /usr
+    # filesystem image; and similarly, we must ensure /usr is empty by
+    # the time we create the / filesystem image
+    # So, a simple reverse alphabetical sort will do the trick
+    debug "sorting partitions\n"
+    sorted_parts=( $(
+        for part in "${partitions[@]}"; do
+            # Partitions that are not mounted can be generated
+            # in any order
+            if [ -n "${values["${part}:fs_root"]}" ]; then
+                printf "%s:%s\n" "${part}" "${values["${part}:fs_root"]}"
+            else
+                printf "%s\n" "${part}"
+            fi
+        done                    \
+        |sort -t: -k2 -r        \
+        |sed -r -e 's/:[^:]+$//;'
+    ) )
+
+    # We do not want to create anything if we only want the dependencies
+    if [ ${SHOW_DEPENDS} -eq 0 ]; then
+        tmp_dir="${BUILD_DIR}/genimages.tmp"
+        rootfs_dir="${tmp_dir}/rootfs"
+        # Since we don't remove it in case of error (to be able to inspect its
+        # content), we must remove it now (a previous run may have left it).
+        rm -rf "${tmp_dir}"
+        mkdir -p "${rootfs_dir}"
+
+        case "${values["global:extract"]}" in
+            tar)
+                # We must be root for the extract to work correctly
+                # (since it may have /dev nodes, or some files may
+                # belong to different users...)
+                if [ $(id -u) -ne 0 ]; then
+                    error "error: not root\n"
+                fi
+                trace "extracting rootfs.tar\n"
+                tar xf "${BINARIES_DIR}/rootfs.tar" -C "${rootfs_dir}"
+            ;;
+            *)  error "unknown extract method '%s'\n" "${extract:-(none)}"
+            ;;
+        esac
+    fi # ! SHOW_DEPENDS
+
+    # Render all partition images
+    for part in "${sorted_parts[@]}"; do
+        trace "preparing filesystem for partition '%s'\n" "${part}"
+        render_img "${rootfs_dir}" "${part}"                        \
+                   "${tmp_dir}/${partdevs["${part}"]}.${part}.img"
+    done
+
+    # Aggregate all devices images
+    for dev in "${devices[@]}"; do
+        trace "assembling partitions in device '%s'\n" "${dev}"
+        render_img "${rootfs_dir}" "${dev}" "${tmp_dir}/${dev}.img"
+    done
+
+    # If we need the dependencies, we can stop right now
+    if [ ${SHOW_DEPENDS} -eq 1 ]; then
+        return 0
+    fi
+
+    # Copy all partitions and devices images to the image dir
+    if [ "${values["global:keep_partitions"]}" = "yes" ]; then
+        for part in "${sorted_parts[@]}"; do
+            debug "copying partition '%s' to image dir\n" "${part}"
+            dd if="${tmp_dir}/${partdevs["${part}"]}.${part}.img"           \
+               of="${BINARIES_DIR}/$( get_part_dev_node "${part}" ).img"    \
+               bs=4096 conv=sparse                                          2>/dev/null
+        done
+    fi
+    for dev in "${devices[@]}"; do
+        debug "copying device '%s' to image dir\n" "${dev}"
+        dd if="${tmp_dir}/${dev}.img"       \
+           of="${BINARIES_DIR}/${dev}.img"  \
+           bs=4096 conv=sparse              2>/dev/null
+    done
+
+    [ -n "${DEBUG}" ] || rm -rf "${tmp_dir}"
+}
+
+#-----------------------------------------------------------------------------
+render_img() {
+    local rootfs_dir="${1}"
+    local img="${2}"
+    local img_file="${3}"
+    local type sub_type fs_root_dir
+
+    type="${values["${img}:type"]}"
+    sub_type="${values["${img}:${type}_type"]}"
+
+    # Sanity checks
+    [ -n "${type}" ] || error "'%s': unspecified type\n" "${img}"
+    if [ ! -d "fs/custom/${type}" ]; then
+        error "'%s': unsupported type '%s'\n" "${img}" "${type}"
+    fi
+    [ -n "${sub_type}" ] || error "'%s': unspecified %s_type\n" "${img}" "${type}"
+    if [ ! -f "fs/custom/${type}/${sub_type}" ]; then
+        error "'%s': unknown %s_type '%s'\n" "${img}" "${type}" "${sub_type}"
+    fi
+
+    # Need to call the renderer in a subshell so that its definitions
+    # do not pollute our environment
+    subname="${sub_type}"
+    (
+        trap 'exit $?' ERR
+
+        declare -a DEPENDS
+
+        for var in ${variables["${img}"]//,/ }; do
+            eval "${var}=\"${values["${img}:${var}"]}\""
+        done
+        fs_root_dir="${rootfs_dir}${fs_root}"
+        . "fs/custom/${type}/pre-post"
+        . "fs/custom/${type}/${sub_type}"
+        if [ ${SHOW_DEPENDS} -eq 1 ]; then
+            for dep in "${DEPENDS[@]}"; do

fs/custom/genimages new file +134 -1

+    subname=""
+}
+
+#------------------------------------------------------------------------------
+# Parse a .ini file
+#   $1: .ini file to parse
+# The caller should define the following variables:
+#   sections    : array
+#   variables   : associative array
+#   values      : associative array
+# parse_ini() will fill those variables with:
+#   sections    : the list of sections, one section per array index
+#   variables   : the comma-separated list of varibles for a section,
+#                 indexed by the name of the section
+#   values      : the value of a variable in a section, indexed by the
+#                 'section:variable' tuple
+# Eg.:
+#   sections=( [0]='section-0' [1]='section-1' )
+#   variables=( ['section-0']='var-0,var-1' ['section-1']='var-10,var-11' [...] )
+#   values=( ['section-0:var-0']='value-0-0' ['section-1:var-10']='value-1-10' [...] )
+parse_ini() {
+    local ini_file="${1}"
+    local line var val
+    local cur_section
+    local var_sep
+
+    while read line; do
+        line="$( sed -r -e 's/[[:space:]]*#.*$//; //d;' <<<"${line}" )"
+
+        # Detect start of global section, skip anything else
+        case "${line}" in
+        "") continue;;
+        '['*']')
+            cur_section="$( sed -r -e 's/[][]//g;' <<<"${line}" )"
+            debug "  entering section '%s'\n" "${cur_section}"
+            sections+=( "${cur_section}" )
+            continue
+        ;;
+        ?*=*)   ;;
+        *)      error "malformed entry '%s'\n" "${line}";;
+        esac
+
+        var="${line%%=*}"
+        eval val="${line#*=}"
+        debug "    adding '%s'='%s'\n" "${var}" "${val}"
+        var_sep="${variables["${cur_section}"]+,}"
+        variables+=( ["${cur_section}"]="${var_sep}${var}" )
+        values+=( ["${cur_section}:${var}"]="${val}" )
+    done <"${ini_file}"
+}
+
+#-----------------------------------------------------------------------------
+get_part_dev_node() {
+    local part="${1}"
+    local dev
+    local i c p
+
+    dev="${partdevs["${part}"]}"
+    i="${values["${dev}:partstart"]:-1}"
+
+    # If device node ends with a number, partitions are denoted
+    # with a 'p' before the partition number, eg.:
+    #   /dev/mmcblk0    --> /dev/mmcblk0p1
+    #   /dev/sda        --> /dev/sda1
+    case "${dev#${dev%?}}" in
+        [0-9])  c="p";;
+        *)      c="";;
+    esac
+
+    for p in ${values["${dev}:partitions"]//,/ }; do
+        if [ "${p}" = "${part}" ]; then
+            printf "%s%s%d" "${dev}" "${c}" ${i}
+            return 0
+        fi
+        i=$((i+1))
+    done
+
+    error "'%s': partition not found. WTF?\n" "${part}"
+}
+
+#------------------------------------------------------------------------------
+align_val() {
+    local val="${1}"
+    local align="${2}"
+    local aligned
+
+    aligned=$(( ( (val+align-1) / align ) * align ))
+
+    printf "%d" ${aligned}
+}
+
+#------------------------------------------------------------------------------
+# Some trace functions
+_trace() {
+    local fmt="${1}"
+    shift
+
+    printf "%s" "${myname}"
+    if [ -n "${subname}" ]; then
+        printf "(%s)" "${subname}"
+    fi
+    printf ": ${fmt}" "${@}"
+}
+
+trace() {
+    _trace "${@}"
+}
+
+debug() { :; }
+if [ -n "${DEBUG}" ]; then
+    debug() {
+        _trace "${@}" >&2
+    }
+fi
+
+error() {
+    _trace "${@}" >&2
+    exit 1
+}
+
+on_error() {
+    local ret=${?}
+
+    error "unexpected error caught: %d\n" ${ret}
+}
+trap on_error ERR
+set -E -e
+
+#-----------------------------------------------------------------------------
+export myname="${0##*/}"
+
+main "${@}"
+
+# vim: ft=sh
--

1.8.1.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

When doing scripting in Buildroot, do we accept the uses of various bash-ism or do we want to stick to POSIX shell ?

docs/manual/customize-filesystems.txt

Quoted 6 lines

docs/manual/customize-filesystems.txt new file +0 -0

new file mode 100644
index 0000000..fd65c97
@@ -0,0 +1,36 @@

How about comments ? Are they supported ?
In a next step, it would be really nice to have support for "human" input for the different size, like 1G/M/K.

fs/custom/boot/gpt

Quoted 6 lines

fs/custom/boot/gpt new file +0 -0

new file mode 100644
index 0000000..f978524
@@ -0,0 +1,129 @@

fs/custom/boot/gpt line 31

Quoted 1 line

fs/custom/boot/gpt new file +1 -0

+#   Simg = 512 + 2*(Sgpt) + ?( aligned(Spart,512) )

Don't we want only ASCII char there ?

fs/custom/boot/gpt line 41

Quoted 1 line

fs/custom/boot/gpt new file +1 -0

+# Sicne 4k-large sectors are not really explained on Wikipedia, we can

Small typo here.

fs/custom/boot/gpt line 66

Quoted 3 lines

fs/custom/boot/gpt new file +3 -0

+    dd if=/dev/zero of="${img}"     \
+       bs=1 seek=${begin} count=0   \
+       conv=sparse                  2>/dev/null

Why the redirect to /dev/null ? Don't we want to see if there was an error ? I know it's unlikely to happen but you never know. I have noticed you did several times so maybe there is a reason I am not seeing here.

fs/custom/boot/gpt line 7

Quoted 2 lines

fs/custom/boot/gpt new file +2 -0

+    ret=${?}
+    [ ${ret} -eq 0 ] || exit ${ret}

Why not doing: [ ${ret} -ne 0 ] && exit ${ret} ?

Maxime, All,

Thanks for the review! :-)

On 2014-01-04 17:38 +0100, Maxime Hadjinlian spake thusly:

Quoted 2 lines

When doing scripting in Buildroot, do we accept the uses of various bash-ism or do we want to stick to POSIX shell ?

bash is already a hard-dependency of Buildroot.

Quoted 1 line · Yann E. MORIN

[--SNIP--]

Quoted 6 lines

docs/manual/customize-filesystems.txt

Quoted 6 lines

docs/manual/customize-filesystems.txt new file +0 -0

new file mode 100644
index 0000000..fd65c97
@@ -0,0 +1,36 @@

[--SNIP--]

Quoted 3 lines

How about comments ? Are they supported ?
In a next step, it would be really nice to have support for "human" input for the different size, like 1G/M/K.

Yes, comments are supported; I've now added that to the manual.

I've also added that values can be expresed as shell arith,metic expansion, such as $((16*1048576)) (aka 16MiB).

Not that G/M/k are SI-prefixes that respectively mean 10^9, 10^6 and 10^3. The binary prefixes are Gi/Mi/ki, respectively 2^30, 2^20 and 2^10.

If/when we add support for this kind of values, we'll need to add both decimal and binary prefixes.

Quoted 7 lines

fs/custom/boot/gpt

Quoted 7 lines

fs/custom/boot/gpt new file +0 -0

new file mode 100644
index 0000000..f978524
@@ -0,0 +1,129 @@

[--SNIP--]

Quoted 2 lines

fs/custom/boot/gpt line 31

Quoted 1 line

fs/custom/boot/gpt new file +1 -0

+#   Simg = 512 + 2*(Sgpt) + ?( aligned(Spart,512) )

Don't we want only ASCII char there ?

I don't know.

I find it convenient not to limit ourselves to ASCII.

[--SNIP--]

Quoted 2 lines

fs/custom/boot/gpt line 41

Quoted 1 line

fs/custom/boot/gpt new file +1 -0

+# Sicne 4k-large sectors are not really explained on Wikipedia, we can

Small typo here.

Thanks, fixed.

[--SNIP--]

Quoted 7 lines

fs/custom/boot/gpt line 66

Quoted 3 lines

fs/custom/boot/gpt new file +3 -0

+    dd if=/dev/zero of="${img}"     \
+       bs=1 seek=${begin} count=0   \
+       conv=sparse                  2>/dev/null

Why the redirect to /dev/null ? Don't we want to see if there was an error ? I know it's unlikely to happen but you never know. I have noticed you did several times so maybe there is a reason I am not seeing here.

dd will always print something to stderr, even if successful. We do not want it to pollute the output, since we're using nice traces.

[--SNIP--]

Quoted 3 lines

fs/custom/boot/gpt line 7

Quoted 2 lines

fs/custom/boot/gpt new file +2 -0

+    ret=${?}
+    [ ${ret} -eq 0 ] || exit ${ret}

Why not doing: [ ${ret} -ne 0 ] && exit ${ret} ?

Same answer as to your comment in squashfs, since we're using set -e.

But here, it's a bit complex. POSIX does not specify whether a sub-shell that exits with a return code !0 should be caught by set -e. And bash-4.0 does not catch it, while bash-4.2 does catch it. Both are POSIX complianti in this respect, since POSIX does not state what should be done in this case.

So, we have to explicitly store the return code, and test it.

But if ret==0, your test would fail, and the complete shell compound command " [ ... ] && ... " would also be false, and we would create an error that the shell would catch because of set -e, when there was no error to begin with.

Thank you for the comments! :-)

Regards,
Yann E. MORIN.

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

From: "Yann E. MORIN" <[email protected]>

Contrary to the existing fs/ schemes, which each generate only a single filesystem image for the root filesystem, this new scheme allows the user to generate more complex images.

The basis behind this is a .ini-like description of the layout of the final target storage:
- the list of device(s)
- per-device, the list of partition(s)
- per-partition, the content

It is possible to create MBR- or GPT-based partitoining schemes. Adding new ones should be relatively easy (but would need adequate host tools).

For now, the only content possible for partitions is a filesystem. It should be pretty easy to add new types (eg. un-formated, or raw blob).

Also, only two filesystems are supported: ext{2,3,4} and vfat. Adding more will be relatively easy, provided we have the necessary host packages to deal with those filesystems.

Hi Yann,

Quoted 5 lines

From: "Yann E. MORIN" <[email protected]>

Contrary to the existing fs/ schemes, which each generate only a single filesystem image for the root filesystem, this new scheme allows the user to generate more complex images.

if found this an interesting idea, but apparently it has not found its way into master yet. What is the current state?

Quoted 12 lines

The basis behind this is a .ini-like description of the layout of the final target storage:
- the list of device(s)
- per-device, the list of partition(s)
- per-partition, the content

It is possible to create MBR- or GPT-based partitoining schemes. Adding new ones should be relatively easy (but would need adequate host tools).

For now, the only content possible for partitions is a filesystem. It should be pretty easy to add new types (eg. un-formated, or raw blob).

Quoted 3 lines

Also, only two filesystems are supported: ext{2,3,4} and vfat. Adding more will be relatively easy, provided we have the necessary host packages to deal with those filesystems.

However, I'not sure if it actually is what I am looking for. Which is dividing the target content into multiple partitions, which are then mounted into the rootfs (like /data, /etc/somespecialconfig, /var/log).?

Are there other ways to do that?

Quoted 1 line

into master yet. What is the current state?

This has been somewhat refused: we do not want to add new tools in Buildroot, and this 'genimages' stuff is seen as a new tool, rather than as infrastructure.

However, it has been suggested that we:

- transform this genimages into a separate project which provides a new filesystem generator, and which Buildroot could use like it is currently using the other filesystem generators;

- look at Pengutronix' genimage (without a trailing 's') to see if it would fit our needs.

I haven't had time to properly look at Pengutronix' genimage, but last I looked, it seemd a bit different than my genimages. I am not too fond of genimage, but that's mostly a personal feeling, and is not really founded on technical merits. I prefer my genimages implementation, which I know how to deal with.

As for my genimages: I am still working on it, and it so far fits all my personal needs. I still have some feature that are missing for now, that are really needed before it can be widely deployed:

- mixed filesystem hierarchies split between an initramfs and partitions

     on a real device;

- partitions to be mounted that are not to be written (eg. when you upgrade, you do not want to overwrite a config partition, but you still want to have an fstab entry for it).

That's being hamerred out, but it takes a bit of time...

Quoted 12 lines

Quoted 12 lines

The basis behind this is a .ini-like description of the layout of the final target storage:
- the list of device(s)
- per-device, the list of partition(s)
- per-partition, the content

It is possible to create MBR- or GPT-based partitoining schemes. Adding new ones should be relatively easy (but would need adequate host tools).

For now, the only content possible for partitions is a filesystem. It should be pretty easy to add new types (eg. un-formated, or raw blob).

Quoted 7 lines

Quoted 3 lines

Also, only two filesystems are supported: ext{2,3,4} and vfat. Adding more will be relatively easy, provided we have the necessary host packages to deal with those filesystems.

However, I'not sure if it actually is what I am looking for. Which is dividing the target content into multiple partitions, which are then mounted into the rootfs (like /data, /etc/somespecialconfig, /var/log).?

Yes, that's exactly what genimages is supposed to cover: a filesystem hierarchy made of more than the / mountpoint.

Quoted 1 line

Are there other ways to do that?

Depends what you mean. I am using a 'wrapper' project arounf Buildroot, which genimages originates from. In this project, I'm using genimages as a post-image script that just extract the rootfs.tar and works from there to generate the multiple filesystem images for the different partitions.

But it is not really usable outside of my project, since it is tightly coupled with the post-build script.

Basically, if all you have are filesystem on reald decives (flash, HDD...), then genimages could world stand-alone.

But if your setup involves an initramfs for the rootfs, plus one or more partitions on a real device, you have to fill entries for those partitions in etc/fstab, but doing so in post-image is too late, because the etc/fstab is in the initramfs, which is in the kernel image, which is assembled during the image phase. So we can't fill-in etc/fstab from a post-image script in this case. [*]

However, it is always possible to fill-in etc/fstab from a post-build (aka pre-image) script, whether your rootfs is on a real device or an initramfs. And that's what my post-build script does (among many other things).

So my genimages is tightly coupled to my rootfs-fixup post-build script, and I currently do not see a clean way to provide genimages as just another filesystem generator. If we wanted to truly have genimages in Buildroot, we'd need more than hooking after the rootfs.tar generator.

[*] Well, it could be possible to call back to Buildroot to regenerate the kernel image with the initramfs once we tweaked etc/fstab, but that's pretty ugly, and prone to errors.)

If you want to have a look, here's my "buildroot.config' project:
http://ymorin.is-a-geek.org/git/buildroot.config/

There's currently pretty much no documentation, besides the one I wrote for inclusion in Buildroot (available in this thread). That's something I should really write... The README is a bit out-dated, so not completely accurate, but the idea is there.

I have a tutorial on how to use it to build an XBMC for the RPi:
http://ymorin.is-a-geek.org/projects/rpi-xbmc

Any suggestion/idea on how to progress on Buildroot is welcome, of course! ;-)

Regards,
Yann E. MORIN.

From: "Yann E. MORIN" <[email protected]>

Now we can generate a complete target storage image with the genimages infra, add a partition table layout description for the Raspberry Pi as an example for how to use genimages.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Ryan Barnett <[email protected]>

Cc Thomas Petazzoni <[email protected]>

---
 board/raspberrypi/partitions  | 35 +++++++++++++++++++++++++++++++++++
 configs/raspberrypi_defconfig |  3 +++
 2 files changed, 38 insertions(+)
 create mode 100644 board/raspberrypi/partitions

board/raspberrypi/partitions new file +35 -0

new file mode 100644
index 0000000..dea9aff
@@ -0,0 +1,35 @@
+[global]
+extract=tar
+devices=mmcblk0
+
+[mmcblk0]
+type=boot
+boot_type=mbr
+partitions=boot,root
+partalign=$((1048576))
+
+[root]
+type=fs
+size=$((32*1048576))
+mbr_type=$((0x83))
+fs_type=ext
+fs_vfstype=ext4
+fs_mntops=discard
+fs_root=/
+fs_label=ROOT
+ext_gen=4
+ext_rev=1
+
+[boot]
+type=fs
+size=$((9*1048576))
+mbr_type=$((0xc))
+fs_type=vfat
+fs_files_0=rpi-firmware/bootcode.bin
+fs_files_1=rpi-firmware/start.elf
+fs_files_2=rpi-firmware/fixup.dat
+fs_files_3=rpi-firmware/config.txt
+fs_files_4=rpi-firmware/cmdline.txt
+fs_files_5=zImage
+fs_label=BOOT
+vfat_size=16

configs/raspberrypi_defconfig +3 -1

index 6a17165..50caf5f 100644
@@ -21,3 +21,6 @@ BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION="75d03120bc36b1cc3166973b8edc11f033ab7c0d"
 BR2_LINUX_KERNEL_USE_DEFCONFIG=y
 BR2_LINUX_KERNEL_DEFCONFIG="bcmrpi_quick"
 BR2_LINUX_KERNEL_ZIMAGE=y
+
+BR2_TARGET_ROOTFS_CUSTOM=y
+BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE="$(TOPDIR)/board/raspberrypi/partitions"
--

1.8.1.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Quoted 63 lines

From: "Yann E. MORIN" <[email protected]>

Now we can generate a complete target storage image with the genimages infra, add a partition table layout description for the Raspberry Pi as an example for how to use genimages.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Ryan Barnett <[email protected]>

Cc Thomas Petazzoni <[email protected]>

---
 board/raspberrypi/partitions  | 35 +++++++++++++++++++++++++++++++++++
 configs/raspberrypi_defconfig |  3 +++
 2 files changed, 38 insertions(+)
 create mode 100644 board/raspberrypi/partitions

board/raspberrypi/partitions new file +35 -0

new file mode 100644
index 0000000..dea9aff
@@ -0,0 +1,35 @@
+[global]
+extract=tar
+devices=mmcblk0
+
+[mmcblk0]
+type=boot
+boot_type=mbr
+partitions=boot,root
+partalign=$((1048576))
+
+[root]
+type=fs
+size=$((32*1048576))
+mbr_type=$((0x83))
+fs_type=ext
+fs_vfstype=ext4
+fs_mntops=discard
+fs_root=/
+fs_label=ROOT
+ext_gen=4
+ext_rev=1
+
+[boot]
+type=fs
+size=$((9*1048576))
+mbr_type=$((0xc))
+fs_type=vfat
+fs_files_0=rpi-firmware/bootcode.bin
+fs_files_1=rpi-firmware/start.elf
+fs_files_2=rpi-firmware/fixup.dat
+fs_files_3=rpi-firmware/config.txt
+fs_files_4=rpi-firmware/cmdline.txt
+fs_files_5=zImage
+fs_label=BOOT
+vfat_size=16

configs/raspberrypi_defconfig +0 -0

index 6a17165..50caf5f 100644
@@ -21,3 +21,6 @@ BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION="75d03120bc36b1cc3166973b8edc11f033ab7c0d"
 BR2_LINUX_KERNEL_USE_DEFCONFIG=y
 BR2_LINUX_KERNEL_DEFCONFIG="bcmrpi_quick"

Quoted 6 lines

configs/raspberrypi_defconfig +3 -1

 BR2_LINUX_KERNEL_ZIMAGE=y
+
+BR2_TARGET_ROOTFS_CUSTOM=y
+BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE="$(TOPDIR)/board/raspberrypi/partitions"
--

1.8.1.2

Could you please add comments over the various size and also the different type used, 0x83 is not really user friendly to read.

Maxime, All,

On 2014-01-04 17:40 +0100, Maxime Hadjinlian spake thusly:

Quoted 64 lines · Yann E. MORIN

Quoted 63 lines

From: "Yann E. MORIN" <[email protected]>

Now we can generate a complete target storage image with the genimages infra, add a partition table layout description for the Raspberry Pi as an example for how to use genimages.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Ryan Barnett <[email protected]>

Cc Thomas Petazzoni <[email protected]>

---
 board/raspberrypi/partitions  | 35 +++++++++++++++++++++++++++++++++++
 configs/raspberrypi_defconfig |  3 +++
 2 files changed, 38 insertions(+)
 create mode 100644 board/raspberrypi/partitions

board/raspberrypi/partitions new file +35 -0

new file mode 100644
index 0000000..dea9aff
@@ -0,0 +1,35 @@
+[global]
+extract=tar
+devices=mmcblk0
+
+[mmcblk0]
+type=boot
+boot_type=mbr
+partitions=boot,root
+partalign=$((1048576))
+
+[root]
+type=fs
+size=$((32*1048576))
+mbr_type=$((0x83))
+fs_type=ext
+fs_vfstype=ext4
+fs_mntops=discard
+fs_root=/
+fs_label=ROOT
+ext_gen=4
+ext_rev=1
+
+[boot]
+type=fs
+size=$((9*1048576))
+mbr_type=$((0xc))
+fs_type=vfat
+fs_files_0=rpi-firmware/bootcode.bin
+fs_files_1=rpi-firmware/start.elf
+fs_files_2=rpi-firmware/fixup.dat
+fs_files_3=rpi-firmware/config.txt
+fs_files_4=rpi-firmware/cmdline.txt
+fs_files_5=zImage
+fs_label=BOOT
+vfat_size=16

configs/raspberrypi_defconfig +0 -0

index 6a17165..50caf5f 100644
@@ -21,3 +21,6 @@ BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION="75d03120bc36b1cc3166973b8edc11f033ab7c0d"
 BR2_LINUX_KERNEL_USE_DEFCONFIG=y
 BR2_LINUX_KERNEL_DEFCONFIG="bcmrpi_quick"

Quoted 8 lines

Quoted 6 lines

configs/raspberrypi_defconfig +3 -1

 BR2_LINUX_KERNEL_ZIMAGE=y
+
+BR2_TARGET_ROOTFS_CUSTOM=y
+BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE="$(TOPDIR)/board/raspberrypi/partitions"
--

1.8.1.2

Could you please add comments over the various size and also the different type used, 0x83 is not really user friendly to read.

0x83 and 0xc are partition types, and there is a pointer in the manual for where to find their meanings and the different values:

* mbr_type (mandatory): the partition [type]

where [type] is a pointer to:
https://en.wikipedia.org/wiki/Partition_type#List_of_partition_IDs

But I can add such comments as this is an example.

Regards,
Yann E. MORIN.

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

Quoted 15 lines

On 2014-01-04 17:40 +0100, Maxime Hadjinlian spake thusly:

Quoted 8 lines · Yann E. MORIN

board/raspberrypi/partitions line 26

Quoted 5 lines

board/raspberrypi/partitions new file +5 -0

+partalign=$((1048576))
+size=$((32*1048576))
+mbr_type=$((0x83))
+size=$((9*1048576))
+mbr_type=$((0xc))

Could you please add comments over the various size and also the different type used, 0x83 is not really user friendly to read.

0x83 and 0xc are partition types, and there is a pointer in the manual for where to find their meanings and the different values:

* mbr_type (mandatory): the partition [type]

Quoted 4 lines

where [type] is a pointer to:
https://en.wikipedia.org/wiki/Partition_type#List_of_partition_IDs

But I can add such comments as this is an example.

Even if it's not an example.
This kind of file are meant to be read by humans, and as such, every information such as partition type or size, should be along side a comments for the human reader.

Maxime, All,

On 2014-01-04 18:12 +0100, Maxime Hadjinlian spake thusly:

Quoted 16 lines · Yann E. MORIN

Quoted 15 lines

On 2014-01-04 17:40 +0100, Maxime Hadjinlian spake thusly:

Quoted 8 lines · Yann E. MORIN

board/raspberrypi/partitions line 26

Quoted 5 lines

board/raspberrypi/partitions new file +5 -0

+partalign=$((1048576))
+size=$((32*1048576))
+mbr_type=$((0x83))
+size=$((9*1048576))
+mbr_type=$((0xc))

Could you please add comments over the various size and also the different type used, 0x83 is not really user friendly to read.

0x83 and 0xc are partition types, and there is a pointer in the manual for where to find their meanings and the different values:

* mbr_type (mandatory): the partition [type]

Quoted 8 lines

Quoted 4 lines

where [type] is a pointer to:
https://en.wikipedia.org/wiki/Partition_type#List_of_partition_IDs

But I can add such comments as this is an example.

Even if it's not an example.
This kind of file are meant to be read by humans, and as such, every information such as partition type or size, should be along side a comments for the human reader.

<humour>
Sorry, but I can read 9*1048576 very easily. But I'm no good reference, since I know by heart those powers of 2 up to about 16777216... :-p

As for 0x83 and 0xc, I have run fdisk so many a time that I know what they mean, too! :-p
</humour>

Regards,
Yann E. MORIN.

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

From: "Yann E. MORIN" <[email protected]>

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc Ryan Barnett <[email protected]>

---
 docs/manual/partition-layout.txt | 11 ++++++++++-
 fs/custom/fs/squashfs            | 19 +++++++++++++++++++
 2 files changed, 29 insertions(+), 1 deletion(-)
 create mode 100644 fs/custom/fs/squashfs

docs/manual/partition-layout.txt +10 -1

index 092ae1b..e9027bf 100644
@@ -133,7 +133,8 @@ Properties for +type=fs+
 
 * +fs_type+ (mandatory): the type of filesystem to generate
 ** +ext+: generate an extended filesystem (ext2, ext3, ext4)
-** +vfat+: generate a VFAT filesystem (FAT16, FAT32)
+** +squashfs+: generate a squashfs (version 4) filesystem
+** +vfat+: generate a VFAT filesystem (FAT12, FAT16 or FAT32)
 
 * +fs_label+ (optional): the label to assign to this filesystem, if that
   filesystem supports a label

@@ -177,6 +178,14 @@ Properties for +fs_type=ext+
 ** +1+ (mandatory for ext3 or ext4): generate a revision 1 extended
    filesystem
 
+Properties for +fs_type=squashfs+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +squashfs_comp+ (mandatory): the compression type to use
+** +gzip+, +xz+, +lzo+: gzip, xz or lzo compression
+
+* +squashfs_block+ (optional): the size of blocks, in bytes
+
 Properties for +fs_type=vfat+
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

fs/custom/fs/squashfs new file +3 -0

new file mode 100644
index 0000000..132d3ef
@@ -0,0 +1,19 @@
+# Create a squashfs filesystem
+
+#-----------------------------------------------------------------------------

fs/custom/fs/squashfs new file +6 -1

+    [ -z "${squashfs_block}" ] || fs_opts+=( -b ${squahsfs_block} )
+
+    mksquashfs "${root_dir}" "${img}" "${fs_opts[@]}" >/dev/null
+}
+
+# vim: ft=sh
--

1.8.1.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Quoted 11 lines

From: "Yann E. MORIN" <[email protected]>

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc Ryan Barnett <[email protected]>

---
 docs/manual/partition-layout.txt | 11 ++++++++++-
 fs/custom/fs/squashfs            | 19 +++++++++++++++++++
 2 files changed, 29 insertions(+), 1 deletion(-)
 create mode 100644 fs/custom/fs/squashfs

Quoted 14 lines

fs/custom/fs/squashfs new file +8 -0

new file mode 100644
index 0000000..132d3ef
@@ -0,0 +1,19 @@
+# Create a squashfs filesystem
+
+#-----------------------------------------------------------------------------
+DEPENDS+=( squashfs )
+
+#-----------------------------------------------------------------------------
+do_image() {
+    local root_dir="${1}"

fs/custom/fs/squashfs line 13

Quoted 5 lines

fs/custom/fs/squashfs new file +5 -0

+    local img="${2}"
+    local -a fs_opts
+
+    fs_opts+=( -no-progress -noappend )
+    [ -z "${squashfs_comp}"  ] || fs_opts+=( -comp "${squashfs_comp}" )

It just a question of taste, but I really do prefer to read something like:
[ -n "${squashfs_comp}" ] && fs_opts+=( -comp "${squashfs_comp}" ) But that's just a personal taste.