Skip to contentSkip to conversation

fs/ext2: misc enhancements (branch yem/fs-vol-id)

PATCH Superseded Yann E. MORIN 2 participants 8 messages Dec 4, 2014 – Dec 6, 2014 ext2 file system

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover fs/ext2: misc enhancements (branch yem/fs-vol-id)

Hello All!

This series enhances the fs/ext2 generation. Not much to say beside the individual commit logs... ;-)

Changes v1 -> v2:
- move our genext2fs.sh wrapper and make it a proper host package, with extensive tweaks to the code (Thomas)

Regards,
Yann E. MORIN.

The following changes since commit c5f0e93d2cc6107cce249e1f11158aeccb4a3449:

rp-pppoe: install default conf file (2014-12-03 23:26:21 +0100)

are available in the git repository at:

git://git.busybox.net/~ymorin/git/buildroot yem/fs-vol-id

for you to fetch changes up to d98afa0b717cf9585358913fd3d97ba6101d2a2b:

fs/ext2: add option to specify a filesystem label (2014-12-04 22:48:55 +0100)

----------------------------------------------------------------
Yann E. MORIN (3):
package/mke2img: new package
fs/ext2: use the new mke2img host package fs/ext2: add option to specify a filesystem label

 fs/ext2/Config.in          |   3 +
 fs/ext2/ext2.mk            |  17 +++---
 fs/ext2/genext2fs.sh       | 121 ------------------------------------
 package/mke2img/mke2img    | 148 +++++++++++++++++++++++++++++++++++++++++++++
 package/mke2img/mke2img.mk |  14 +++++
 5 files changed, 175 insertions(+), 128 deletions(-)
 delete mode 100755 fs/ext2/genext2fs.sh
 create mode 100755 package/mke2img/mke2img
 create mode 100644 package/mke2img/mke2img.mk
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH v2 1/3 package/mke2img: new package

Currently, we are using a shell script called genext2fs, that impersonates the real genext2fs. But that script does much more than just call genextfs: it also calls tune2fs and e2fsck.

Because it impersonates genext2fs, we can not easily add new options, and are constrained by the genext2fs options.

But it turns out that, of all the options supported by the real genext2fs, we only really care for a subset, namely:
- number of blocks
- number of inodes
- percentage of blocks reeserved to root
- the root directory which to generate the image from

So, we introduce a new host package, mke2img, that is intended to eventually replace genext2fs.sh.

This new script is highly modeled from the existing genext2fs.sh, but was slightly refreshed, and a new, supposedly sane set of options has been choosen for the features we need -b -i -r -d; a new option, -o, has been added to specify the output file, rather than rely on the ordering of arguments.

Since the upstream e2fsprogs are expected to release a new mke2fs that will be able to generate a filesystem image from a directory, we then will be able to replace all the logic in mke2img, to use mke2fs instead of the (relatively fragile) combination of the three tools we currently use.

Cc Thomas Petazzoni <[email protected]>

---
Note: there is no way to use that tool for now. There is no menuconfig option in "Host utilities", and nothing depends on it yet. In the following patches, we'll convert fs/ext2/ext2.mk over to using that tool, so it will be automatically enabled as a dependency of the ext2 filesystem type.

---
 package/mke2img/mke2img    | 148 +++++++++++++++++++++++++++++++++++++++++++++
 package/mke2img/mke2img.mk |  14 +++++
 2 files changed, 162 insertions(+)
 create mode 100755 package/mke2img/mke2img
 create mode 100644 package/mke2img/mke2img.mk
package/mke2img/mke2img new file +148 -0
new file mode 100755
index 0000000..98ac7e6
@@ -0,0 +1,148 @@
+#!/bin/bash
+
+# Buildroot wrapper to the collection of ext2/3/4 filesystem tools:
+# - genext2fs, to generate ext2 filesystem images
+# - tune2fs, to modify an ext2/3/4 filesystem (possibly in an image file)
+# - e2fsck, to check and fix an ext2/3/4 filesystem (possibly in an image file)
+
+set -e
+
+main() {
+    local OPT OPTARG
+    local nb_blocks nb_inodes nb_res_blocks root_dir image gen rev label uuid
+    local -a genext2fs_opts
+    local -a tune2fs_opts
+    local tune2fs_O_opts
+
+    while getopts :b:i:r:d:o:G:R:l:u: OPT; do
+        case "${OPT}" in
+        b)  nb_blocks=${OPTARG};;
+        i)  nb_inodes=${OPTARG};;
+        r)  nb_res_blocks=${OPTARG};;
+        d)  root_dir="${OPTARG}";;
+        o)  image="${OPTARG}";;
+        G)  gen=${OPTARG};;
+        R)  rev=${OPTARG};;
+        l)  label="${OPTARG}";;
+        u)  uuid="${OPTARG}";;
+        :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+        \?) error "unknown option '%s'\n" "${OPTARG}";;
+        esac
+    done
+
+    # Sanity checks
+    if [ -z "${root_dir}" ]; then
+        error "you must specify a root directory with '-d'\n"
+    fi
+    if [ -z "${image}" ]; then
+        error "you must specify an output image file with '-o'\n"
+    fi
+    case "${gen}:${rev}" in
+    2:0|2:1|3:1|4:1)    ;;
+    *)  error "unknown ext generation '%s' and/or revision '%s'\n" \
+               "${gen}" "${rev}"
+        ;;
+    esac
+
+    # calculate needed inodes
+    if [ -z "${nb_inodes}" ]; then
+        nb_inodes=$(find "${root_dir}" | wc -l)
+        nb_inodes=$((nb_inodes+400))
+    fi
+
+    # calculate needed blocks
+    if [ -z "${nb_blocks}" ]; then
+        # size ~= superblock, block+inode bitmaps, inodes (8 per block),
+        # blocks; we scale inodes / blocks with 10% to compensate for
+        # bitmaps size + slack
+        nb_blocks=$(du -s -k "${root_dir}" |sed -r -e 's/[[:space:]]+.*$//')
+        nb_blocks=$((500+(nb_blocks+nb_inodes/8)*11/10))
+    fi
+
+    # Upgrade to rev1 if needed
+    if [ ${rev} -ge 1 ]; then
+        tune2fs_O_opts+=",filetype"
+    fi
+
+    # Add a journal for ext3 and above
+    if [ ${gen} -ge 3 ]; then
+        tune2fs_opts+=( -j -J size=1 )
+        # we add 1300 blocks (a bit more than 1 MiB, assuming 1KiB blocks)
+        # for the journal
+        # Note: I came to 1300 blocks after trial-and-error checks. YMMV.
+        nb_blocks=$((nb_blocks+1300))
+    fi
+
+    # Add ext4 specific features
+    if [ ${gen} -ge 4 ]; then
+        tune2fs_O_opts+=",extents,uninit_bg,dir_index"
+    fi
+
+    # Add our -O options (there will be at most one leading comma, remove it)
+    if [ -n "${tune2fs_O_opts}" ]; then
+        tune2fs_opts+=( -O "${tune2fs_O_opts#,}" )
+    fi
+
+    # Add the label if specified
+    if [ -n "${label}" ]; then
+        tune2fs_opts+=( -L "${label}" )
+    fi
+
+    # Generate the filesystem
+    genext2fs_opts=( -b ${nb_blocks} -N ${nb_inodes} -d "${root_dir}" )
+    if [ -n "${nb_res_blocks}" ]; then
+        genext2fs_opts+=( -m ${nb_res_blocks} )
+    fi
+    genext2fs "${genext2fs_opts[@]}" "${image}"
+
+    # genext2fs does not generate a UUID, but fsck will whine if one is
+    # is missing, so we need to add a UUID.
+    # Of course, this has to happend _before_ we run fsck.
+    # Also, some ext4 metadata are based on the UUID, so we must
+    # set it before we can convert the filesystem to ext4.
+    # If the user did not specify a UUID, we generate a random one.
+    # Although a random UUID may seem bad for reproducibility, there
+    # already are so many things that are not reproducible in a
+    # filesystem: file dates, file ordering, content of the files...
+    tune2fs -U "${uuid:-random}" "${image}"
+
+    # Upgrade the filesystem
+    if [ ${#tune2fs_opts[@]} -ne 0 ]; then
+        tune2fs "${tune2fs_opts[@]}" "${image}"
+    fi
+
+    # After changing filesystem options, running fsck is required
+    # (see: man tune2fs). Running e2fsck in other cases will ensure
+    # coherency of the filesystem, although it is not required.
+    # 'e2fsck -pDf' means:
+    #  - automatically repair
+    #  - optimise and check for duplicate entries
+    #  - force checking
+    # Sending output to oblivion, as e2fsck can be *very* verbose,
+    # especially with filesystems generated by genext2fs.
+    # Exit codes 1 & 2 are OK, it means fs errors were successfully
+    # corrected, hence our little trick with $ret.
+    ret=0
+    e2fsck -pDf "${image}" >/dev/null || ret=$?
+    case ${ret} in
+       0|1|2) ;;
+       *)   errorN ${ret} "failed to run e2fsck on '%s' (ext%d)\n" \
+                   "${image}" ${gen}
+    esac
+    printf "\n"
+    trace "e2fsck was successfully run on '%s' (ext%d)\n" "${image}" ${gen}
+    printf "\n"
+
+    # Remove count- and time-based checks, they are not welcome
+    # on embedded devices, where they can cause serious boot-time
+    # issues by tremendously slowing down the boot.
+    tune2fs -c 0 -i 0 "${image}"
+}
+
+trace()  { local msg="${1}"; shift; printf "%s: ${msg}" "${my_name}" "${@}"; }
+warn()   { trace "${@}" >&2; }
+errorN() { local ret="${1}"; shift; warn "${@}"; exit ${ret}; }
+error()  { errorN 1 "${@}"; }
+
+my_name="${0##*/}"
+main "$@"
package/mke2img/mke2img.mk new file +14 -0
new file mode 100644
index 0000000..04aaa8f
@@ -0,0 +1,14 @@
+################################################################################
+#
+# mke2img
+#
+################################################################################
+
+HOST_MKE2IMG_SOURCE =
+HOST_MKE2IMG_DEPENDENCIES = host-genext2fs host-e2fsprogs
+
+define HOST_MKE2IMG_INSTALL_CMDS
+	$(INSTALL) -D -m 0755 package/mke2img/mke2img $(HOST_DIR)/usr/bin/mke2img
+endef
+
+$(eval $(host-generic-package))
Signature · 1 line
1.9.1
#2 lore raw

PATCH v2 1/3 package/mke2img: new package

Hi Yann, Thomas, list,

Quoted 3 lines
Cc Thomas Petazzoni <[email protected]>

---

Quoted 6 lines

Note: there is no way to use that tool for now. There is no menuconfig option in "Host utilities", and nothing depends on it yet. In the following patches, we'll convert fs/ext2/ext2.mk over to using that tool, so it will be automatically enabled as a dependency of the ext2 filesystem type.

Tested the whole series of 3 together, successfully built different rootfs.ext images and booted them.
Tested with master branch, x86_64 arch in QEMU, internal toolchain GCC 4.9.2 w/ uclibc, Linux 3.17.4.

Regards,
Karoly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20141206/5cb49738/attachment.html>

#3 lore raw

PATCH v2 2/3 fs/ext2: use the new mke2img host package

In reply to Yann E. MORIN

And get rid of the old genext2fs.sh wrapper.

Cc Thomas Petazzoni <[email protected]>
---
 fs/ext2/ext2.mk      |  13 +++---
 fs/ext2/genext2fs.sh | 121 ---------------------------------------------------
 2 files changed, 5 insertions(+), 129 deletions(-)
 delete mode 100755 fs/ext2/genext2fs.sh
fs/ext2/ext2.mk +5 -8
index 425f7bf..96b7a4a 100644
@@ -4,27 +4,24 @@
 #
 ################################################################################
 
-EXT2_OPTS :=
+EXT2_OPTS = -G $(BR2_TARGET_ROOTFS_EXT2_GEN) -R $(BR2_TARGET_ROOTFS_EXT2_REV)
 
 ifneq ($(strip $(BR2_TARGET_ROOTFS_EXT2_BLOCKS)),0)
 EXT2_OPTS += -b $(BR2_TARGET_ROOTFS_EXT2_BLOCKS)
 endif
 
 ifneq ($(strip $(BR2_TARGET_ROOTFS_EXT2_INODES)),0)
-EXT2_OPTS += -N $(BR2_TARGET_ROOTFS_EXT2_INODES)
+EXT2_OPTS += -i $(BR2_TARGET_ROOTFS_EXT2_INODES)
 endif
 
 ifneq ($(strip $(BR2_TARGET_ROOTFS_EXT2_RESBLKS)),0)
-EXT2_OPTS += -m $(BR2_TARGET_ROOTFS_EXT2_RESBLKS)
+EXT2_OPTS += -r $(BR2_TARGET_ROOTFS_EXT2_RESBLKS)
 endif
 
-ROOTFS_EXT2_DEPENDENCIES = host-genext2fs host-e2fsprogs
-
-EXT2_ENV = GEN=$(BR2_TARGET_ROOTFS_EXT2_GEN)
-EXT2_ENV += REV=$(BR2_TARGET_ROOTFS_EXT2_REV)
+ROOTFS_EXT2_DEPENDENCIES = host-mke2img
 
 define ROOTFS_EXT2_CMD
-	PATH=$(BR_PATH) $(EXT2_ENV) fs/ext2/genext2fs.sh -d $(TARGET_DIR) $(EXT2_OPTS) $@
+	PATH=$(BR_PATH) mke2img -d $(TARGET_DIR) $(EXT2_OPTS) -o $@
 endef
 
 rootfs-ext2-symlink:
fs/ext2/genext2fs.sh deleted +0 -121

121 lines not on this page · raw message

Signature · 1 line
1.9.1
#4 lore raw

PATCH v2 2/3 fs/ext2: use the new mke2img host package

Hi Yann, Thomas, list,

Quoted 2 lines
Cc Thomas Petazzoni <[email protected]>

Tested the whole series of 3 together, successfully built different rootfs.ext images and booted them.
Tested with master branch, x86_64 arch in QEMU, internal toolchain GCC 4.9.2 w/ uclibc, Linux 3.17.4.

Regards,
Karoly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20141206/25444418/attachment.html>

#5 lore raw

PATCH v2 3/3 fs/ext2: add option to specify a filesystem label

In reply to Yann E. MORIN

Filesystems of the ext familly can carry a filesystem label. Add an option for the user to specify such a label.

Cc Thomas Petazzoni <[email protected]>
---
 fs/ext2/Config.in | 3 +++
 fs/ext2/ext2.mk   | 6 ++++++
 2 files changed, 9 insertions(+)
fs/ext2/Config.in +3 -0
index adba6f3..65828f2 100644
@@ -42,6 +42,9 @@ config BR2_TARGET_ROOTFS_EXT2_REV
 	default 0   if BR2_TARGET_ROOTFS_EXT2_2r0
 	default 1   if !BR2_TARGET_ROOTFS_EXT2_2r0
 
+config BR2_TARGET_ROOTFS_EXT2_LABEL
+	string "filesystem label"
+
 config BR2_TARGET_ROOTFS_EXT2_BLOCKS
 	int "size in blocks (leave at 0 for auto calculation)"
 	default 0
fs/ext2/ext2.mk +6 -0
index 96b7a4a..758b967 100644
@@ -18,6 +18,12 @@ ifneq ($(strip $(BR2_TARGET_ROOTFS_EXT2_RESBLKS)),0)
 EXT2_OPTS += -r $(BR2_TARGET_ROOTFS_EXT2_RESBLKS)
 endif
 
+# Not qstrip-ing the variable, because it may contain spaces,
+# but we must qstrip it when checking.
+ifneq ($(call qstrip,$(BR2_TARGET_ROOTFS_EXT2_LABEL)),)
+EXT2_OPTS += -l $(BR2_TARGET_ROOTFS_EXT2_LABEL)
+endif
+
 ROOTFS_EXT2_DEPENDENCIES = host-mke2img
 
 define ROOTFS_EXT2_CMD
Signature · 1 line
1.9.1
#6 lore raw

PATCH v2 3/3 fs/ext2: add option to specify a filesystem label

Hi Yann, Thomas, list,

--- a/fs/ext2/ext2.mk

fs/ext2/ext2.mk ifneq ($(strip $(BR2_TARGET_ROOTFS_EXT2_RESBLKS)),0) · line 25

Quoted 10 lines

+++ b/fs/ext2/ext2.mk

fs/ext2/ext2.mk +4 -0
@@ -18,6 +18,12 @@ ifneq ($(strip $(BR2_TARGET_ROOTFS_EXT2_RESBLKS)),0)
 EXT2_OPTS += -r $(BR2_TARGET_ROOTFS_EXT2_RESBLKS)
 endif
 
+# Not qstrip-ing the variable, because it may contain spaces,
+# but we must qstrip it when checking.
+ifneq ($(call qstrip,$(BR2_TARGET_ROOTFS_EXT2_LABEL)),)
+EXT2_OPTS += -l $(BR2_TARGET_ROOTFS_EXT2_LABEL)
 

Unfortunately this won't work with labels with spaces in them. I recommend using escaped quotation marks like this:

+EXT2_OPTS += -l \"$(BR2_TARGET_ROOTFS_EXT2_LABEL)\"

Quoted 1 line

---

Tested the whole series of 3 together, successfully built different rootfs.ext images and booted them.
Tested with master branch, x86_64 arch in QEMU, internal toolchain GCC 4.9.2 w/ uclibc, Linux 3.17.4.

Regards,
Karoly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20141206/7beb7270/attachment.html>

#7 lore raw

PATCH v2 3/3 fs/ext2: add option to specify a filesystem label

K?roly, All,

On 2014-12-06 12:59 +0100, K?roly Kasza spake thusly:

Quoted 15 lines

--- a/fs/ext2/ext2.mk

fs/ext2/ext2.mk ifneq ($(strip $(BR2_TARGET_ROOTFS_EXT2_RESBLKS)),0) · line 25

Quoted 10 lines

+++ b/fs/ext2/ext2.mk

fs/ext2/ext2.mk +4 -0
@@ -18,6 +18,12 @@ ifneq ($(strip $(BR2_TARGET_ROOTFS_EXT2_RESBLKS)),0)
 EXT2_OPTS += -r $(BR2_TARGET_ROOTFS_EXT2_RESBLKS)
 endif
 
+# Not qstrip-ing the variable, because it may contain spaces,
+# but we must qstrip it when checking.
+ifneq ($(call qstrip,$(BR2_TARGET_ROOTFS_EXT2_LABEL)),)
+EXT2_OPTS += -l $(BR2_TARGET_ROOTFS_EXT2_LABEL)
 

Unfortunately this won't work with labels with spaces in them. I recommend using escaped quotation marks like this:

Quoted 1 line

+EXT2_OPTS += -l \"$(BR2_TARGET_ROOTFS_EXT2_LABEL)\"

Damit... Right, I'll fix and respin.

Thanks! :-)

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'