Skip to contentSkip to conversation

Fully support ext3/ext4 rootfs generation

RFC PATCH In review Robert Yang 3 participants 11 messages Mar 2, 2013 – Mar 5, 2013

lore
Share & tools
lore
#0 lore raw

cover Fully support ext3/ext4 rootfs generation

Replace genext2fs with populate-extfs.sh in image_types.bbclass to fully support ext3/ext4 rootfs.

Comments from Darren:
We used genext2fs to create ext3/ext4 rootfs in the past, basically, genext2fs doesn't support creating ext4 filesystems. It creates, as I understand it, an ext2 filesystem, then adds a journal, and sets some bits. It can't support the newer features like extents. So what we end up with is a bit of a hack for a filesystem.

The ext tools (e2fsprogs) unfortunately don't provide an integrated solution for generating prepopulated filesystem images as many other mkfs* tools do. One thing missing was symlink support in libext2fs. I added that support and demonstrated a script which uses the e2fsprogs debugfs tool that can populate the newly formatted filesystem from a directory and without root privileges.

This patches integrate this stage of development into OE-Core. We can go about this in two ways. One is to just prototype this in a branch and use it to validate the functionality and not make any changes to oe-core image generation until mke2fs has initial directory support. The other is to merge this and get broader testing of the concept and later move to the full mke2fs implementation once it becomes available. I understand the resistance to the latter, but long term I think it will result in a more robust solution as we will have caught more of the corner cases and have been able to do a better job integrating into mke2fs the first time.

* Impact:
+ Rootfs generation time:
- For a core-image-minimal image, about more 3 seconds are needed
- For a core-image-sato image, about more 15 seconds are needed

+ Disk space usage:
- Nearly no changes:
$ ls -lh BEFORE.rootfs.ext3 AFTER.rootfs.ext3 | awk '{print $5"\t"$NF}' 357M BEFORE.rootfs.ext3
357M AFTER.rootfs.ext3

$ du -sh BEFORE.rootfs.ext3 AFTER.rootfs.ext3
238M BEFORE.rootfs.ext3
357M AFTER.rootfs.ext3
# This is different because BEFORE.rootfs.ext3 has sparse files, # they are very similar (less than 1M gap) after mount them and run # "du -sh".

// Robert

The following changes since commit 8264863ea0674f6cb105f5f7301861408fdc409b:

Add KERNEL_EXTRA_ARGS parameter (2013-03-01 14:57:52 +0000)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib robert/e2fsprogs
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/e2fsprogs

Robert Yang (2):
e2fsprogs: add populate-extfs.sh image_types.bbclass: replace genext2fs with populate-extfs.sh

 meta/classes/image_types.bbclass                   | 45 ++++++--------
 .../e2fsprogs/e2fsprogs-1.42.7/populate-extfs.sh   | 69 ++++++++++++++++++++++
 .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.7.bb |  2 +
 3 files changed, 88 insertions(+), 28 deletions(-)
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.7/populate-extfs.sh
Signature · 1 line
1.7.11.2
#1 lore raw

RFC PATCH 1/2 e2fsprogs: add populate-extfs.sh

This script is originally from Darren Hart, it will be used for creating the ext* filesystem from a given directory, which will replace the genext2fs in image_types.bbclass at the moment, we may use the mke2fs to replace this script again when it has the initial directory support.

Changes of the script:
* Rename it from mkdebugfs.sh to populate-extfs.sh
* Add a simple usage
* Add checking for the number of the parameters
* Add the "regular empty file" and "fifo" file type
* Set mode, uid and gid for the file
* Save the command lines to a file and batch run them
* Change the error message
* Improve the performance

[YOCTO #3848]

---
 .../e2fsprogs/e2fsprogs-1.42.7/populate-extfs.sh   | 69 ++++++++++++++++++++++
 .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.7.bb |  2 +
 2 files changed, 71 insertions(+)
 create mode 100644 meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.7/populate-extfs.sh
meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.7/populate-extfs.sh new file +69 -0
new file mode 100644
index 0000000..638bd74
@@ -0,0 +1,69 @@
+#!/bin/sh
+
+do_usage () {
+	cat << _EOF
+Usage: populate-extfs.sh <source> <device>
+Create an ext2/ext3/ext4 filesystem from a directory or file
+
+  source: The source directory or file
+  device: The target device
+
+_EOF
+	exit 1
+}
+
+[ $# -ne 2 ] && do_usage
+
+SRCDIR=$1
+DEVICE=$2
+DEBUGFS="debugfs"
+
+{
+	CWD="/"
+	find $SRCDIR | while read FILE; do
+                TGT="${FILE##*/}"
+                DIR="${FILE#$SRCDIR}"
+                DIR="${DIR%$TGT}"
+
+		# Skip the root dir
+		[ ! -z "$TGT" ] || continue
+
+		if [ "$DIR" != "$CWD" ]; then
+			echo "cd $DIR"
+			CWD="$DIR"
+		fi
+
+		# Only stat once since stat is a time consuming command
+		STAT=$(stat -c "TYPE=\"%F\";DEVNO=\"%t %T\";MODE=\"%f\";U=\"%u\";G=\"%g\"" $FILE)
+		eval $STAT
+
+		case $TYPE in
+		"directory")
+			echo "mkdir $TGT"
+			;;
+		"regular file" | "regular empty file")
+			echo "write $FILE $TGT"
+			;;
+		"symbolic link")
+			LINK_TGT=$(readlink $FILE)
+			echo "symlink $TGT $LINK_TGT"
+			;;
+		"block special file" | "character special file")
+			echo "mknod $TGT b $DEVNO"
+			;;
+		"fifo")
+			echo "mknod $TGT p"
+			;;
+		*)
+			echo "Unknown/unhandled file type '$TYPE' file: $FILE" 1>&2
+			;;
+		esac
+
+		# Set the file mode
+		echo "sif $TGT mode 0x$MODE"
+
+		# Set uid and gid
+		echo "sif $TGT uid $U"
+		echo "sif $TGT gid $G"
+	done
+} | $DEBUGFS -w -f - $DEVICE
meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.7.bb +2 -0
index 9e22563..9b65184 100644
@@ -4,6 +4,7 @@ PR = "r0"
 
 SRC_URI += "file://acinclude.m4 \
             file://remove.ldconfig.call.patch \
+            file://populate-extfs.sh \
 "
 
 SRC_URI[md5sum] = "a1ec22ef003688dae9f76c74881b22b9"
@@ -41,6 +42,7 @@ do_install_append () {
 		mv ${D}${base_libdir}/e2initrd_helper ${D}${libdir}
 		mv ${D}${base_libdir}/pkgconfig ${D}${libdir}
 	fi
+	install -m 0755 ${WORKDIR}/populate-extfs.sh ${D}${bindir}
 }
 
 RDEPENDS_e2fsprogs = "e2fsprogs-badblocks"
Signature · 1 line
1.7.11.2
#2 lore raw

RFC PATCH 2/2 image_types.bbclass: replace genext2fs with populate-extfs.sh

In reply to Robert Yang

Comments from Darren Hart:
Basically, genext2fs doesn't support creating ext4 filesystems. It creates, as I understand it, an ext2 filesystem, then adds a journal, and sets some bits. It can't support the newer features like extents. So what we end up with is a bit of a hack for a filesystem.

The ext tools (e2fsprogs) unfortunately don't provide an integrated solution for generating prepopulated filesystem images as many other mkfs* tools do. One thing missing was symlink support in libext2fs. I added that support and demonstrated a script which uses the e2fsprogs debugfs tool that can populate the newly formatted filesystem from a directory and without root privileges.

This patches integrate this stage of development into OE-Core. We can go about this in two ways. One is to just prototype this in a branch and use it to validate the functionality and not make any changes to oe-core image generation until mke2fs has initial directory support. The other is to merge this and get broader testing of the concept and later move to the full mke2fs implementation once it becomes available. I understand the resistance to the latter, but long term I think it will result in a more robust solution as we will have caught more of the corner cases and have been able to do a better job integrating into mke2fs the first time.

[YOCTO #3848]

---
 meta/classes/image_types.bbclass | 45 +++++++++++++++-------------------------
 1 file changed, 17 insertions(+), 28 deletions(-)
meta/classes/image_types.bbclass +17 -28
index 6bb113d..7d543b3 100644
@@ -135,34 +135,23 @@ IMAGE_CMD_sum.jffs2 = "${IMAGE_CMD_jffs2} && sumtool -i ${DEPLOY_DIR_IMAGE}/${IM
 
 IMAGE_CMD_cramfs = "mkcramfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.cramfs ${EXTRA_IMAGECMD}"
 
-IMAGE_CMD_ext2 () {
-	rm -rf ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN} && mkdir ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN}
-	genext2fs -b $ROOTFS_SIZE -d ${IMAGE_ROOTFS} ${EXTRA_IMAGECMD} ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN}/${IMAGE_NAME}.rootfs.ext2
-	mv ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN}/${IMAGE_NAME}.rootfs.ext2 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext2
-	rmdir ${DEPLOY_DIR_IMAGE}/tmp.gz-${PN}
-}
+oe_mkext234fs () {
+	fstype=$1
+	extra_imagecmd=""
 
-IMAGE_CMD_ext3 () {
-	genext2fs -b $ROOTFS_SIZE -d ${IMAGE_ROOTFS} ${EXTRA_IMAGECMD} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext3
-	tune2fs -j ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext3
-}
+	if [ $# -gt 1 ]; then
+		shift
+		extra_imagecmd=$@
+	fi
 
-oe_mkext4fs () {
-	genext2fs -b $ROOTFS_SIZE -d ${IMAGE_ROOTFS} ${EXTRA_IMAGECMD} $1
-	tune2fs -O extents,uninit_bg,dir_index,has_journal $1
-	e2fsck -yfDC0 $1 || chk=$?
-	case $chk in
-	0|1|2)
-	    ;;
-	*)
-	    return $chk
-	    ;;
-	esac
+	dd if=/dev/zero of=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.$fstype count=$ROOTFS_SIZE bs=1k
+	yes | mkfs.$fstype $extra_imagecmd ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.$fstype
+	populate-extfs.sh ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.$fstype
 }
 
-IMAGE_CMD_ext4 () {
-	oe_mkext4fs ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext4
-}
+IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}"
+IMAGE_CMD_ext3 = "oe_mkext234fs ext3 ${EXTRA_IMAGECMD}"
+IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}"
 
 IMAGE_CMD_btrfs () {
 	mkfs.btrfs -b `expr ${ROOTFS_SIZE} \* 1024` ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.btrfs
@@ -212,7 +201,7 @@ JFFS2_ENDIANNESS ?= "${@base_conditional('SITEINFO_ENDIANNESS', 'le', '--little-
 JFFS2_ERASEBLOCK ?= "0x40000"
 EXTRA_IMAGECMD_jffs2 ?= "--pad ${JFFS2_ENDIANNESS} --eraseblock=${JFFS2_ERASEBLOCK} --no-cleanmarkers"
 
-# Change these if you want default genext2fs behavior (i.e. create minimal inode number)
+# Change these if you want default mkfs behavior (i.e. create minimal inode number)
 EXTRA_IMAGECMD_ext2 ?= "-i 8192"
 EXTRA_IMAGECMD_ext3 ?= "-i 8192"
 EXTRA_IMAGECMD_ext4 ?= "-i 8192"
@@ -223,9 +212,9 @@ IMAGE_DEPENDS = ""
 IMAGE_DEPENDS_jffs2 = "mtd-utils-native"
 IMAGE_DEPENDS_sum.jffs2 = "mtd-utils-native"
 IMAGE_DEPENDS_cramfs = "cramfs-native"
-IMAGE_DEPENDS_ext2 = "genext2fs-native"
-IMAGE_DEPENDS_ext3 = "genext2fs-native e2fsprogs-native"
-IMAGE_DEPENDS_ext4 = "genext2fs-native e2fsprogs-native"
+IMAGE_DEPENDS_ext2 = "e2fsprogs-native"
+IMAGE_DEPENDS_ext3 = "e2fsprogs-native"
+IMAGE_DEPENDS_ext4 = "e2fsprogs-native"
 IMAGE_DEPENDS_btrfs = "btrfs-tools-native"
 IMAGE_DEPENDS_squashfs = "squashfs-tools-native"
 IMAGE_DEPENDS_squashfs-lzma = "squashfs-tools-native"
Signature · 1 line
1.7.11.2
#3 lore raw
In reply to Robert Yang
Quoted 40 lines · Robert Yang

Replace genext2fs with populate-extfs.sh in image_types.bbclass to fully support ext3/ext4 rootfs.

Comments from Darren:
We used genext2fs to create ext3/ext4 rootfs in the past, basically, genext2fs doesn't support creating ext4 filesystems. It creates, as I understand it, an ext2 filesystem, then adds a journal, and sets some bits. It can't support the newer features like extents. So what we end up with is a bit of a hack for a filesystem.

The ext tools (e2fsprogs) unfortunately don't provide an integrated solution for generating prepopulated filesystem images as many other mkfs* tools do. One thing missing was symlink support in libext2fs. I added that support and demonstrated a script which uses the e2fsprogs debugfs tool that can populate the newly formatted filesystem from a directory and without root privileges.

This patches integrate this stage of development into OE-Core. We can go about this in two ways. One is to just prototype this in a branch and use it to validate the functionality and not make any changes to oe-core image generation until mke2fs has initial directory support. The other is to merge this and get broader testing of the concept and later move to the full mke2fs implementation once it becomes available. I understand the resistance to the latter, but long term I think it will result in a more robust solution as we will have caught more of the corner cases and have been able to do a better job integrating into mke2fs the first time.

* Impact:
+ Rootfs generation time:
- For a core-image-minimal image, about more 3 seconds are needed
- For a core-image-sato image, about more 15 seconds are needed

+ Disk space usage:
- Nearly no changes:
$ ls -lh BEFORE.rootfs.ext3 AFTER.rootfs.ext3 | awk '{print $5"\t"$NF}' 357M BEFORE.rootfs.ext3
357M AFTER.rootfs.ext3

Quoted 6 lines

$ du -sh BEFORE.rootfs.ext3 AFTER.rootfs.ext3
238M BEFORE.rootfs.ext3
357M AFTER.rootfs.ext3
# This is different because BEFORE.rootfs.ext3 has sparse files, # they are very similar (less than 1M gap) after mount them and run # "du -sh".

I can live with the performance issues however as I understand this code, its breaking both sparse files and also likely hardlinked files. I'm not sure we have many sparse ones but we do have packages with heavy hardlinking (the sdk image toolchain packages for example).

Adding the script to e2fsprogs isn't a problem but I am tempted to wait until this work is completed before we start using it instead of genext2fs.

Cheers,

Richard

#4 lore raw
Quoted 48 lines · Richard Purdie
Quoted 40 lines · Robert Yang

Replace genext2fs with populate-extfs.sh in image_types.bbclass to fully support ext3/ext4 rootfs.

Comments from Darren:
We used genext2fs to create ext3/ext4 rootfs in the past, basically, genext2fs doesn't support creating ext4 filesystems. It creates, as I understand it, an ext2 filesystem, then adds a journal, and sets some bits. It can't support the newer features like extents. So what we end up with is a bit of a hack for a filesystem.

The ext tools (e2fsprogs) unfortunately don't provide an integrated solution for generating prepopulated filesystem images as many other mkfs* tools do. One thing missing was symlink support in libext2fs. I added that support and demonstrated a script which uses the e2fsprogs debugfs tool that can populate the newly formatted filesystem from a directory and without root privileges.

This patches integrate this stage of development into OE-Core. We can go about this in two ways. One is to just prototype this in a branch and use it to validate the functionality and not make any changes to oe-core image generation until mke2fs has initial directory support. The other is to merge this and get broader testing of the concept and later move to the full mke2fs implementation once it becomes available. I understand the resistance to the latter, but long term I think it will result in a more robust solution as we will have caught more of the corner cases and have been able to do a better job integrating into mke2fs the first time.

* Impact:
+ Rootfs generation time:
- For a core-image-minimal image, about more 3 seconds are needed
- For a core-image-sato image, about more 15 seconds are needed

+ Disk space usage:
- Nearly no changes:
$ ls -lh BEFORE.rootfs.ext3 AFTER.rootfs.ext3 | awk '{print $5"\t"$NF}' 357M BEFORE.rootfs.ext3
357M AFTER.rootfs.ext3

Quoted 6 lines

$ du -sh BEFORE.rootfs.ext3 AFTER.rootfs.ext3
238M BEFORE.rootfs.ext3
357M AFTER.rootfs.ext3
# This is different because BEFORE.rootfs.ext3 has sparse files, # they are very similar (less than 1M gap) after mount them and run # "du -sh".

Quoted 8 lines

I can live with the performance issues however as I understand this code, its breaking both sparse files and also likely hardlinked files. I'm not sure we have many sparse ones but we do have packages with heavy hardlinking (the sdk image toolchain packages for example).

Adding the script to e2fsprogs isn't a problem but I am tempted to wait until this work is completed before we start using it instead of genext2fs.

Agreed, I thought we were within 1MB on size from my reading of the previous discussion. Robert, do you have any thoughts on what is needed to address the hardlinks? That seems like something we should be able to address quickly.

Same question for the sparse files, although I could see that taking some additional effort.

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Technical Lead - Linux Kernel
#5 lore raw
Quoted 61 lines · Darren Hart
Quoted 48 lines · Richard Purdie
Quoted 40 lines · Robert Yang

Replace genext2fs with populate-extfs.sh in image_types.bbclass to fully support ext3/ext4 rootfs.

Comments from Darren:
We used genext2fs to create ext3/ext4 rootfs in the past, basically, genext2fs doesn't support creating ext4 filesystems. It creates, as I understand it, an ext2 filesystem, then adds a journal, and sets some bits. It can't support the newer features like extents. So what we end up with is a bit of a hack for a filesystem.

The ext tools (e2fsprogs) unfortunately don't provide an integrated solution for generating prepopulated filesystem images as many other mkfs* tools do. One thing missing was symlink support in libext2fs. I added that support and demonstrated a script which uses the e2fsprogs debugfs tool that can populate the newly formatted filesystem from a directory and without root privileges.

This patches integrate this stage of development into OE-Core. We can go about this in two ways. One is to just prototype this in a branch and use it to validate the functionality and not make any changes to oe-core image generation until mke2fs has initial directory support. The other is to merge this and get broader testing of the concept and later move to the full mke2fs implementation once it becomes available. I understand the resistance to the latter, but long term I think it will result in a more robust solution as we will have caught more of the corner cases and have been able to do a better job integrating into mke2fs the first time.

* Impact:
+ Rootfs generation time:
- For a core-image-minimal image, about more 3 seconds are needed
- For a core-image-sato image, about more 15 seconds are needed

+ Disk space usage:
- Nearly no changes:
$ ls -lh BEFORE.rootfs.ext3 AFTER.rootfs.ext3 | awk '{print $5"\t"$NF}' 357M BEFORE.rootfs.ext3
357M AFTER.rootfs.ext3

Quoted 6 lines

$ du -sh BEFORE.rootfs.ext3 AFTER.rootfs.ext3
238M BEFORE.rootfs.ext3
357M AFTER.rootfs.ext3
# This is different because BEFORE.rootfs.ext3 has sparse files, # they are very similar (less than 1M gap) after mount them and run # "du -sh".

Quoted 8 lines

I can live with the performance issues however as I understand this code, its breaking both sparse files and also likely hardlinked files. I'm not sure we have many sparse ones but we do have packages with heavy hardlinking (the sdk image toolchain packages for example).

Adding the script to e2fsprogs isn't a problem but I am tempted to wait until this work is completed before we start using it instead of genext2fs.

Quoted 7 lines

Agreed, I thought we were within 1MB on size from my reading of the previous discussion. Robert, do you have any thoughts on what is needed to address the hardlinks? That seems like something we should be able to address quickly.

Same question for the sparse files, although I could see that taking some additional effort.

By the way, this is exactly the kind of thing we were hoping to catch by using the debugfs as an intermediate step. If anything needs to be added to libext2fs, we want to know before we jump into the mke2fs solution. So this is working as planned - just maybe sooner than expected, which is great.

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Technical Lead - Linux Kernel
#6 lore raw
Quoted 59 lines · Darren Hart
Quoted 56 lines · Darren Hart
Quoted 53 lines · Richard Purdie
Quoted 40 lines · Robert Yang

Replace genext2fs with populate-extfs.sh in image_types.bbclass to fully support ext3/ext4 rootfs.

Comments from Darren:
We used genext2fs to create ext3/ext4 rootfs in the past, basically, genext2fs doesn't support creating ext4 filesystems. It creates, as I understand it, an ext2 filesystem, then adds a journal, and sets some bits. It can't support the newer features like extents. So what we end up with is a bit of a hack for a filesystem.

The ext tools (e2fsprogs) unfortunately don't provide an integrated solution for generating prepopulated filesystem images as many other mkfs* tools do. One thing missing was symlink support in libext2fs. I added that support and demonstrated a script which uses the e2fsprogs debugfs tool that can populate the newly formatted filesystem from a directory and without root privileges.

This patches integrate this stage of development into OE-Core. We can go about this in two ways. One is to just prototype this in a branch and use it to validate the functionality and not make any changes to oe-core image generation until mke2fs has initial directory support. The other is to merge this and get broader testing of the concept and later move to the full mke2fs implementation once it becomes available. I understand the resistance to the latter, but long term I think it will result in a more robust solution as we will have caught more of the corner cases and have been able to do a better job integrating into mke2fs the first time.

* Impact:
+ Rootfs generation time:
- For a core-image-minimal image, about more 3 seconds are needed
- For a core-image-sato image, about more 15 seconds are needed

+ Disk space usage:
- Nearly no changes:
$ ls -lh BEFORE.rootfs.ext3 AFTER.rootfs.ext3 | awk '{print $5"\t"$NF}' 357M BEFORE.rootfs.ext3
357M AFTER.rootfs.ext3

Quoted 6 lines

$ du -sh BEFORE.rootfs.ext3 AFTER.rootfs.ext3
238M BEFORE.rootfs.ext3
357M AFTER.rootfs.ext3
# This is different because BEFORE.rootfs.ext3 has sparse files, # they are very similar (less than 1M gap) after mount them and run # "du -sh".

I can live with the performance issues however as I understand this code, its breaking both sparse files and also likely hardlinked files. I'm not sure we have many sparse ones but we do have packages with heavy hardlinking (the sdk image toolchain packages for example).

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

Adding the script to e2fsprogs isn't a problem but I am tempted to wait until this work is completed before we start using it instead of genext2fs.

Agreed, I thought we were within 1MB on size from my reading of the previous discussion. Robert, do you have any thoughts on what is needed to address the hardlinks? That seems like something we should be able to

Yes, we do have hard links in the rootfs, e.g: core-image-sato and core-image-sato-sdk:

# The sato

$ find core-image-sato/1.0-r0/rootfs/ -type f -printf "%p %i %n\n" | grep -v '1$'
core-image-sato/1.0-r0/rootfs/etc/terminfo/v/vt200 26977609 2
core-image-sato/1.0-r0/rootfs/etc/terminfo/v/vt220 26977609 2
core-image-sato/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld.bfd 26978690 2
core-image-sato/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld 26978690 2

# The sato-sdk:

$ find core-image-sato-sdk/1.0-r0/rootfs/ -type f -printf "%p %i %n\n" | grep -v '1$'
core-image-sato-sdk/1.0-r0/rootfs/etc/terminfo/v/vt200 26903519 2
core-image-sato-sdk/1.0-r0/rootfs/etc/terminfo/v/vt220 26903519 2
core-image-sato-sdk/1.0-r0/rootfs/usr/include/et/com_err.h 28058656 2
core-image-sato-sdk/1.0-r0/rootfs/usr/include/com_err.h 28058656 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld.bfd 26900939 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/perlbug 26899680 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/gawk-4.0.1 26900368 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/psed 26899667 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/s2p 26899667 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/c2ph 26899695 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/perlthanks 26899680 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/pstruct 26899695 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld 26900939 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/gawk 26900368 2

We can make hard links according to these information, and the find command is very fast, so it would not cost much time.

Quoted 5 lines
Quoted 4 lines

address quickly.

Same question for the sparse files, although I could see that taking some additional effort.

I don't find any obvious sparse files by the following command:

# sato:

$ find core-image-sato/1.0-r0/rootfs/ -size +8 -printf "%p #%S\n" | grep -v '#0\|#1' | wc -l
0

# sato-sdk:

$ find core-image-sato-sdk/1.0-r0/rootfs/ -size +8 -printf "%p #%S\n" | grep -v '#0\|#1' | wc -l
0

I will send an official patch for review with the hard links solution if you are fine with it.

// Robert

Quoted 6 lines

By the way, this is exactly the kind of thing we were hoping to catch by using the debugfs as an intermediate step. If anything needs to be added to libext2fs, we want to know before we jump into the mke2fs solution. So this is working as planned - just maybe sooner than expected, which is great.

#7 lore raw
Quoted 103 lines · Robert Yang
Quoted 59 lines · Darren Hart
Quoted 56 lines · Darren Hart
Quoted 53 lines · Richard Purdie
Quoted 40 lines · Robert Yang

Replace genext2fs with populate-extfs.sh in image_types.bbclass to fully support ext3/ext4 rootfs.

Comments from Darren:
We used genext2fs to create ext3/ext4 rootfs in the past, basically, genext2fs doesn't support creating ext4 filesystems. It creates, as I understand it, an ext2 filesystem, then adds a journal, and sets some bits. It can't support the newer features like extents. So what we end up with is a bit of a hack for a filesystem.

The ext tools (e2fsprogs) unfortunately don't provide an integrated solution for generating prepopulated filesystem images as many other mkfs* tools do. One thing missing was symlink support in libext2fs. I added that support and demonstrated a script which uses the e2fsprogs debugfs tool that can populate the newly formatted filesystem from a directory and without root privileges.

This patches integrate this stage of development into OE-Core. We can go about this in two ways. One is to just prototype this in a branch and use it to validate the functionality and not make any changes to oe-core image generation until mke2fs has initial directory support. The other is to merge this and get broader testing of the concept and later move to the full mke2fs implementation once it becomes available. I understand the resistance to the latter, but long term I think it will result in a more robust solution as we will have caught more of the corner cases and have been able to do a better job integrating into mke2fs the first time.

* Impact:
+ Rootfs generation time:
- For a core-image-minimal image, about more 3 seconds are needed
- For a core-image-sato image, about more 15 seconds are needed

+ Disk space usage:
- Nearly no changes:
$ ls -lh BEFORE.rootfs.ext3 AFTER.rootfs.ext3 | awk '{print $5"\t"$NF}' 357M BEFORE.rootfs.ext3
357M AFTER.rootfs.ext3

Quoted 6 lines

$ du -sh BEFORE.rootfs.ext3 AFTER.rootfs.ext3
238M BEFORE.rootfs.ext3
357M AFTER.rootfs.ext3
# This is different because BEFORE.rootfs.ext3 has sparse files, # they are very similar (less than 1M gap) after mount them and run # "du -sh".

I can live with the performance issues however as I understand this code, its breaking both sparse files and also likely hardlinked files. I'm not sure we have many sparse ones but we do have packages with heavy hardlinking (the sdk image toolchain packages for example).

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

Adding the script to e2fsprogs isn't a problem but I am tempted to wait until this work is completed before we start using it instead of genext2fs.

Agreed, I thought we were within 1MB on size from my reading of the previous discussion. Robert, do you have any thoughts on what is needed to address the hardlinks? That seems like something we should be able to

Yes, we do have hard links in the rootfs, e.g: core-image-sato and core-image-sato-sdk:

# The sato

$ find core-image-sato/1.0-r0/rootfs/ -type f -printf "%p %i %n\n" | grep -v '1$'
core-image-sato/1.0-r0/rootfs/etc/terminfo/v/vt200 26977609 2
core-image-sato/1.0-r0/rootfs/etc/terminfo/v/vt220 26977609 2
core-image-sato/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld.bfd 26978690 2
core-image-sato/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld 26978690 2

# The sato-sdk:

$ find core-image-sato-sdk/1.0-r0/rootfs/ -type f -printf "%p %i %n\n" | grep -v
'1$'
core-image-sato-sdk/1.0-r0/rootfs/etc/terminfo/v/vt200 26903519 2
core-image-sato-sdk/1.0-r0/rootfs/etc/terminfo/v/vt220 26903519 2
core-image-sato-sdk/1.0-r0/rootfs/usr/include/et/com_err.h 28058656 2
core-image-sato-sdk/1.0-r0/rootfs/usr/include/com_err.h 28058656 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld.bfd 26900939 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/perlbug 26899680 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/gawk-4.0.1 26900368 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/psed 26899667 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/s2p 26899667 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/c2ph 26899695 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/perlthanks 26899680 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/pstruct 26899695 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld 26900939 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/gawk 26900368 2

We can make hard links according to these information, and the find command is very fast, so it would not cost much time.

Quoted 8 lines
Quoted 5 lines
Quoted 4 lines

address quickly.

Same question for the sparse files, although I could see that taking some additional effort.

I don't find any obvious sparse files by the following command:

Sorry, we do have several sparse files, the command was wrong, it should be:

# sato:

$ find core-image-sato/1.0-r0/rootfs/ -size +8 -printf "%p #%S\n" | grep -v '#1'
core-image-sato/1.0-r0/rootfs/var/lib/rpm/Sha1header #0.916667
core-image-sato/1.0-r0/rootfs/var/lib/rpm/log/log.0000000001 #0.837109
core-image-sato/1.0-r0/rootfs/var/lib/rpm/Filedigests #0.857143

3 files

# sato-sdk:

$ find core-image-sato-sdk/1.0-r0/rootfs/ -size +8 -printf "%p #%S\n" | grep -v '#1'
core-image-sato-sdk/1.0-r0/rootfs/var/lib/rpm/Sigmd5 #0.904762
core-image-sato-sdk/1.0-r0/rootfs/var/lib/rpm/Sha1header #0.857143
core-image-sato-sdk/1.0-r0/rootfs/var/lib/rpm/log/log.0000000007 #0.150781
core-image-sato-sdk/1.0-r0/rootfs/var/lib/rpm/Filedigests #0.847619

4 files

I don't have any good ideas on how to fix the sparse files currently, it seems that we should modify the debugfs' "write" command to let it can create the sparse files, just like cp's --sparse=WHEN.

// Robert

Quoted 29 lines

# sato:

$ find core-image-sato/1.0-r0/rootfs/ -size +8 -printf "%p #%S\n" | grep -v
'#0\|#1' | wc -l
0

# sato-sdk:

$ find core-image-sato-sdk/1.0-r0/rootfs/ -size +8 -printf "%p #%S\n" | grep -v
'#0\|#1' | wc -l
0

I will send an official patch for review with the hard links solution if you are fine with it.

// Robert

Quoted 6 lines

By the way, this is exactly the kind of thing we were hoping to catch by using the debugfs as an intermediate step. If anything needs to be added to libext2fs, we want to know before we jump into the mke2fs solution. So this is working as planned - just maybe sooner than expected, which is great.

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

#8 lore raw
In reply to Robert Yang
Quoted 9 lines · Robert Yang
Quoted 3 lines · Darren Hart

On 03/03/2013 09:15 AM, Darren Hart wrote:

Yes, we do have hard links in the rootfs, e.g: core-image-sato and core-image-sato-sdk:

Quoted 6 lines

# The sato

$ find core-image-sato/1.0-r0/rootfs/ -type f -printf "%p %i %n\n" | grep -v '1$'
core-image-sato/1.0-r0/rootfs/etc/terminfo/v/vt200 26977609 2
core-image-sato/1.0-r0/rootfs/etc/terminfo/v/vt220 26977609 2
core-image-sato/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld.bfd 26978690 2
core-image-sato/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld 26978690 2

What if the count was 11? or 21? I'm just mentioning we need to improve that command!

Quoted 17 lines

# The sato-sdk:

$ find core-image-sato-sdk/1.0-r0/rootfs/ -type f -printf "%p %i %n\n" | grep -v 
'1$'
core-image-sato-sdk/1.0-r0/rootfs/etc/terminfo/v/vt200 26903519 2
core-image-sato-sdk/1.0-r0/rootfs/etc/terminfo/v/vt220 26903519 2
core-image-sato-sdk/1.0-r0/rootfs/usr/include/et/com_err.h 28058656 2
core-image-sato-sdk/1.0-r0/rootfs/usr/include/com_err.h 28058656 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld.bfd 26900939 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/perlbug 26899680 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/gawk-4.0.1 26900368 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/psed 26899667 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/s2p 26899667 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/c2ph 26899695 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/perlthanks 26899680 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/pstruct 26899695 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld 26900939 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/gawk 26900368 2

I'm slightly surprised there aren't more hardlinks for gcc in here. A meta-toolchain-sdk tarball will probably have a load of hardlinks in too if you want something to test with (we don't usually make extX images for the toolchain though).

Cheers,

Richard

#9 lore raw
Quoted 17 lines · Richard Purdie
Quoted 9 lines · Robert Yang
Quoted 3 lines · Darren Hart

On 03/03/2013 09:15 AM, Darren Hart wrote:

Yes, we do have hard links in the rootfs, e.g: core-image-sato and core-image-sato-sdk:

Quoted 6 lines

# The sato

$ find core-image-sato/1.0-r0/rootfs/ -type f -printf "%p %i %n\n" | grep -v '1$'
core-image-sato/1.0-r0/rootfs/etc/terminfo/v/vt200 26977609 2
core-image-sato/1.0-r0/rootfs/etc/terminfo/v/vt220 26977609 2
core-image-sato/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld.bfd 26978690 2
core-image-sato/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld 26978690 2
Quoted 2 lines

What if the count was 11? or 21? I'm just mentioning we need to improve that command!

Yes a -v ' 1$' would be better.

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Technical Lead - Linux Kernel
#10 lore raw
In reply to Richard Purdie
Quoted 38 lines · Richard Purdie
Quoted 9 lines · Robert Yang
Quoted 3 lines · Darren Hart

On 03/03/2013 09:15 AM, Darren Hart wrote:

Yes, we do have hard links in the rootfs, e.g: core-image-sato and core-image-sato-sdk:

Quoted 6 lines

# The sato

$ find core-image-sato/1.0-r0/rootfs/ -type f -printf "%p %i %n\n" | grep -v '1$'
core-image-sato/1.0-r0/rootfs/etc/terminfo/v/vt200 26977609 2
core-image-sato/1.0-r0/rootfs/etc/terminfo/v/vt220 26977609 2
core-image-sato/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld.bfd 26978690 2
core-image-sato/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld 26978690 2

What if the count was 11? or 21? I'm just mentioning we need to improve that command!

Quoted 17 lines

# The sato-sdk:

$ find core-image-sato-sdk/1.0-r0/rootfs/ -type f -printf "%p %i %n\n" | grep -v
'1$'
core-image-sato-sdk/1.0-r0/rootfs/etc/terminfo/v/vt200 26903519 2
core-image-sato-sdk/1.0-r0/rootfs/etc/terminfo/v/vt220 26903519 2
core-image-sato-sdk/1.0-r0/rootfs/usr/include/et/com_err.h 28058656 2
core-image-sato-sdk/1.0-r0/rootfs/usr/include/com_err.h 28058656 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld.bfd 26900939 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/perlbug 26899680 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/gawk-4.0.1 26900368 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/psed 26899667 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/s2p 26899667 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/c2ph 26899695 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/perlthanks 26899680 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/pstruct 26899695 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/arm-poky-linux-gnueabi-ld 26900939 2
core-image-sato-sdk/1.0-r0/rootfs/usr/bin/gawk 26900368 2
Quoted 5 lines

I'm slightly surprised there aren't more hardlinks for gcc in here. A meta-toolchain-sdk tarball will probably have a load of hardlinks in too if you want something to test with (we don't usually make extX images for the toolchain though).

The hardlinks result from the meta-toolchain-sdk is similar:

$ find sdk/image/opt/poky/1.3+snapshot/sysroots/ -type f -printf "%p %i %n\n" | grep -v ' 1$'
sdk/image/opt/poky/1.3+snapshot/sysroots/armv5te-poky-linux-gnueabi/etc/terminfo/v/vt200 27337495 2
sdk/image/opt/poky/1.3+snapshot/sysroots/armv5te-poky-linux-gnueabi/etc/terminfo/v/vt220 27337495 2
sdk/image/opt/poky/1.3+snapshot/sysroots/armv5te-poky-linux-gnueabi/usr/include/et/com_err.h 27468703 2
sdk/image/opt/poky/1.3+snapshot/sysroots/armv5te-poky-linux-gnueabi/usr/include/com_err.h 27468703 2
sdk/image/opt/poky/1.3+snapshot/sysroots/armv5te-poky-linux-gnueabi/usr/bin/gawk-4.0.1 27297079 2
sdk/image/opt/poky/1.3+snapshot/sysroots/armv5te-poky-linux-gnueabi/usr/bin/gawk 27297079 2
sdk/image/opt/poky/1.3+snapshot/sysroots/x86_64-pokysdk-linux/etc/terminfo/v/vt200 27469810 2
sdk/image/opt/poky/1.3+snapshot/sysroots/x86_64-pokysdk-linux/etc/terminfo/v/vt220 27469810 2
sdk/image/opt/poky/1.3+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/armv5te-poky-linux-gnueabi/arm-poky-linux-gnueabi-ld.bfd 27484702 2
sdk/image/opt/poky/1.3+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/armv5te-poky-linux-gnueabi/arm-poky-linux-gnueabi-ld 27484702 2

I wonder what's next step please, can we ignore the sparse files at the moment, and then I will send an patch for review with the hardlink fixes.

// Robert

Quoted 6 lines

Cheers,

Richard