dexuan: more build appliance patches: [Apr 1, 2012]

BTW:
I added a "Upstream-Status: inappropriat" into every patches to genext2fs.

With the patches, I only spent about 10~15 minutes to create a 4.5GB .rootfs.ext3 file by genext2fs.

The following changes since commit 8691a588267472eb5a32b978a0eb9ddfd0c91733:

cross-canadian.bbclass: fix rpath for sdk executables (2012-03-31 18:00:59 +0100)

are available in the git repository at:
git://git.yoctoproject.org/poky-contrib dcui/master
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=dcui/master

Dexuan Cui (5):
genext2fs: support large files and filesystems without using large
amounts of memory
self-hosted-image: improve do_populate_poky_src
self-hosted-image: renew the poky src's SRCREV that's installed into
the target
builder: uncomment the settings for BB_NUMBER_THREADS and
PARALLEL_MAKE
self-hosted-image: fix indentation: 8 spaces --> Tab

 meta/recipes-core/images/self-hosted-image.bb      |   21 +-
 ...01-Fix-warnings-remove-some-unused-macros.patch |   72 ++
 .../0002-Add-put_blk-and-put_nod-routines.patch    | 1123 ++++++++++++++++++++
 .../0003-Add-get_blkmap-and-put_blkmap.patch       |  222 ++++
 ...lker-for-walking-through-directory-entrie.patch |  357 +++++++
 ...05-Make-filesystem-struct-not-an-overloay.patch |  374 +++++++
 ...0006-Improve-the-efficiency-of-extend_blk.patch |  272 +++++
 ...ove-hdlinks-into-the-filesystem-structure.patch |  175 +++
 ...t-the-creation-of-the-filesystem-structur.patch |   95 ++
 ...e-byte-swapping-into-the-get-put-routines.patch |  421 ++++++++
 ...rt-over-to-keeping-the-filesystem-on-disk.patch |  839 +++++++++++++++
 ...les-into-the-filesystem-a-piece-at-a-time.patch |  103 ++
 ...upport-large-file-support-and-rework-hole.patch |  211 ++++
 .../0013-Add-volume-id-support.patch               |   86 ++
 ...014-Remove-unneeded-setting-of-s_reserved.patch |   28 +
 ...-Rework-creating-the-lost-found-directory.patch |   57 +
 ...ix-the-documentation-for-the-new-L-option.patch |   29 +
 .../0017-Fix-file-same-comparison.patch            |   30 +
 ...andle-files-changing-while-we-are-working.patch |   89 ++
 ...ke-sure-superblock-is-clear-on-allocation.patch |   42 +
 .../genext2fs-1.4.1/fix-nbblocks-cast.patch        |   18 +-
 .../genext2fs/genext2fs-1.4.1/update_to_1.95.patch |  119 ++
 meta/recipes-devtools/genext2fs/genext2fs_1.4.1.bb |   24 +-
 meta/recipes-graphics/builder/builder_0.1.bb       |    2 +-
 .../builder/files/builder_hob_start.sh             |    5 +
 25 files changed, 4795 insertions(+), 19 deletions(-)
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0001-Fix-warnings-remove-some-unused-macros.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0002-Add-put_blk-and-put_nod-routines.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0003-Add-get_blkmap-and-put_blkmap.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0004-Add-a-dirwalker-for-walking-through-directory-entrie.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0005-Make-filesystem-struct-not-an-overloay.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0006-Improve-the-efficiency-of-extend_blk.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0007-Move-hdlinks-into-the-filesystem-structure.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0008-Separate-out-the-creation-of-the-filesystem-structur.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0009-Move-byte-swapping-into-the-get-put-routines.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0010-Convert-over-to-keeping-the-filesystem-on-disk.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0011-Copy-files-into-the-filesystem-a-piece-at-a-time.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0012-Add-rev-1-support-large-file-support-and-rework-hole.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0013-Add-volume-id-support.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0014-Remove-unneeded-setting-of-s_reserved.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0015-Rework-creating-the-lost-found-directory.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0016-Fix-the-documentation-for-the-new-L-option.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0017-Fix-file-same-comparison.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0018-Handle-files-changing-while-we-are-working.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0019-Make-sure-superblock-is-clear-on-allocation.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/update_to_1.95.patch

1.7.6

update_to_1.95.patch was generated by making a diff bewteen the 1.4.1 release and the latest 1.9.5 version in the cvs repo:
http://genext2fs.cvs.sourceforge.net/viewvc/genext2fs/genext2fs/genext2fs.c?revision=1.95

The patches 0001-0019 come from mailing list of genext2fs-devel
http://sourceforge.net/mailarchive/forum.php?forum_name=genext2fs-devel&max_rows=100&style=flat&viewmonth=201106

---
 ...01-Fix-warnings-remove-some-unused-macros.patch |   72 ++
 .../0002-Add-put_blk-and-put_nod-routines.patch    | 1123 ++++++++++++++++++++
 .../0003-Add-get_blkmap-and-put_blkmap.patch       |  222 ++++
 ...lker-for-walking-through-directory-entrie.patch |  357 +++++++
 ...05-Make-filesystem-struct-not-an-overloay.patch |  374 +++++++
 ...0006-Improve-the-efficiency-of-extend_blk.patch |  272 +++++
 ...ove-hdlinks-into-the-filesystem-structure.patch |  175 +++
 ...t-the-creation-of-the-filesystem-structur.patch |   95 ++
 ...e-byte-swapping-into-the-get-put-routines.patch |  421 ++++++++
 ...rt-over-to-keeping-the-filesystem-on-disk.patch |  839 +++++++++++++++
 ...les-into-the-filesystem-a-piece-at-a-time.patch |  103 ++
 ...upport-large-file-support-and-rework-hole.patch |  211 ++++
 .../0013-Add-volume-id-support.patch               |   86 ++
 ...014-Remove-unneeded-setting-of-s_reserved.patch |   28 +
 ...-Rework-creating-the-lost-found-directory.patch |   57 +
 ...ix-the-documentation-for-the-new-L-option.patch |   29 +
 .../0017-Fix-file-same-comparison.patch            |   30 +
 ...andle-files-changing-while-we-are-working.patch |   89 ++
 ...ke-sure-superblock-is-clear-on-allocation.patch |   42 +
 .../genext2fs-1.4.1/fix-nbblocks-cast.patch        |   18 +-
 .../genext2fs/genext2fs-1.4.1/update_to_1.95.patch |  119 ++
 meta/recipes-devtools/genext2fs/genext2fs_1.4.1.bb |   24 +-
 22 files changed, 4776 insertions(+), 10 deletions(-)
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0001-Fix-warnings-remove-some-unused-macros.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0002-Add-put_blk-and-put_nod-routines.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0003-Add-get_blkmap-and-put_blkmap.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0004-Add-a-dirwalker-for-walking-through-directory-entrie.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0005-Make-filesystem-struct-not-an-overloay.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0006-Improve-the-efficiency-of-extend_blk.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0007-Move-hdlinks-into-the-filesystem-structure.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0008-Separate-out-the-creation-of-the-filesystem-structur.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0009-Move-byte-swapping-into-the-get-put-routines.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0010-Convert-over-to-keeping-the-filesystem-on-disk.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0011-Copy-files-into-the-filesystem-a-piece-at-a-time.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0012-Add-rev-1-support-large-file-support-and-rework-hole.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0013-Add-volume-id-support.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0014-Remove-unneeded-setting-of-s_reserved.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0015-Rework-creating-the-lost-found-directory.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0016-Fix-the-documentation-for-the-new-L-option.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0017-Fix-file-same-comparison.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0018-Handle-files-changing-while-we-are-working.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0019-Make-sure-superblock-is-clear-on-allocation.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/update_to_1.95.patch

new file mode 100644
index 0000000..f981b44
@@ -0,0 +1,72 @@
+Upstream-Status: inappropriate
+
+From 1399df7672ec309523bcd067da24d72aa624f783 Mon Sep 17 00:00:00 2001
+From: Corey Minyard <[email protected]>
+Date: Wed, 1 Jun 2011 07:51:24 -0500
+Subject: [PATCH 01/19] Fix warnings, remove some unused macros.
+
+These are some annoying warnings with newer toolchains.  And NAMLEN is
+never used, so just get rid of it.
+---
+ genext2fs.c |   15 +++++++++------
+ 1 files changed, 9 insertions(+), 6 deletions(-)
+
+diff --git a/genext2fs.c b/genext2fs.c
+index f0d797d..284862d 100644
+--- a/genext2fs.c
++++ b/genext2fs.c
+@@ -107,10 +107,8 @@
+ 
+ #if HAVE_DIRENT_H
+ # include <dirent.h>
+-# define NAMLEN(dirent) strlen((dirent)->d_name)
+ #else
+ # define dirent direct
+-# define NAMLEN(dirent) (dirent)->d_namlen
+ # if HAVE_SYS_NDIR_H
+ #  include <sys/ndir.h>
+ # endif
+@@ -1441,7 +1439,8 @@ mkfile_fs(filesystem *fs, uint32 parent_nod, const char *name, uint32 mode, size
+ 		if(!(b = (uint8*)calloc(rndup(size, BLOCKSIZE), 1)))
+ 			error_msg_and_die("not enough mem to read file '%s'", name);
+ 		if(f)
+-			fread(b, size, 1, f); // FIXME: ugly. use mmap() ...
++			if (fread(b, size, 1, f) != 1) // FIXME: ugly. use mmap() ...
++				error_msg_and_die("fread failed");
+ 		extend_blk(fs, nod, b, rndup(size, BLOCKSIZE) / BLOCKSIZE);
+ 		free(b);
+ 	}
+@@ -1673,7 +1672,9 @@ add2fs_from_dir(filesystem *fs, uint32 this_nod, int squash_uids, int squash_per
+ 					if(chdir(dent->d_name) < 0)
+ 						perror_msg_and_die(dent->d_name);
+ 					add2fs_from_dir(fs, this_nod, squash_uids, squash_perms, fs_timestamp, stats);
+-					chdir("..");
++					if (chdir("..") == -1)
++						perror_msg_and_die("..");
++
+ 					break;
+ 				default:
+ 					break;
+@@ -1687,7 +1688,8 @@ add2fs_from_dir(filesystem *fs, uint32 this_nod, int squash_uids, int squash_per
+ 					if(chdir(dent->d_name) < 0)
+ 						perror_msg_and_die(name);
+ 					add2fs_from_dir(fs, nod, squash_uids, squash_perms, fs_timestamp, stats);
+-					chdir("..");
++					if (chdir("..") == -1)
++						perror_msg_and_die("..");
+ 				}
+ 				continue;
+ 			}
+@@ -1733,7 +1735,8 @@ add2fs_from_dir(filesystem *fs, uint32 this_nod, int squash_uids, int squash_per
+ 					if(chdir(dent->d_name) < 0)
+ 						perror_msg_and_die(name);
+ 					add2fs_from_dir(fs, nod, squash_uids, squash_perms, fs_timestamp, stats);
+-					chdir("..");
++					if (chdir("..") == -1)
++						perror_msg_and_die("..");
+ 					break;
+ 				default:
+ 					error_msg("ignoring entry %s", name);
+-- 
+1.7.4.1
+

new file mode 100644
index 0000000..de196b0

new file mode 100644
index 0000000..8c24db1

new file mode 100644
index 0000000..40a7807

index 3fd15e0..05b095e 100644

index 012ec6c..702245f 100644

1.7.6

Part of the diff is not shown here. View the raw message.

BTW, we can see http://sourceforge.net/mailarchive/message.php?msg_id=29071716 for what Corey, the author of the patches to genext2fs, said:
"We (MontaVista) and our customers have been using these patches in production since the time I sent them out and we haven't had any issues"

So, the quality of the patches seems pretty good.

Thanks,
-- Dexuan

1) remove the assumption ${DL_DIR} ends with downloads/. Thanks Paul Eggleton for pointing this out.

2) remove downloads/git2_* tarballs to speed up the rootfs creation. This is ok since we still have the git2/.
Thanks Richard Purdie for suggesting this.

---
 meta/recipes-core/images/self-hosted-image.bb |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)

index 5aa670d..e962f30 100644
@@ -4,7 +4,7 @@ LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=3f40d7994397109285ec7b81fdeb3b58 \
                     file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
 
-PR = "r6"
+PR = "r7"
 
 CORE_IMAGE_EXTRA_INSTALL = "\
     task-self-hosted \

@@ -37,7 +37,12 @@ fakeroot do_populate_poky_src () {
 	cp -Rp ${WORKDIR}/git ${IMAGE_ROOTFS}/home/builder/poky
 
 	mkdir -p ${IMAGE_ROOTFS}/home/builder/poky/build/conf
-	cp -Rp ${DL_DIR} ${IMAGE_ROOTFS}/home/builder/poky/build
+	mkdir -p ${IMAGE_ROOTFS}/home/builder/poky/build/downloads
+	cp -RpL ${DL_DIR}/* ${IMAGE_ROOTFS}/home/builder/poky/build/downloads/
+
+	# Remove the git2_* tarballs -- this is ok since we still have the git2/.
+	rm -rf ${IMAGE_ROOTFS}/home/builder/poky/build/downloads/git2_*
+
 	echo "/usr/bin" > ${IMAGE_ROOTFS}/home/builder/poky/build/pseudodone
 	echo "BB_NO_NETWORK = \"1\"" > ${IMAGE_ROOTFS}/home/builder/poky/build/conf/auto.conf
 	echo "INHERIT += \"rm_work\"" >> ${IMAGE_ROOTFS}/home/builder/poky/build/conf/auto.conf

1.7.6

---
 meta/recipes-core/images/self-hosted-image.bb |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

index e962f30..91942ee 100644
@@ -4,7 +4,7 @@ LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=3f40d7994397109285ec7b81fdeb3b58 \
                     file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
 
-PR = "r7"
+PR = "r8"
 
 CORE_IMAGE_EXTRA_INSTALL = "\
     task-self-hosted \

@@ -25,7 +25,7 @@ IMAGE_FSTYPES = "vmdk"
 
 inherit core-image
 
-SRCREV = "26a46938d3ea1821e7bec4fa6cc8379babad238b"
+SRCREV = "8691a588267472eb5a32b978a0eb9ddfd0c91733"
 SRC_URI = "git://git.yoctoproject.org/poky;protocol=git"
 
 fakeroot do_populate_poky_src () {

1.7.6

We use sed to enable the 2 settingis in conf/local.conf.

---
 meta/recipes-graphics/builder/builder_0.1.bb       |    2 +-
 .../builder/files/builder_hob_start.sh             |    5 +++++
 2 files changed, 6 insertions(+), 1 deletions(-)

index 1324e5b..af6ba51 100644
@@ -1,7 +1,7 @@
 SUMMARY = "New user to do specific job"
 DESCRIPTION = "This recipe create a new user named ${PN}, who is used for specific jobs like building. The task can be auto started via mini X"
 SECTION = "x11"
-PR = "r1"
+PR = "r2"
 LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://builder_hob_start.sh;endline=5;md5=84796c3c41785d86100fdabcbdade00e"
 

index f43e6c5..98b2adb 100644
@@ -10,6 +10,11 @@ export PSEUDO_LIBDIR=/usr/lib/pseudo/lib64
 
 cd /home/builder/poky
 . ./oe-init-build-env
+
+#uncomment the settings for BB_NUMBER_THREADS and PARALLEL_MAKE
+sed -i 's/^#BB_NUMBER_THREADS =/BB_NUMBER_THREADS =/g' conf/local.conf
+sed -i 's/^#PARALLEL_MAKE =/PARALLEL_MAKE =/g' conf/local.conf
+
 hob &
 
 matchbox-terminal&

1.7.6

---
 meta/recipes-core/images/self-hosted-image.bb |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)

index 91942ee..39f0fc4 100644
@@ -46,12 +46,12 @@ fakeroot do_populate_poky_src () {
 	echo "/usr/bin" > ${IMAGE_ROOTFS}/home/builder/poky/build/pseudodone
 	echo "BB_NO_NETWORK = \"1\"" > ${IMAGE_ROOTFS}/home/builder/poky/build/conf/auto.conf
 	echo "INHERIT += \"rm_work\"" >> ${IMAGE_ROOTFS}/home/builder/poky/build/conf/auto.conf
-        mkdir -p ${IMAGE_ROOTFS}/home/builder/pseudo
-        echo "export PSEUDO_PREFIX=/usr" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
-        echo "export PSEUDO_LOCALSTATEDIR=/home/builder/pseudo" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
-        echo "export PSEUDO_LIBDIR=/usr/lib/pseudo/lib64" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
+	mkdir -p ${IMAGE_ROOTFS}/home/builder/pseudo
+	echo "export PSEUDO_PREFIX=/usr" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
+	echo "export PSEUDO_LOCALSTATEDIR=/home/builder/pseudo" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
+	echo "export PSEUDO_LIBDIR=/usr/lib/pseudo/lib64" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
 
-        chown builder.builder ${IMAGE_ROOTFS}/home/builder/pseudo
+	chown builder.builder ${IMAGE_ROOTFS}/home/builder/pseudo
 
 	chown -R builder.builder  ${IMAGE_ROOTFS}/home/builder/poky
 }

1.7.6

I've merged these to master, thanks. Whilst I'm reluctant to do so at this point in the release process for such major changes, I think in this case it does make sense since the genext2fs changes seem to fix a number of problematic issues.

Cheers,

Richard

Op 1 apr. 2012, om 20:28 heeft Richard Purdie het volgende geschreven:

I'm getting:

| genext2fs: set_file_size: ftruncate: Invalid argument

Trying to see why that happens and find out how to fix it.

Koen Kooi wrote on 2012-04-04:

Hi Koen, did you get the cause?

This is the prototype of the function:
int ftruncate(int fd, off_t length);

I suppose you're using a 32-bit building host so the off_t in your host can't exceed 4G?

We do need to test it in a 32-bit host and at least we need to document the issue.

Thanks,
-- Dexuan

Op 5 apr. 2012 om 10:21 heeft "Cui, Dexuan" <[email protected]> het volgende geschreven:

no, 64bit host

Koen Kooi wrote on 2012-04-06:

Strange...
I use x86-64 Ubuntu 11.04 and x86-64 openSUSE 11.4 and I don't see the issue.

Thanks,
-- Dexuan