genext2fs: support large files and filesystems without using large amounts of memor

Hi RP, Saul, Paul, Darren, Mark, Josh and joaohf and all, please comment.

Let's figure out if this big patch is accepatable or not...

With this patch, I can successfully create a 8.5GB .ext3 file with genext2fs. The speed is slow -- I spent about 1.5 hours.

The following changes since commit 265903bdffb10c95ceaf7a892151a50b67939c71:

procps: don't print error message with kernel 3.0+ (2012-03-28 10:16:30 +0100)

are available in the git repository at:
git://git.pokylinux.org/poky-contrib dcui/master
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dcui/master

Dexuan Cui (1):
genext2fs: support large files and filesystems without using large
amounts of memory

 ...01-Fix-warnings-remove-some-unused-macros.patch |   70 ++
 .../0002-Add-put_blk-and-put_nod-routines.patch    | 1121 ++++++++++++++++++++
 .../0003-Add-get_blkmap-and-put_blkmap.patch       |  220 ++++
 ...lker-for-walking-through-directory-entrie.patch |  355 +++++++
 ...05-Make-filesystem-struct-not-an-overloay.patch |  372 +++++++
 ...0006-Improve-the-efficiency-of-extend_blk.patch |  270 +++++
 ...ove-hdlinks-into-the-filesystem-structure.patch |  173 +++
 ...t-the-creation-of-the-filesystem-structur.patch |   93 ++
 ...e-byte-swapping-into-the-get-put-routines.patch |  419 ++++++++
 ...rt-over-to-keeping-the-filesystem-on-disk.patch |  837 +++++++++++++++
 ...les-into-the-filesystem-a-piece-at-a-time.patch |  101 ++
 ...upport-large-file-support-and-rework-hole.patch |  209 ++++
 .../0013-Add-volume-id-support.patch               |   84 ++
 ...014-Remove-unneeded-setting-of-s_reserved.patch |   26 +
 ...-Rework-creating-the-lost-found-directory.patch |   55 +
 ...ix-the-documentation-for-the-new-L-option.patch |   27 +
 .../0017-Fix-file-same-comparison.patch            |   28 +
 ...andle-files-changing-while-we-are-working.patch |   87 ++
 ...ke-sure-superblock-is-clear-on-allocation.patch |   40 +
 .../genext2fs-1.4.1/fix-nbblocks-cast.patch        |   18 +-
 .../genext2fs/genext2fs-1.4.1/update_to_1.95.patch |  119 +++
 meta/recipes-devtools/genext2fs/genext2fs_1.4.1.bb |   24 +-
 22 files changed, 4738 insertions(+), 10 deletions(-)
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0001-Fix-warnings-remove-some-unused-macros.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0002-Add-put_blk-and-put_nod-routines.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0003-Add-get_blkmap-and-put_blkmap.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0004-Add-a-dirwalker-for-walking-through-directory-entrie.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0005-Make-filesystem-struct-not-an-overloay.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0006-Improve-the-efficiency-of-extend_blk.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0007-Move-hdlinks-into-the-filesystem-structure.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0008-Separate-out-the-creation-of-the-filesystem-structur.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0009-Move-byte-swapping-into-the-get-put-routines.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0010-Convert-over-to-keeping-the-filesystem-on-disk.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0011-Copy-files-into-the-filesystem-a-piece-at-a-time.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0012-Add-rev-1-support-large-file-support-and-rework-hole.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0013-Add-volume-id-support.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0014-Remove-unneeded-setting-of-s_reserved.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0015-Rework-creating-the-lost-found-directory.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0016-Fix-the-documentation-for-the-new-L-option.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0017-Fix-file-same-comparison.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0018-Handle-files-changing-while-we-are-working.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0019-Make-sure-superblock-is-clear-on-allocation.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/update_to_1.95.patch

1.7.6

update_to_1.95.patch was generated by making a diff bewteen the 1.4.1 release and the latest 1.9.5 version in the cvs repo:
http://genext2fs.cvs.sourceforge.net/viewvc/genext2fs/genext2fs/genext2fs.c?revision=1.95

The patches 0001-0019 come from mailing list of genext2fs-devel
http://sourceforge.net/mailarchive/forum.php?forum_name=genext2fs-devel&max_rows=100&style=flat&viewmonth=201106

---
 ...01-Fix-warnings-remove-some-unused-macros.patch |   70 ++
 .../0002-Add-put_blk-and-put_nod-routines.patch    | 1121 ++++++++++++++++++++
 .../0003-Add-get_blkmap-and-put_blkmap.patch       |  220 ++++
 ...lker-for-walking-through-directory-entrie.patch |  355 +++++++
 ...05-Make-filesystem-struct-not-an-overloay.patch |  372 +++++++
 ...0006-Improve-the-efficiency-of-extend_blk.patch |  270 +++++
 ...ove-hdlinks-into-the-filesystem-structure.patch |  173 +++
 ...t-the-creation-of-the-filesystem-structur.patch |   93 ++
 ...e-byte-swapping-into-the-get-put-routines.patch |  419 ++++++++
 ...rt-over-to-keeping-the-filesystem-on-disk.patch |  837 +++++++++++++++
 ...les-into-the-filesystem-a-piece-at-a-time.patch |  101 ++
 ...upport-large-file-support-and-rework-hole.patch |  209 ++++
 .../0013-Add-volume-id-support.patch               |   84 ++
 ...014-Remove-unneeded-setting-of-s_reserved.patch |   26 +
 ...-Rework-creating-the-lost-found-directory.patch |   55 +
 ...ix-the-documentation-for-the-new-L-option.patch |   27 +
 .../0017-Fix-file-same-comparison.patch            |   28 +
 ...andle-files-changing-while-we-are-working.patch |   87 ++
 ...ke-sure-superblock-is-clear-on-allocation.patch |   40 +
 .../genext2fs-1.4.1/fix-nbblocks-cast.patch        |   18 +-
 .../genext2fs/genext2fs-1.4.1/update_to_1.95.patch |  119 +++
 meta/recipes-devtools/genext2fs/genext2fs_1.4.1.bb |   24 +-
 22 files changed, 4738 insertions(+), 10 deletions(-)
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0001-Fix-warnings-remove-some-unused-macros.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0002-Add-put_blk-and-put_nod-routines.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0003-Add-get_blkmap-and-put_blkmap.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0004-Add-a-dirwalker-for-walking-through-directory-entrie.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0005-Make-filesystem-struct-not-an-overloay.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0006-Improve-the-efficiency-of-extend_blk.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0007-Move-hdlinks-into-the-filesystem-structure.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0008-Separate-out-the-creation-of-the-filesystem-structur.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0009-Move-byte-swapping-into-the-get-put-routines.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0010-Convert-over-to-keeping-the-filesystem-on-disk.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0011-Copy-files-into-the-filesystem-a-piece-at-a-time.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0012-Add-rev-1-support-large-file-support-and-rework-hole.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0013-Add-volume-id-support.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0014-Remove-unneeded-setting-of-s_reserved.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0015-Rework-creating-the-lost-found-directory.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0016-Fix-the-documentation-for-the-new-L-option.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0017-Fix-file-same-comparison.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0018-Handle-files-changing-while-we-are-working.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/0019-Make-sure-superblock-is-clear-on-allocation.patch
 create mode 100644 meta/recipes-devtools/genext2fs/genext2fs-1.4.1/update_to_1.95.patch

new file mode 100644
index 0000000..f1dd89d
@@ -0,0 +1,70 @@
+From 1399df7672ec309523bcd067da24d72aa624f783 Mon Sep 17 00:00:00 2001
+From: Corey Minyard <[email protected]>
+Date: Wed, 1 Jun 2011 07:51:24 -0500
+Subject: [PATCH 01/19] Fix warnings, remove some unused macros.
+
+These are some annoying warnings with newer toolchains.  And NAMLEN is
+never used, so just get rid of it.
+---
+ genext2fs.c |   15 +++++++++------
+ 1 files changed, 9 insertions(+), 6 deletions(-)
+
+diff --git a/genext2fs.c b/genext2fs.c
+index f0d797d..284862d 100644
+--- a/genext2fs.c
++++ b/genext2fs.c
+@@ -107,10 +107,8 @@
+ 
+ #if HAVE_DIRENT_H
+ # include <dirent.h>
+-# define NAMLEN(dirent) strlen((dirent)->d_name)
+ #else
+ # define dirent direct
+-# define NAMLEN(dirent) (dirent)->d_namlen
+ # if HAVE_SYS_NDIR_H
+ #  include <sys/ndir.h>
+ # endif
+@@ -1441,7 +1439,8 @@ mkfile_fs(filesystem *fs, uint32 parent_nod, const char *name, uint32 mode, size
+ 		if(!(b = (uint8*)calloc(rndup(size, BLOCKSIZE), 1)))
+ 			error_msg_and_die("not enough mem to read file '%s'", name);
+ 		if(f)
+-			fread(b, size, 1, f); // FIXME: ugly. use mmap() ...
++			if (fread(b, size, 1, f) != 1) // FIXME: ugly. use mmap() ...
++				error_msg_and_die("fread failed");
+ 		extend_blk(fs, nod, b, rndup(size, BLOCKSIZE) / BLOCKSIZE);
+ 		free(b);
+ 	}
+@@ -1673,7 +1672,9 @@ add2fs_from_dir(filesystem *fs, uint32 this_nod, int squash_uids, int squash_per
+ 					if(chdir(dent->d_name) < 0)
+ 						perror_msg_and_die(dent->d_name);
+ 					add2fs_from_dir(fs, this_nod, squash_uids, squash_perms, fs_timestamp, stats);
+-					chdir("..");
++					if (chdir("..") == -1)
++						perror_msg_and_die("..");
++
+ 					break;
+ 				default:
+ 					break;
+@@ -1687,7 +1688,8 @@ add2fs_from_dir(filesystem *fs, uint32 this_nod, int squash_uids, int squash_per
+ 					if(chdir(dent->d_name) < 0)
+ 						perror_msg_and_die(name);
+ 					add2fs_from_dir(fs, nod, squash_uids, squash_perms, fs_timestamp, stats);
+-					chdir("..");
++					if (chdir("..") == -1)
++						perror_msg_and_die("..");
+ 				}
+ 				continue;
+ 			}
+@@ -1733,7 +1735,8 @@ add2fs_from_dir(filesystem *fs, uint32 this_nod, int squash_uids, int squash_per
+ 					if(chdir(dent->d_name) < 0)
+ 						perror_msg_and_die(name);
+ 					add2fs_from_dir(fs, nod, squash_uids, squash_perms, fs_timestamp, stats);
+-					chdir("..");
++					if (chdir("..") == -1)
++						perror_msg_and_die("..");
+ 					break;
+ 				default:
+ 					error_msg("ignoring entry %s", name);
+-- 
+1.7.4.1
+

new file mode 100644
index 0000000..f538b7e

new file mode 100644
index 0000000..94ac91e

new file mode 100644
index 0000000..57c745b

new file mode 100644
index 0000000..83d4e4e

index 3fd15e0..05b095e 100644

index 012ec6c..702245f 100644

1.7.6

Part of the diff is not shown here. View the raw message.

Dexuan Cui wrote on 2012-03-29:

Based on the latest poky master, I built from scratch with DL_DIR pointing to an empty directory. After the building finished, I got more data:

the size of DL_DIR is about 3.1 GB.
genext2fs took about 65 minutes to create a 5.4GB .ext3 file. The final .vmdk file's size is about 3.5GB.

Thanks,
-- Dexuan

If these patches solved all the problems and made things work wonderfully I'd probably say we'd take them. Unfortunately I don't consider taking 1.5 hours to build am 8GB filesystem "wonderful", its rather worrying and I don't think its performing any where need fast enough for our needs :(.

Patching genext2fs at this point in the cycle is a rather risky undertaking too and I'm getting very concerned about this.

We might have to look at alternative ways to solve this problem. I think Darren said he might help look at this and has some other ideas. Darren?

Cheers,

Richard

Can fuse offer some option here?

-M

Only if we could rely on fuse being available on everyone's system, which I'm not sure is the case.

Cheers,
Paul

Paul Eggleton
Intel Open Source Technology Centre

Without having looked into this very far, I seem to recall this was generating sparse files. This means every time we copy something into the filesystem it has to allocate the space on the drive. This is great for building big filesystems that will be mostly empty - but for big filesystems that we plan to mostly populate, I believe we would be better off doing pre-allocation (I'm taking this from my experience creating VMs in virtmanager). However, since using dd to create the file would require mounting it in order to copy files across (and we want to avoid requiring root permissions during build), this may not be an option.

Have we tried without the -z option? (-z allows holes in files - I presume this means sparse files).

Depending on how much of the 8.5 GB image we are filling, we may get a significant speed increase by first generating a smaller image and then using resize2fs to grow it to the desired size (referencing
http://landley.livejournal.com/47024.html).

A quick test without passing an initial rootfs showed no difference between -z and no -z. It may still be worth trying with the actual population to see if that makes a difference.

Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Darren Hart wrote on 2012-03-30:

I agree.
I understand the risk.

But, BTW, I personally tend to think the quality of the patches (which Come from the genext2fs mailing list) are good:
please see my test results
http://sourceforge.net/mailarchive/message.php?msg_id=29062014

I tried the -z option of genext2fs, but the speed to create the filesystem is the same.

Thanks,
-- Dexuan

Op 29 mrt. 2012, om 15:07 heeft Richard Purdie het volgende geschreven:

The biggest issue I have with genext2fs is that it currently allocates RAM for the complete image, which in my case will cause it to swap and for other users cause it to crash, since image > (ram + swap). Fixing the memory issue before the release would be greatly appreciated.

What about putting it on the shortlist for backports after the release? That should give us all more time to test it and eventually get into the hands of stable branch users. Of course I'd still prefer it to go in before the release and backport fixes :)

regards,

Koen

I should point out one of the patches in this mega patch set is one for the memory issue.

I'd appreciate some others giving this patch set a go, see if they can find any regressions for their usual use cases.

Based on the feedback I've had so far I'm leaning towards merging it but I'm not making a final decision yet, I'd like some test reports...

Cheers,

Richard

Op 30 mrt. 2012, om 14:49 heeft Richard Purdie het volgende geschreven:

Hence my mail :)