e2fsprogs October 2014 patchbomb, part 6.1

---
 e2fsck/pass5.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

e2fsck/pass5.c +2 -2

index 8d6032b..64fb7fe 100644
@@ -84,7 +84,7 @@ void e2fsck_pass5(e2fsck_t ctx)
 static void check_inode_bitmap_checksum(e2fsck_t ctx)
 {
 	struct problem_context	pctx;
-	char		*buf;
+	char		*buf = NULL;
 	dgrp_t		i;
 	int		nbytes;
 	ext2_ino_t	ino_itr;

@@ -139,7 +139,7 @@ static void check_inode_bitmap_checksum(e2fsck_t ctx)
 static void check_block_bitmap_checksum(e2fsck_t ctx)
 {
 	struct problem_context	pctx;
-	char		*buf;
+	char		*buf = NULL;
 	dgrp_t		i;
 	int		nbytes;
 	blk64_t		blk_itr;

Theodore Ts'o

Applied

2014-11-04 16:38 UTC #2 lore raw

Re: [PATCH 01/39] misc: fix compiler warnings

Quoted 3 lines · Darrick J. Wong

Fix some gcc-4.8 warnings.
Signed-off-by Darrick J. Wong <[email protected]>

Applied, thanks.

- Ted

Darrick J. Wong

author

2014-10-25 20:57 UTC #3 lore raw

PATCH 2/39 e2fuzz: exercise fuzzed blocks more aggressively

Exercise fuzzed metadata blocks more aggressively by expanding up to 50000 files (instead of just test.1, which might not hit anything). Fix a typo while we're at it.

---
 misc/e2fuzz.sh |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

misc/e2fuzz.sh +3 -3

index a261d5a..4cb7b61 100755
@@ -34,7 +34,7 @@ print_help() {
 	echo "-p:	Use system's mke2fs/e2fsck/tune2fs tools."
 	echo "-s:	Create FS images of this size. (${SZ})"
 	echo "-S:	Copy files from this dir. (${SRCDIR})"
-	echo "-x:	Run e2fck at most this many times. (${MAX_FSCK})"
+	echo "-x:	Run e2fsck at most this many times. (${MAX_FSCK})"
 	test "${HAS_FUSE2FS}" -gt 0 && echo "-u:	Use fuse2fs instead of the kernel."
 	exit 0
 }

@@ -192,12 +192,12 @@ seq 1 "${PASSES}" | while read pass; do
 		find "${TESTMNT}/test.1/" -type f -size -1048576k -print0 | xargs -0 cat > /dev/null 2>> "${OPS_LOG}"
 
 		echo "+++ expand"
-		find "${TESTMNT}/test.1/" -type f 2> /dev/null | while read f; do
+		find "${TESTMNT}/" -type f 2> /dev/null | head -n 50000 | while read f; do
 			attr -l "$f" > /dev/null 2>> "${OPS_LOG}"
-			mv "$f" "$f.longer" > /dev/null 2>> "${OPS_LOG}"
 			if [ -f "$f" -a -w "$f" ]; then
 				dd if=/dev/zero bs="${BLK_SZ}" count=1 >> "$f" 2>> "${OPS_LOG}"
 			fi
+			mv "$f" "$f.longer" > /dev/null 2>> "${OPS_LOG}"
 		done
 		sync

Theodore Ts'o

Applied

2014-11-04 16:36 UTC #4 lore raw

Re: [PATCH 02/39] e2fuzz: exercise fuzzed blocks more aggressively

Quoted 5 lines · Darrick J. Wong

Exercise fuzzed metadata blocks more aggressively by expanding up to 50000 files (instead of just test.1, which might not hit anything). Fix a typo while we're at it.
Signed-off-by Darrick J. Wong <[email protected]>

Applied, thanks.

- Ted

Darrick J. Wong

author

2014-10-25 20:57 UTC #5 lore raw

PATCH 3/39 libext2fs: directory iteration mustn't walk off the buffer end

When we're iterating a directory, the loop control code reads the length of the next directory record, failing to account for the fact that there must be at least 8 bytes (the minimum size of a directory entry) left in the buffer to read the next directory record. Fix the loop conditional so that we don't read off the end of the buffer.

---
 lib/ext2fs/dir_iterate.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

lib/ext2fs/dir_iterate.c +1 -1

index 67152cc..9063c17 100644
@@ -222,7 +222,7 @@ int ext2fs_process_dir_block(ext2_filsys fs,
 					EXT4_FEATURE_RO_COMPAT_METADATA_CSUM))
 		csum_size = sizeof(struct ext2_dir_entry_tail);
 
-	while (offset < buflen) {
+	while (offset < buflen - 8) {
 		dirent = (struct ext2_dir_entry *) (ctx->buf + offset);
 		if (ext2fs_get_rec_len(fs, dirent, &rec_len))
 			return BLOCK_ABORT;

Theodore Ts'o

Applied

2014-11-04 16:37 UTC #6 lore raw

Re: [PATCH 03/39] libext2fs: directory iteration mustn't walk off the buffer end

Quoted 8 lines · Darrick J. Wong

When we're iterating a directory, the loop control code reads the length of the next directory record, failing to account for the fact that there must be at least 8 bytes (the minimum size of a directory entry) left in the buffer to read the next directory record. Fix the loop conditional so that we don't read off the end of the buffer.
Signed-off-by Darrick J. Wong <[email protected]>
Reported-by Sami Liedes <[email protected]>

Applied, thanks.

- Ted

Darrick J. Wong

author

2014-10-25 20:57 UTC #7 lore raw

PATCH 4/39 libext2fs: zero the EA block buffer before filling it

When writing an extended attribute (EA) block, it's quite possible that the EA formatting code will not write the entire buffer. Therefore, we must zero the buffer beforehand to avoid writing random heap contents to disk.

---
 lib/ext2fs/ext_attr.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

lib/ext2fs/ext_attr.c +1 -1

index b52abb5..c397e00 100644
@@ -550,7 +550,7 @@ errcode_t ext2fs_xattrs_write(struct ext2_xattr_handle *handle)
 
 write_ea_block:
 	/* Write the EA block */
-	err = ext2fs_get_mem(handle->fs->blocksize, &block_buf);
+	err = ext2fs_get_memzero(handle->fs->blocksize, &block_buf);
 	if (err)
 		goto out;

Theodore Ts'o

Applied

2014-11-04 16:47 UTC #8 lore raw

Re: [PATCH 04/39] libext2fs: zero the EA block buffer before filling it

Quoted 7 lines · Darrick J. Wong

When writing an extended attribute (EA) block, it's quite possible that the EA formatting code will not write the entire buffer. Therefore, we must zero the buffer beforehand to avoid writing random heap contents to disk.
Signed-off-by Darrick J. Wong <[email protected]>
Reported-by Sami Liedes <[email protected]>

Applied, thanks.

- Ted

Darrick J. Wong

author

2014-10-25 20:57 UTC #9 lore raw

PATCH 5/39 libext2fs: don't memcpy identical pointers when writing a cache block

Sami Liedes found a scenario where we could memcpy incorrectly:

If a block read fails during an e2fsck run, the UNIX IO manager will call the io->read_error routine with a pointer to the internal block cache. The e2fsck read error handler immediately tries to write the buffer back out to disk(!), at which point the block write code will try to copy the buffer contents back into the block cache. Normally this is fine, but not when the write buffer is the cache itself!

So, plumb in a trivial check for this condition. A more thorough solution would pass a duplicated buffer to the IO error handlers, but I don't know if that happens frequently enough to be worth the extra point of failure.

---
 lib/ext2fs/unix_io.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

lib/ext2fs/unix_io.c +2 -1

index 189adce..c3a8ea5 100644
@@ -823,7 +823,8 @@ static errcode_t unix_write_blk64(io_channel channel, unsigned long long block,
 			cache = reuse;
 			reuse_cache(channel, data, cache, block);
 		}
-		memcpy(cache->buf, cp, channel->block_size);
+		if (cache->buf != cp)
+			memcpy(cache->buf, cp, channel->block_size);
 		cache->dirty = !writethrough;
 		count--;
 		block++;

Theodore Ts'o

Applied

2014-11-04 16:48 UTC #10 lore raw

Re: [PATCH 05/39] libext2fs: don't memcpy identical pointers when writing a cache block

Quoted 9 lines · Darrick J. Wong

Sami Liedes found a scenario where we could memcpy incorrectly:
If a block read fails during an e2fsck run, the UNIX IO manager will call the io->read_error routine with a pointer to the internal block cache. The e2fsck read error handler immediately tries to write the buffer back out to disk(!), at which point the block write code will try to copy the buffer contents back into the block cache. Normally this is fine, but not when the write buffer is the cache itself!

Quoted 7 lines

So, plumb in a trivial check for this condition. A more thorough solution would pass a duplicated buffer to the IO error handlers, but I don't know if that happens frequently enough to be worth the extra point of failure.
Signed-off-by Darrick J. Wong <[email protected]>
Reported-by Sami Liedes <[email protected]>

Applied, thanks.

- Ted

Darrick J. Wong

author

2014-10-25 20:57 UTC #11 lore raw

PATCH 6/39 misc: fix broken libmagic interaction with plausibility check

If we get as far as calling libmagic, return the correct error code so that mkfs asks for confirmation if libmagic finds something and doesn't ask if nothing is found.

---
 misc/plausible.c                     |    9 ++++++---
 tests/f_detect_junk/expect           |    6 +++---
 tests/f_detect_junk/expect.nodebugfs |    4 ++--
 tests/f_detect_junk/script           |    2 +-
 4 files changed, 12 insertions(+), 9 deletions(-)

misc/plausible.c +6 -3

index 00bc6e2..1848a26 100644
@@ -243,6 +243,7 @@ int check_plausibility(const char *device, int flags, int *ret_is_dev)
 	if ((flags & CHECK_FS_EXIST) && magic_library_available()) {
 		const char *msg;
 		magic_t mag;
+		int has_magic = 0;
 
 		mag = dl_magic_open(MAGIC_RAW | MAGIC_SYMLINK | MAGIC_DEVICES |
 				    MAGIC_ERROR | MAGIC_NO_CHECK_ELF |

@@ -250,11 +251,13 @@ int check_plausibility(const char *device, int flags, int *ret_is_dev)
 		dl_magic_load(mag, NULL);
 
 		msg = dl_magic_file(mag, device);
-		if (msg && strcmp(msg, "data") && strcmp(msg, "empty"))
-			printf(_("%s contains a `%s'\n"), device, msg);
+		if (msg && strcmp(msg, "data") && strcmp(msg, "empty")) {
+			printf(_("%s contains `%s' data\n"), device, msg);
+			has_magic = 1;
+		}
 
 		dl_magic_close(mag);
-		return 0;
+		return !has_magic;
 	}
 #endif

tests/f_detect_junk/expect +3 -3

index 57f7f89..f3300de 100644
@@ -11,13 +11,13 @@ is corrupt, and you might try running e2fsck with an alternate superblock:
  or
     e2fsck -b 32768 <device>
 
-test.img contains a `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced'
+test.img contains `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced' data
 *** debugfs
 test.img: Bad magic number in super-block while opening filesystem
-test.img contains a `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced'
+test.img contains `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced' data
 *** tune2fs
 ../misc/tune2fs: Bad magic number in super-block while trying to open test.img
-test.img contains a `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced'
+test.img contains `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced' data
 *** mke2fs
 Creating filesystem with 16384 1k blocks and 4096 inodes
 Superblock backups stored on blocks:

tests/f_detect_junk/expect.nodebugfs +2 -2

index d9281a0..0d4ba54 100644
@@ -11,11 +11,11 @@ is corrupt, and you might try running e2fsck with an alternate superblock:
  or
     e2fsck -b 32768 <device>
 
-test.img contains a `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced'
+test.img contains `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced' data
 *** debugfs
 *** tune2fs
 ../misc/tune2fs: Bad magic number in super-block while trying to open test.img
-test.img contains a `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced'
+test.img contains `PNG image data, 148 x 31, 8-bit/color RGBA, non-interlaced' data
 *** mke2fs
 Creating filesystem with 16384 1k blocks and 4096 inodes
 Superblock backups stored on blocks:

tests/f_detect_junk/script +1 -1

index 8409fdd..3aebcd0 100644
@@ -1,6 +1,6 @@
 #!/bin/bash
 
-if [ "$(grep -c 'define HAVE_MAGIC_H' $test_dir/../../lib/config.h)" -gt 0 ]; then
+if [ "$(grep -c 'define HAVE_MAGIC_H' ../lib/config.h)" -gt 0 ]; then
 
 FSCK_OPT=-fn
 IMAGE=$test_dir/image.bz2

Theodore Ts'o

Applied

2014-11-04 16:52 UTC #12 lore raw

Re: [PATCH 06/39] misc: fix broken libmagic interaction with plausibility check

Quoted 5 lines · Darrick J. Wong

If we get as far as calling libmagic, return the correct error code so that mkfs asks for confirmation if libmagic finds something and doesn't ask if nothing is found.
Signed-off-by Darrick J. Wong <[email protected]>

Applied, thanks.

- Ted

Darrick J. Wong

author

2014-10-25 20:57 UTC #13 lore raw

PATCH 7/39 tune2fs: speed up rewriting extent tree when enabling metadata_csum

When enabling checksums, tune2fs naively rewrites every extent in the entire tree! This is unnecessary since we only need to rewrite each extent tree block; therefore, only rewrite the extent if it's the first one in an internal extent tree block.

Also, don't bother iterating the extent tree when clearing checksums.

---
 misc/tune2fs.c |   49 +++++++++++++++++++++++++++++++++----------------
 1 file changed, 33 insertions(+), 16 deletions(-)

misc/tune2fs.c +33 ~~-16~~

index d17c8de..ecbdea0 100644
@@ -442,39 +442,56 @@ static errcode_t rewrite_extents(ext2_filsys fs, ext2_ino_t ino,
 {
 	ext2_extent_handle_t	handle;
 	struct ext2fs_extent	extent;
-	int			op = EXT2_EXTENT_ROOT;
 	errcode_t		errcode;
+	struct ext2_extent_info	info;
 
-	if (!(inode->i_flags & EXT4_EXTENTS_FL))
+	if (!(inode->i_flags & EXT4_EXTENTS_FL) ||
+	    !EXT2_HAS_RO_COMPAT_FEATURE(fs->super,
+					EXT4_FEATURE_RO_COMPAT_METADATA_CSUM))
 		return 0;
 
 	errcode = ext2fs_extent_open(fs, ino, &handle);
 	if (errcode)
 		return errcode;
 
-	while (1) {
-		errcode = ext2fs_extent_get(handle, op, &extent);
+	errcode = ext2fs_extent_get(handle, EXT2_EXTENT_ROOT, &extent);
+	if (errcode)
+		goto out;
+
+	do {
+		errcode = ext2fs_extent_get_info(handle, &info);
 		if (errcode)
 			break;
 
-		/* Root node is in the separately checksummed inode */
-		if (op == EXT2_EXTENT_ROOT) {
-			op = EXT2_EXTENT_NEXT;
-			continue;
+		/*
+		 * If this is the first extent in an extent block that we
+		 * haven't visited, rewrite the extent to force the ETB
+		 * checksum to be rewritten.
+		 */
+		if (info.curr_entry == 1 && info.curr_level != 0 &&
+		    !(extent.e_flags & EXT2_EXTENT_FLAGS_SECOND_VISIT)) {
+			errcode = ext2fs_extent_replace(handle, 0, &extent);
+			if (errcode)
+				break;
 		}
-		op = EXT2_EXTENT_NEXT;
 
-		/* Only visit the first extent in each extent block */
-		if (extent.e_flags & EXT2_EXTENT_FLAGS_SECOND_VISIT)
-			continue;
-		errcode = ext2fs_extent_replace(handle, 0, &extent);
-		if (errcode)
-			break;
-	}
+		/* Skip to the end of a block of leaf nodes */
+		if (extent.e_flags & EXT2_EXTENT_FLAGS_LEAF) {
+			errcode = ext2fs_extent_get(handle,
+						    EXT2_EXTENT_LAST_SIB,
+						    &extent);
+			if (errcode)
+				break;
+		}
 
+		errcode = ext2fs_extent_get(handle, EXT2_EXTENT_NEXT, &extent);
+	} while (errcode == 0);
+
+out:
 	/* Ok if we run off the end */
 	if (errcode == EXT2_ET_EXTENT_NO_NEXT)
 		errcode = 0;
+	ext2fs_extent_free(handle);
 	return errcode;
 }

Theodore Ts'o

Applied

2014-11-04 16:54 UTC #14 lore raw

Re: [PATCH 07/39] tune2fs: speed up rewriting extent tree when enabling metadata_csum

Quoted 8 lines · Darrick J. Wong

When enabling checksums, tune2fs naively rewrites every extent in the entire tree! This is unnecessary since we only need to rewrite each extent tree block; therefore, only rewrite the extent if it's the first one in an internal extent tree block.
Also, don't bother iterating the extent tree when clearing checksums.
Signed-off-by Darrick J. Wong <[email protected]>

Applied, thanks.

- Ted

Darrick J. Wong

author

2014-10-25 20:57 UTC #15 lore raw

PATCH 8/39 tune2fs: don't change metadata_csum on a mounted fs

Don't let users change metadata_csum on a mounted filesystem because there's no way to tell the kernel to turn on the feature; there's no way to prevent the kernel from rewriting on-disk structures while tune2fs is also rewriting them; and there's no way to tell the kernel to reload them after tune2fs is finished.

---
 misc/tune2fs.c |    6 ++++++
 1 file changed, 6 insertions(+)

misc/tune2fs.c +6 -0

index ecbdea0..7fee870 100644
@@ -1110,6 +1110,9 @@ mmp_error:
 		       EXT4_FEATURE_RO_COMPAT_METADATA_CSUM)) {
 		if (check_fsck_needed(fs))
 			exit(1);
+		if (mount_flags & EXT2_MF_MOUNTED)
+			fputs(_("Cannot enable metadata_csum on a mounted "
+				"filesystem!\n"), stderr);
 		rewrite_checksums = 1;
 		/* metadata_csum supersedes uninit_bg */
 		fs->super->s_feature_ro_compat &=

@@ -1132,6 +1135,9 @@ mmp_error:
 			EXT4_FEATURE_RO_COMPAT_METADATA_CSUM)) {
 		if (check_fsck_needed(fs))
 			exit(1);
+		if (mount_flags & EXT2_MF_MOUNTED)
+			fputs(_("Cannot disable metadata_csum on a mounted "
+				"filesystem!\n"), stderr);
 		rewrite_checksums = 1;
 		/*
 		 * If we're turning off metadata_csum and not turning on

Theodore Ts'o

Applied

2014-11-04 16:52 UTC #16 lore raw

Re: [PATCH 08/39] tune2fs: don't change metadata_csum on a mounted fs

Quoted 7 lines · Darrick J. Wong

Don't let users change metadata_csum on a mounted filesystem because there's no way to tell the kernel to turn on the feature; there's no way to prevent the kernel from rewriting on-disk structures while tune2fs is also rewriting them; and there's no way to tell the kernel to reload them after tune2fs is finished.
Signed-off-by Darrick J. Wong <[email protected]>

Applied, thanks.

- Ted

Darrick J. Wong

author

2014-10-25 20:57 UTC #17 lore raw

PATCH 9/39 resize2fs: don't exit if shrinking sparse_super2 fs to one bg

If we're shrinking a sparse_super2 filesystem to a single block group, the superblock will be in block 0. This is perfectly valid (for block group 0 with a blocksize > 1024) so don't exit.

---
 resize/resize2fs.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

resize/resize2fs.c +1 -1

index b59f482..ff18817 100644
@@ -2126,7 +2126,7 @@ static errcode_t reserve_sparse_super2_last_group(ext2_resize_t rfs,
 	if (retval)
 		return retval;
 
-	if (!sb) {
+	if (last_bg && !sb) {
 		fputs(_("Should never happen!  No sb in last super_sparse bg?\n"),
 		      stderr);
 		exit(1);

Darrick J. Wong

author

2014-10-25 20:58 UTC #18 lore raw

PATCH 10/39 resize2fs: quickly rewrite extent blocks when moving an inode w/ metadata_csum

When we're moving an inode on a metadata_csum filesystem, we need to rewrite the checksum of all interior nodes of the extent tree. The current code does this inefficiently via set_bmap, but we can do this more efficiently through direct iteration of the extent tree.

---
 resize/resize2fs.c |   74 +++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 64 insertions(+), 10 deletions(-)

resize/resize2fs.c +64 ~~-10~~

index ff18817..212fee2 100644
@@ -1492,16 +1492,6 @@ static int process_block(ext2_filsys fs, blk64_t	*block_nr,
 		}
 	}
 
-	/*
-	 * If we moved inodes and metadata_csum is enabled, we must force the
-	 * extent block to be rewritten with new checksum.
-	 */
-	if (EXT2_HAS_RO_COMPAT_FEATURE(fs->super,
-				       EXT4_FEATURE_RO_COMPAT_METADATA_CSUM) &&
-	    pb->has_extents &&
-	    pb->old_ino != pb->ino)
-		ret |= BLOCK_CHANGED;
-
 	if (pb->is_dir) {
 		retval = ext2fs_add_dir_block2(fs->dblist, pb->ino,
 					       block, (int) blockcnt);

@@ -1581,6 +1571,59 @@ out:
 	return err;
 }
 
+/* Rewrite extents */
+static errcode_t rewrite_extents(ext2_filsys fs, ext2_ino_t ino)
+{
+	ext2_extent_handle_t	handle;
+	struct ext2fs_extent	extent;
+	errcode_t		errcode;
+	struct ext2_extent_info	info;
+
+	errcode = ext2fs_extent_open(fs, ino, &handle);
+	if (errcode)
+		return errcode;
+
+	errcode = ext2fs_extent_get(handle, EXT2_EXTENT_ROOT, &extent);
+	if (errcode)
+		goto out;
+
+	do {
+		errcode = ext2fs_extent_get_info(handle, &info);
+		if (errcode)
+			break;
+
+		/*
+		 * If this is the first extent in an extent block that we
+		 * haven't visited, rewrite the extent to force the ETB
+		 * checksum to be rewritten.
+		 */
+		if (info.curr_entry == 1 && info.curr_level != 0 &&
+		    !(extent.e_flags & EXT2_EXTENT_FLAGS_SECOND_VISIT)) {
+			errcode = ext2fs_extent_replace(handle, 0, &extent);
+			if (errcode)
+				break;
+		}
+
+		/* Skip to the end of a block of leaf nodes */
+		if (extent.e_flags & EXT2_EXTENT_FLAGS_LEAF) {
+			errcode = ext2fs_extent_get(handle,
+						    EXT2_EXTENT_LAST_SIB,
+						    &extent);
+			if (errcode)
+				break;
+		}
+
+		errcode = ext2fs_extent_get(handle, EXT2_EXTENT_NEXT, &extent);
+	} while (errcode == 0);
+
+out:
+	/* Ok if we run off the end */
+	if (errcode == EXT2_ET_EXTENT_NO_NEXT)
+		errcode = 0;
+	ext2fs_extent_free(handle);
+	return errcode;
+}
+
 static errcode_t inode_scan_and_fix(ext2_resize_t rfs)
 {
 	struct process_block_struct	pb;

@@ -1699,6 +1742,17 @@ remap_blocks:
 		if (retval)
 			goto errout;
 
+		/* Rewrite extent block checksums with new inode number */
+		if (EXT2_HAS_RO_COMPAT_FEATURE(rfs->old_fs->super,
+				       EXT4_FEATURE_RO_COMPAT_METADATA_CSUM) &&
+		    (inode->i_flags & EXT4_EXTENTS_FL)) {
+			rfs->old_fs->flags |= EXT2_FLAG_IGNORE_CSUM_ERRORS;
+			retval = rewrite_extents(rfs->old_fs, new_inode);
+			rfs->old_fs->flags &= ~EXT2_FLAG_IGNORE_CSUM_ERRORS;
+			if (retval)
+				goto errout;
+		}
+
 		/*
 		 * Update inodes to point to new blocks; schedule directory
 		 * blocks for inode remapping.  Need to write out dir blocks

Darrick J. Wong

author

2014-10-25 20:58 UTC #19 lore raw

PATCH 11/39 resize2fs: use old_fs to detect per-bg metadata blocks to free

When shrinking a filesystem, resize2fs wants to free per-bg metadata blocks that are no longer needed. This behavior is gated on whether there's a superblock in the group as told by new_fs. The check really should be against old_fs, since we're effectively freeing blocks out of old_fs in the transition to new_fs, but prior to sparse_super2 this didn't matter since superblocks didn't move, so it didn't matter.

Under sparse_super2, however, there's a superblock in the last group, so now we need to change the test to use old_fs as it should.

---
 resize/resize2fs.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

resize/resize2fs.c +1 -1

index 212fee2..a8a6850 100644
@@ -1042,7 +1042,7 @@ static errcode_t blocks_to_move(ext2_resize_t rfs)
 		}
 
 		for (i = 0; i < max_groups; i++) {
-			if (!ext2fs_bg_has_super(fs, i)) {
+			if (!ext2fs_bg_has_super(old_fs, i)) {
 				group_blk += fs->super->s_blocks_per_group;
 				continue;
 			}

Darrick J. Wong

author

2014-10-25 20:58 UTC #20 lore raw

PATCH 12/39 resize2fs: don't interpret bitmap shift while crossing flexbg as raid stride

resize2fs tries to infer the RAID stride by observing differences between the locations of adjacent block groups' block and inode bitmaps within the block group. If the two block groups being compared belong to different flexbgs, however, it'll be fooled by the large offset into thinking that the FS has an abnormally large RAID stride.

Therefore, teach it not to get confused by crossing a flexbg.

Reported-by TR Reardon <[email protected]>

---
 resize/main.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

resize/main.c +3 -1

index 983d8c2..e017792 100644
@@ -105,6 +105,7 @@ static void determine_fs_stride(ext2_filsys fs)
 	unsigned long long sum;
 	unsigned int	has_sb, prev_has_sb = 0, num;
 	int		i_stride, b_stride;
+	int		flexbg_size = 1 << fs->super->s_log_groups_per_flex;
 
 	if (fs->stride)
 		return;

@@ -120,7 +121,8 @@ static void determine_fs_stride(ext2_filsys fs)
 			ext2fs_inode_bitmap_loc(fs, group - 1) -
 			fs->super->s_blocks_per_group;
 		if (b_stride != i_stride ||
-		    b_stride < 0)
+		    b_stride < 0 ||
+		    (flexbg_size > 1 && (group % flexbg_size == 0)))
 			goto next;
 
 		/* printf("group %d has stride %d\n", group, b_stride); */

Darrick J. Wong

author

2014-10-25 20:58 UTC #21 lore raw

PATCH 13/39 resize2fs: set block_uninit in former last bg when expanding fs

If expanding a filesystem and the last bg of the old fs has no blocks in use, sets the block_uninit flag. This might save a small amount of time at mount, but will be particularly useful if e2fsck ever starts to verify that block_uninit is set for empty block groups.

---
 resize/resize2fs.c              |   54 +++++++++++++++++++++++++++++++++++++++
 tests/r_fixup_lastbg/expect     |   36 ++++++++++++++++++++++++++
 tests/r_fixup_lastbg/script     |   37 +++++++++++++++++++++++++++
 tests/r_fixup_lastbg_big/expect |   39 ++++++++++++++++++++++++++++
 tests/r_fixup_lastbg_big/script |   37 +++++++++++++++++++++++++++
 5 files changed, 203 insertions(+)
 create mode 100644 tests/r_fixup_lastbg/expect
 create mode 100755 tests/r_fixup_lastbg/script
 create mode 100644 tests/r_fixup_lastbg_big/expect
 create mode 100755 tests/r_fixup_lastbg_big/script

resize/resize2fs.c +54 -0

index a8a6850..e71acf3 100644
@@ -50,6 +50,7 @@ static errcode_t inode_ref_fix(ext2_resize_t rfs);
 static errcode_t move_itables(ext2_resize_t rfs);
 static errcode_t fix_resize_inode(ext2_filsys fs);
 static errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs);
+static errcode_t fixup_last_bg(ext2_resize_t rfs);
 static errcode_t fix_sb_journal_backup(ext2_filsys fs);
 static errcode_t mark_table_blocks(ext2_filsys fs,
 				   ext2fs_block_bitmap bmap);

@@ -180,6 +181,9 @@ errcode_t resize_fs(ext2_filsys fs, blk64_t *new_size, int flags,
 	retval = ext2fs_calculate_summary_stats(rfs->new_fs);
 	if (retval)
 		goto errout;
+	retval = fixup_last_bg(rfs);
+	if (retval)
+		goto errout;
 	print_resource_track(rfs, &rtrack, fs->io);
 
 	init_resource_track(&rtrack, "fix_resize_inode", fs->io);

@@ -2388,6 +2392,56 @@ static errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs)
 }
 
 /*
+ * Fix up the bg flags + ununsed inode count in the last group of the fs if
+ * we're expanding.
+ */
+static errcode_t fixup_last_bg(ext2_resize_t rfs)
+{
+	dgrp_t		group;
+	ext2_ino_t	start, end, j, new_unused;
+	int		dirty = 0;
+	ext2_filsys	fs = rfs->new_fs;
+
+	if (rfs->old_fs->group_desc_count > rfs->new_fs->group_desc_count ||
+	    !ext2fs_has_group_desc_csum(fs))
+		return 0;
+
+	group = rfs->old_fs->group_desc_count - 1;
+	if (fs->super->s_blocks_per_group ==
+	    ext2fs_bg_free_blocks_count(fs, group)) {
+		ext2fs_bg_flags_set(fs, group, EXT2_BG_BLOCK_UNINIT);
+		dirty = 1;
+	}
+
+	if (fs->super->s_inodes_per_group ==
+	    ext2fs_bg_free_inodes_count(fs, group)) {
+		ext2fs_bg_itable_unused_set(fs, group,
+					    fs->super->s_inodes_per_group);
+		ext2fs_bg_flags_set(fs, group, EXT2_BG_INODE_UNINIT);
+		dirty = 1;
+	} else {
+		start = (fs->super->s_inodes_per_group * group) + 1;
+		end = (fs->super->s_inodes_per_group * (group + 1));
+		end -= ext2fs_bg_itable_unused(fs, group);
+		for (j = end; j >= start; j--)
+			if (ext2fs_test_inode_bitmap2(fs->inode_map, j))
+				break;
+
+		new_unused = fs->super->s_inodes_per_group - (j - start + 1);
+		if (ext2fs_bg_itable_unused(fs, group) != new_unused) {
+			ext2fs_bg_itable_unused_set(fs, group, new_unused);
+			dirty = 1;
+		}
+	}
+
+	if (dirty) {
+		ext2fs_group_desc_csum_set(fs, group);
+		ext2fs_mark_super_dirty(fs);
+	}
+	return 0;
+}
+
+/*
  *  Journal may have been relocated; update the backup journal blocks
  *  in the superblock.
  */

tests/r_fixup_lastbg/expect new file +36 -0

new file mode 100644
index 0000000..f9945b3
@@ -0,0 +1,36 @@
+Creating filesystem with 20000 1k blocks and 1248 inodes
+Superblock backups stored on blocks: 
+	8193
+
+Allocating group tables:    done                            
+Writing inode tables:    done                            
+Creating journal (1024 blocks): done
+Writing superblocks and filesystem accounting information:    done
+
+Group 2: (Blocks 16385-19999) [INODE_UNINIT, ITABLE_ZEROED]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Group 2: (Blocks 16385-19999)
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Resizing the filesystem on test.img to 20004 (1k) blocks.
+The filesystem on test.img is now 20004 (1k) blocks long.
+
+Group 2: (Blocks 16385-20003) [INODE_UNINIT]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3619 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-20003
+  Free inodes: 833-1248
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test.img: 11/1248 files (0.0% non-contiguous), 1517/20004 blocks

tests/r_fixup_lastbg/script new file +37 -0

new file mode 100755
index 0000000..ad223d4
@@ -0,0 +1,37 @@
+if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+test_description="fix up last bg when expanding within the last bg"
+
+EXP=$test_dir/expect
+OUT=$test_name.out
+LOG=$test_name.log
+E2FSCK=../e2fsck/e2fsck
+
+$MKE2FS -T ext4 -b 1024 -F -U 56d3ee50-8532-4f29-8181-d7c6ea4a94a6 $TMPFILE 20000 > $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$DEBUGFS -R "set_bg 2 itable_unused 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 flags 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 checksum 0xd318" -w $TMPFILE > /dev/null 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+dd if=/dev/zero of=$TMPFILE bs=1 count=1 seek=$((1024 * 20004)) conv=notrunc >> $OUT 2> /dev/null
+$RESIZE2FS_EXE -f -p $TMPFILE >> $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$E2FSCK -fy $TMPFILE >> $OUT 2>&1
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" < $OUT > $LOG
+rm -rf $OUT
+
+cmp -s $LOG $EXP
+RC=$?
+if [ $RC -eq 0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff -u $EXP $LOG > $test_name.failed
+fi
+
+unset EXP LOG OUT E2FSCK
+
+else #if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi

tests/r_fixup_lastbg_big/expect new file +39 -0

new file mode 100644
index 0000000..1aa36fc
@@ -0,0 +1,39 @@
+Creating filesystem with 20000 1k blocks and 1248 inodes
+Superblock backups stored on blocks: 
+	8193
+
+Allocating group tables:    done                            
+Writing inode tables:    done                            
+Creating journal (1024 blocks): done
+Writing superblocks and filesystem accounting information:    done
+
+Group 2: (Blocks 16385-19999) [INODE_UNINIT, ITABLE_ZEROED]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Group 2: (Blocks 16385-19999)
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Resizing the filesystem on test.img to 40000 (1k) blocks.
+The filesystem on test.img is now 40000 (1k) blocks long.
+
+Group 2: (Blocks 16385-24576) [INODE_UNINIT, BLOCK_UNINIT]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  8192 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-24576
+  Free inodes: 833-1248
+Group 3: (Blocks 24577-32768) [INODE_UNINIT]
+  Backup superblock at 24577, Group descriptors at 24578-24578
+  Reserved GDT blocks at 24579-24656
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test.img: 11/2080 files (0.0% non-contiguous), 1809/40000 blocks

tests/r_fixup_lastbg_big/script new file +37 -0

new file mode 100755
index 0000000..4c4a351
@@ -0,0 +1,37 @@
+if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+test_description="fix up last bg when expanding beyond the last bg"
+
+EXP=$test_dir/expect
+OUT=$test_name.out
+LOG=$test_name.log
+E2FSCK=../e2fsck/e2fsck
+
+$MKE2FS -T ext4 -b 1024 -F -U 56d3ee50-8532-4f29-8181-d7c6ea4a94a6 $TMPFILE 20000 > $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$DEBUGFS -R "set_bg 2 itable_unused 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 flags 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 checksum 0xd318" -w $TMPFILE > /dev/null 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+dd if=/dev/zero of=$TMPFILE bs=1 count=1 seek=$((1024 * 40000)) conv=notrunc >> $OUT 2> /dev/null
+$RESIZE2FS_EXE -f -p $TMPFILE >> $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$E2FSCK -fy $TMPFILE >> $OUT 2>&1
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" < $OUT > $LOG
+rm -rf $OUT
+
+cmp -s $LOG $EXP
+RC=$?
+if [ $RC -eq 0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff -u $EXP $LOG > $test_name.failed
+fi
+
+unset EXP LOG OUT E2FSCK
+
+else #if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi

Darrick J. Wong

author

2014-10-27 23:29 UTC #22 lore raw

PATCH v2 13/39 resize2fs: set bg flags and unused inode count when resizing

Recalculate the unused inode count and the block/inode uninit flags when resizing a filesystem. This can speed up future e2fsck runs considerably and will reduce mount times.

v2: Use a library function to take care of this.

---
 resize/resize2fs.c              |    4 ++++
 tests/r_fixup_lastbg/expect     |   36 ++++++++++++++++++++++++++++++++++++
 tests/r_fixup_lastbg/script     |   37 +++++++++++++++++++++++++++++++++++++
 tests/r_fixup_lastbg_big/expect |   39 +++++++++++++++++++++++++++++++++++++++
 tests/r_fixup_lastbg_big/script |   37 +++++++++++++++++++++++++++++++++++++
 5 files changed, 153 insertions(+)
 create mode 100644 tests/r_fixup_lastbg/expect
 create mode 100755 tests/r_fixup_lastbg/script
 create mode 100644 tests/r_fixup_lastbg_big/expect
 create mode 100755 tests/r_fixup_lastbg_big/script

resize/resize2fs.c +4 -0

index a8a6850..6d96a35 100644
@@ -198,6 +198,10 @@ errcode_t resize_fs(ext2_filsys fs, blk64_t *new_size, int flags,
 	if (retval)
 		goto errout;
 
+	retval = ext2fs_set_gdt_csum(rfs->new_fs);
+	if (retval)
+		goto errout;
+
 	rfs->new_fs->super->s_state &= ~EXT2_ERROR_FS;
 	rfs->new_fs->flags &= ~EXT2_FLAG_MASTER_SB_ONLY;

tests/r_fixup_lastbg/expect new file +36 -0

new file mode 100644
index 0000000..f9945b3
@@ -0,0 +1,36 @@
+Creating filesystem with 20000 1k blocks and 1248 inodes
+Superblock backups stored on blocks: 
+	8193
+
+Allocating group tables:    done                            
+Writing inode tables:    done                            
+Creating journal (1024 blocks): done
+Writing superblocks and filesystem accounting information:    done
+
+Group 2: (Blocks 16385-19999) [INODE_UNINIT, ITABLE_ZEROED]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Group 2: (Blocks 16385-19999)
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Resizing the filesystem on test.img to 20004 (1k) blocks.
+The filesystem on test.img is now 20004 (1k) blocks long.
+
+Group 2: (Blocks 16385-20003) [INODE_UNINIT]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3619 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-20003
+  Free inodes: 833-1248
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test.img: 11/1248 files (0.0% non-contiguous), 1517/20004 blocks

tests/r_fixup_lastbg/script new file +37 -0

new file mode 100755
index 0000000..ad223d4
@@ -0,0 +1,37 @@
+if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+test_description="fix up last bg when expanding within the last bg"
+
+EXP=$test_dir/expect
+OUT=$test_name.out
+LOG=$test_name.log
+E2FSCK=../e2fsck/e2fsck
+
+$MKE2FS -T ext4 -b 1024 -F -U 56d3ee50-8532-4f29-8181-d7c6ea4a94a6 $TMPFILE 20000 > $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$DEBUGFS -R "set_bg 2 itable_unused 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 flags 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 checksum 0xd318" -w $TMPFILE > /dev/null 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+dd if=/dev/zero of=$TMPFILE bs=1 count=1 seek=$((1024 * 20004)) conv=notrunc >> $OUT 2> /dev/null
+$RESIZE2FS_EXE -f -p $TMPFILE >> $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$E2FSCK -fy $TMPFILE >> $OUT 2>&1
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" < $OUT > $LOG
+rm -rf $OUT
+
+cmp -s $LOG $EXP
+RC=$?
+if [ $RC -eq 0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff -u $EXP $LOG > $test_name.failed
+fi
+
+unset EXP LOG OUT E2FSCK
+
+else #if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi

tests/r_fixup_lastbg_big/expect new file +39 -0

new file mode 100644
index 0000000..1aa36fc
@@ -0,0 +1,39 @@
+Creating filesystem with 20000 1k blocks and 1248 inodes
+Superblock backups stored on blocks: 
+	8193
+
+Allocating group tables:    done                            
+Writing inode tables:    done                            
+Creating journal (1024 blocks): done
+Writing superblocks and filesystem accounting information:    done
+
+Group 2: (Blocks 16385-19999) [INODE_UNINIT, ITABLE_ZEROED]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Group 2: (Blocks 16385-19999)
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  3615 free blocks, 416 free inodes, 0 directories
+  Free blocks: 16385-19999
+  Free inodes: 833-1248
+Resizing the filesystem on test.img to 40000 (1k) blocks.
+The filesystem on test.img is now 40000 (1k) blocks long.
+
+Group 2: (Blocks 16385-24576) [INODE_UNINIT, BLOCK_UNINIT]
+  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Inode table at 295-398 (bg #0 + 294)
+  8192 free blocks, 416 free inodes, 0 directories, 416 unused inodes
+  Free blocks: 16385-24576
+  Free inodes: 833-1248
+Group 3: (Blocks 24577-32768) [INODE_UNINIT]
+  Backup superblock at 24577, Group descriptors at 24578-24578
+  Reserved GDT blocks at 24579-24656
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test.img: 11/2080 files (0.0% non-contiguous), 1809/40000 blocks

tests/r_fixup_lastbg_big/script new file +37 -0

new file mode 100755
index 0000000..4c4a351
@@ -0,0 +1,37 @@
+if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+test_description="fix up last bg when expanding beyond the last bg"
+
+EXP=$test_dir/expect
+OUT=$test_name.out
+LOG=$test_name.log
+E2FSCK=../e2fsck/e2fsck
+
+$MKE2FS -T ext4 -b 1024 -F -U 56d3ee50-8532-4f29-8181-d7c6ea4a94a6 $TMPFILE 20000 > $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$DEBUGFS -R "set_bg 2 itable_unused 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 flags 0" -w $TMPFILE > /dev/null 2>&1
+$DEBUGFS -R "set_bg 2 checksum 0xd318" -w $TMPFILE > /dev/null 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+dd if=/dev/zero of=$TMPFILE bs=1 count=1 seek=$((1024 * 40000)) conv=notrunc >> $OUT 2> /dev/null
+$RESIZE2FS_EXE -f -p $TMPFILE >> $OUT 2>&1
+$DUMPE2FS $TMPFILE 2>&1 | grep -A10 '^Group 2:' >> $OUT
+$E2FSCK -fy $TMPFILE >> $OUT 2>&1
+
+sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" < $OUT > $LOG
+rm -rf $OUT
+
+cmp -s $LOG $EXP
+RC=$?
+if [ $RC -eq 0 ]; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff -u $EXP $LOG > $test_name.failed
+fi
+
+unset EXP LOG OUT E2FSCK
+
+else #if test -x $RESIZE2FS_EXE -a -x $DEBUGFS_EXE; then
+	echo "$test_name: $test_description: skipped"
+fi

Darrick J. Wong

author

2014-10-25 20:58 UTC #23 lore raw

PATCH 14/39 mke2fs: don't zero inode table blocks that are already zeroed

At mke2fs time, if we discard the device and discard zeroes data, don't bother zeroing the inode table blocks a second time.

---
 misc/mke2fs.c |   14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

misc/mke2fs.c +8 -6

index a7bd35a..c6a2c2f 100644
@@ -421,12 +421,14 @@ static void write_inode_tables(ext2_filsys fs, int lazy_flag, int itable_zeroed)
 			ext2fs_bg_flags_set(fs, i, EXT2_BG_INODE_ZEROED);
 			ext2fs_group_desc_csum_set(fs, i);
 		}
-		retval = ext2fs_zero_blocks2(fs, blk, num, &blk, &num);
-		if (retval) {
-			fprintf(stderr, _("\nCould not write %d "
-				  "blocks in inode table starting at %llu: %s\n"),
-				num, blk, error_message(retval));
-			exit(1);
+		if (!itable_zeroed) {
+			retval = ext2fs_zero_blocks2(fs, blk, num, &blk, &num);
+			if (retval) {
+				fprintf(stderr, _("\nCould not write %d "
+					  "blocks in inode table starting at %llu: %s\n"),
+					num, blk, error_message(retval));
+				exit(1);
+			}
 		}
 		if (sync_kludge) {
 			if (sync_kludge == 1)

Darrick J. Wong

author

2014-10-25 20:58 UTC #24 lore raw

PATCH 15/39 dumpe2fs: 80 column outputs, please

Try to reduce dumpe2fs output to 80 columns or less.

---
 misc/dumpe2fs.c                                    |    6 +++++-
 tests/f_jnl_errno/expect.0                         |    3 ++-
 tests/j_short_revoke_trans/script                  |    2 +-
 tests/j_short_revoke_trans_mcsum_64bit/script      |    2 +-
 tests/j_short_trans/script                         |    2 +-
 tests/j_short_trans_64bit/script                   |    2 +-
 tests/j_short_trans_mcsum_64bit/script             |    2 +-
 tests/j_short_trans_old_csum/script                |    2 +-
 tests/j_short_trans_open_recover/script            |    2 +-
 tests/j_short_trans_recover/script                 |    2 +-
 tests/j_short_trans_recover_mcsum_64bit/script     |    2 +-
 tests/j_short_uncommitted_trans/script             |    2 +-
 tests/j_short_uncommitted_trans_mcsum_64bit/script |    2 +-
 tests/m_64bit_flexbg/expect.1                      |    3 ++-
 tests/m_bigjournal/script                          |    3 +++
 tests/m_dasd_bs/expect.1                           |    6 ++++--
 tests/m_desc_size_128/script                       |    3 +++
 tests/m_extent_journal/script                      |    3 +++
 tests/m_large_file/expect.1                        |    3 ++-
 tests/m_meta_bg/script                             |    3 +++
 tests/m_mmp/expect.1                               |    6 ++++--
 tests/m_no_opt/script                              |    3 +++
 tests/m_quota/script                               |    3 +++
 tests/m_raid_opt/script                            |    3 +++
 tests/m_root_owner/expect.1                        |    3 ++-
 tests/m_std/script                                 |    3 +++
 tests/m_uninit/script                              |    3 +++
 tests/r_fixup_lastbg/expect                        |    9 ++++++---
 tests/r_fixup_lastbg_big/expect                    |   10 ++++++----
 tests/r_move_itable/script                         |    8 ++++----
 tests/r_resize_inode/script                        |    6 +++---
 31 files changed, 78 insertions(+), 34 deletions(-)

misc/dumpe2fs.c +5 -1

index e565f2d..f39cb3c 100644
@@ -264,7 +264,11 @@ static void list_desc(ext2_filsys fs, int grp_only)
 		    EXT4_FEATURE_RO_COMPAT_METADATA_CSUM)
 			printf(_(", csum 0x%08x"),
 			       ext2fs_block_bitmap_checksum(fs, i));
-		fputs(_(", Inode bitmap at "), stdout);
+		if (getenv("DUMPE2FS_IGNORE_80COL"))
+			fputs(_(","), stdout);
+		else
+			fputs(_("\n "), stdout);
+		fputs(_(" Inode bitmap at "), stdout);
 		print_number(ext2fs_inode_bitmap_loc(fs, i));
 		print_bg_rel_offset(fs, ext2fs_inode_bitmap_loc(fs, i), 0,
 				    first_block, last_block);

tests/f_jnl_errno/expect.0 +2 -1

index 6dad72a..2a9426d 100644
@@ -40,7 +40,8 @@ Journal start:            0
 Group 0: (Blocks 1-8191) [ITABLE_ZEROED]
   Primary superblock at 1, Group descriptors at 2-2
   Reserved GDT blocks at 3-33
-  Block bitmap at 34 (+33), Inode bitmap at 50 (+49)
+  Block bitmap at 34 (+33)
+  Inode bitmap at 50 (+49)
   Inode table at 66-321 (+65)
   6862 free blocks, 2037 free inodes, 2 directories, 2037 unused inodes
   Free blocks: 1330-8191

tests/j_short_revoke_trans/script +1 -1

index edf28f9..4c8a7a9 100644
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 echo "debugfs write journal" >> $OUT
 echo "jo" > $TMPFILE.cmd

tests/j_short_revoke_trans_mcsum_64bit/script +1 -1

index 8bd6e08..4c735dc 100644
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 $DUMPE2FS $TMPFILE 2>&1 | grep '^Journal features:' >> $OUT

tests/j_short_trans/script +1 -1

index 00f4d6e..d03a548 100644
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 echo "debugfs write journal" >> $OUT
 echo "jo" > $TMPFILE.cmd

tests/j_short_trans_64bit/script +1 -1

index daf0866..cb4af22 100644
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 $DUMPE2FS $TMPFILE 2>&1 | grep '^Journal features:' >> $OUT

tests/j_short_trans_mcsum_64bit/script +1 -1

index 0534941..eab3c17 100644
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 $DUMPE2FS $TMPFILE 2>&1 | grep '^Journal features:' >> $OUT

tests/j_short_trans_old_csum/script +1 -1

index b968bc1..8d1d036 100644
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 $DUMPE2FS $TMPFILE 2>&1 | grep '^Journal features:' >> $OUT

tests/j_short_trans_open_recover/script +1 -1

index fbbb714..7a7fa73 100644
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 echo "debugfs write journal" >> $OUT
 echo "jo" > $TMPFILE.cmd

tests/j_short_trans_recover/script +1 -1

index 7fa6158..9d30f97 100644
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 echo "debugfs write journal" >> $OUT
 echo "jo" > $TMPFILE.cmd

tests/j_short_trans_recover_mcsum_64bit/script +1 -1

index c283fe2..5b0246e 100644
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 $DUMPE2FS $TMPFILE 2>&1 | grep '^Journal features:' >> $OUT

tests/j_short_uncommitted_trans/script +1 -1

index 207d746..583c19d 100644
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 echo "debugfs write journal" >> $OUT
 echo "jo" > $TMPFILE.cmd

tests/j_short_uncommitted_trans_mcsum_64bit/script +1 -1

index de506be..969372e 100644
@@ -19,7 +19,7 @@ echo Exit status is $status >> $OUT.new
 sed -f $cmd_dir/filter.sed -e "s;$TMPFILE;test.img;" $OUT.new >> $OUT
 rm -f $OUT.new
 
-bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*Block bitmap at \([0-9]*\) .*Inode bitmap at \([0-9]*\).*$/\1,\2/g' | tr '\n' ',')"
+bitmaps="$($DUMPE2FS $TMPFILE 2>&1 | grep 'bitmap at' | sed -e 's/^.*bitmap at \([0-9]*\).*$/\1/g' | tr '\n' ',')"
 
 $DUMPE2FS $TMPFILE 2>&1 | grep '^Journal features:' >> $OUT

tests/m_64bit_flexbg/expect.1 +2 -1

index 3635318..cfa3bc9 100644
@@ -48,7 +48,8 @@ Default directory hash:   half_md4
 Group 0: (Blocks 1-1023)
   Primary superblock at 1, Group descriptors at 2-2
   Reserved GDT blocks at 3-9
-  Block bitmap at 10 (+9), Inode bitmap at 26 (+25)
+  Block bitmap at 10 (+9)
+  Inode bitmap at 26 (+25)
   Inode table at 42-57 (+41)
   982 free blocks, 117 free inodes, 2 directories
   Free blocks: 24-25, 28-41, 58-1023

tests/m_bigjournal/script +3 -0

index 4c1ed9a..576d99e 100644
@@ -1,8 +1,11 @@
 DESCRIPTION="journal over 4GB in size"
 FS_SIZE=11000000
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 MKE2FS_OPTS="-t ext4 -G 512 -N 1280 -J size=5000 -q -E lazy_journal_init,lazy_itable_init,nodiscard"
 if [ $(uname -s) = "Darwin" ]; then
 	echo "$test_name: $DESCRIPTION: skipped for HFS+ (no sparse files)"
 	return 0
 fi
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL

tests/m_dasd_bs/expect.1 +4 -2

index 75a401d..0e55e8f 100644
@@ -48,7 +48,8 @@ Default directory hash:   half_md4
 Group 0: (Blocks 0-16383)
   Primary superblock at 0, Group descriptors at 1-1
   Reserved GDT blocks at 2-32
-  Block bitmap at 33 (+33), Inode bitmap at 34 (+34)
+  Block bitmap at 33 (+33)
+  Inode bitmap at 34 (+34)
   Inode table at 35-546 (+35)
   15827 free blocks, 8181 free inodes, 2 directories
   Free blocks: 557-16383

@@ -56,7 +57,8 @@ Group 0: (Blocks 0-16383)
 Group 1: (Blocks 16384-32767)
   Backup superblock at 16384, Group descriptors at 16385-16385
   Reserved GDT blocks at 16386-16416
-  Block bitmap at 16417 (+33), Inode bitmap at 16418 (+34)
+  Block bitmap at 16417 (+33)
+  Inode bitmap at 16418 (+34)
   Inode table at 16419-16930 (+35)
   15837 free blocks, 8192 free inodes, 0 directories
   Free blocks: 16931-32767

tests/m_desc_size_128/script +3 -0

index de3def9..a4def21 100644
@@ -1,4 +1,7 @@
 DESCRIPTION="enable 128-byte group descriptor on mkfs"
 FS_SIZE=131072
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 MKE2FS_OPTS="-b 1024 -O 64bit,extents -g 1024 -N 8192 -E desc_size=128"
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL

tests/m_extent_journal/script +3 -0

index 5e0cac2..efade21 100644
@@ -1,4 +1,7 @@
 DESCRIPTION="extent-mapped journal"
 FS_SIZE=65536
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 MKE2FS_OPTS="-O extents -j"
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL

tests/m_large_file/expect.1 +2 -1

index 8ab2711..4acca41 100644
@@ -48,7 +48,8 @@ Default directory hash:   half_md4
 Group 0: (Blocks 0-16383)
   Primary superblock at 0, Group descriptors at 1-1
   Reserved GDT blocks at 2-4
-  Block bitmap at 5 (+5), Inode bitmap at 6 (+6)
+  Block bitmap at 5 (+5)
+  Inode bitmap at 6 (+6)
   Inode table at 7-10 (+7)
   16367 free blocks, 53 free inodes, 2 directories
   Free blocks: 17-16383

tests/m_meta_bg/script +3 -0

index 7ca2bea..5e285b4 100644
@@ -1,4 +1,7 @@
 DESCRIPTION="meta blockgroup feature"
 FS_SIZE=131072
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 MKE2FS_OPTS="-O meta_bg,sparse_super,^resize_inode -g 1024"
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL

tests/m_mmp/expect.1 +4 -2

index aec0a84..a1452e6 100644
@@ -51,7 +51,8 @@ MMP update interval:      5
 Group 0: (Blocks 0-32767)
   Primary superblock at 0, Group descriptors at 1-1
   Reserved GDT blocks at 2-16
-  Block bitmap at 17 (+17), Inode bitmap at 18 (+18)
+  Block bitmap at 17 (+17)
+  Inode bitmap at 18 (+18)
   Inode table at 19-1042 (+19)
   31718 free blocks, 32757 free inodes, 2 directories
   Free blocks: 1050-32767

@@ -59,7 +60,8 @@ Group 0: (Blocks 0-32767)
 Group 1: (Blocks 32768-65535)
   Backup superblock at 32768, Group descriptors at 32769-32769
   Reserved GDT blocks at 32770-32784
-  Block bitmap at 32785 (+17), Inode bitmap at 32786 (+18)
+  Block bitmap at 32785 (+17)
+  Inode bitmap at 32786 (+18)
   Inode table at 32787-33810 (+19)
   31725 free blocks, 32768 free inodes, 0 directories
   Free blocks: 33811-65535

tests/m_no_opt/script +3 -0

index 6d1d0eb..223f117 100644
@@ -1,4 +1,7 @@
 DESCRIPTION="no filesystem extensions"
 FS_SIZE=65536
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 MKE2FS_OPTS="-O ^sparse_super,^filetype,^resize_inode,^dir_index,^ext_attr"
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL

tests/m_quota/script +3 -0

index 36ab630..fe63939 100644
@@ -5,4 +5,7 @@ if [ "$QUOTA" != "y" ]; then
 	echo "$test_name: $DESCRIPTION: skipped"
 	return 0
 fi
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL

tests/m_raid_opt/script +3 -0

index 296fe94..8c859e0 100644
@@ -1,4 +1,7 @@
 DESCRIPTION="raid options"
 FS_SIZE=131072
 MKE2FS_OPTS="-E stride=13 -O sparse_super -g 1024"
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL

tests/m_root_owner/expect.1 +2 -1

index cd94993..30d119e 100644
@@ -46,7 +46,8 @@ Default directory hash:   half_md4
 Group 0: (Blocks 1-1023)
   Primary superblock at 1, Group descriptors at 2-2
   Reserved GDT blocks at 3-5
-  Block bitmap at 6 (+5), Inode bitmap at 7 (+6)
+  Block bitmap at 6 (+5)
+  Inode bitmap at 7 (+6)
   Inode table at 8-23 (+7)
   986 free blocks, 117 free inodes, 2 directories
   Free blocks: 38-1023

tests/m_std/script +3 -0

index a2f2cda..61c50c7 100644
@@ -1,3 +1,6 @@
 DESCRIPTION="standard filesystem options"
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 FS_SIZE=65536
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL

tests/m_uninit/script +3 -0

index 0b565ac..d4d373a 100644
@@ -1,4 +1,7 @@
 DESCRIPTION="uninitialized group feature"
+DUMPE2FS_IGNORE_80COL=1
+export DUMPE2FS_IGNORE_80COL
 FS_SIZE=131072
 MKE2FS_OPTS="-O uninit_bg"
 . $cmd_dir/run_mke2fs
+unset DUMPE2FS_IGNORE_80COL

tests/r_fixup_lastbg/expect +6 -3

index f9945b3..96b154a 100644
@@ -8,13 +8,15 @@ Creating journal (1024 blocks): done
 Writing superblocks and filesystem accounting information:    done
 
 Group 2: (Blocks 16385-19999) [INODE_UNINIT, ITABLE_ZEROED]
-  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Block bitmap at 83 (bg #0 + 82)
+  Inode bitmap at 86 (bg #0 + 85)
   Inode table at 295-398 (bg #0 + 294)
   3615 free blocks, 416 free inodes, 0 directories, 416 unused inodes
   Free blocks: 16385-19999
   Free inodes: 833-1248
 Group 2: (Blocks 16385-19999)
-  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Block bitmap at 83 (bg #0 + 82)
+  Inode bitmap at 86 (bg #0 + 85)
   Inode table at 295-398 (bg #0 + 294)
   3615 free blocks, 416 free inodes, 0 directories
   Free blocks: 16385-19999

@@ -23,7 +25,8 @@ Resizing the filesystem on test.img to 20004 (1k) blocks.
 The filesystem on test.img is now 20004 (1k) blocks long.
 
 Group 2: (Blocks 16385-20003) [INODE_UNINIT]
-  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Block bitmap at 83 (bg #0 + 82)
+  Inode bitmap at 86 (bg #0 + 85)
   Inode table at 295-398 (bg #0 + 294)
   3619 free blocks, 416 free inodes, 0 directories, 416 unused inodes
   Free blocks: 16385-20003

tests/r_fixup_lastbg_big/expect +6 -4

index 1aa36fc..badf6cf 100644
@@ -8,13 +8,15 @@ Creating journal (1024 blocks): done
 Writing superblocks and filesystem accounting information:    done
 
 Group 2: (Blocks 16385-19999) [INODE_UNINIT, ITABLE_ZEROED]
-  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Block bitmap at 83 (bg #0 + 82)
+  Inode bitmap at 86 (bg #0 + 85)
   Inode table at 295-398 (bg #0 + 294)
   3615 free blocks, 416 free inodes, 0 directories, 416 unused inodes
   Free blocks: 16385-19999
   Free inodes: 833-1248
 Group 2: (Blocks 16385-19999)
-  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Block bitmap at 83 (bg #0 + 82)
+  Inode bitmap at 86 (bg #0 + 85)
   Inode table at 295-398 (bg #0 + 294)
   3615 free blocks, 416 free inodes, 0 directories
   Free blocks: 16385-19999

@@ -23,14 +25,14 @@ Resizing the filesystem on test.img to 40000 (1k) blocks.
 The filesystem on test.img is now 40000 (1k) blocks long.
 
 Group 2: (Blocks 16385-24576) [INODE_UNINIT, BLOCK_UNINIT]
-  Block bitmap at 83 (bg #0 + 82), Inode bitmap at 86 (bg #0 + 85)
+  Block bitmap at 83 (bg #0 + 82)
+  Inode bitmap at 86 (bg #0 + 85)
   Inode table at 295-398 (bg #0 + 294)
   8192 free blocks, 416 free inodes, 0 directories, 416 unused inodes
   Free blocks: 16385-24576
   Free inodes: 833-1248
 Group 3: (Blocks 24577-32768) [INODE_UNINIT]
   Backup superblock at 24577, Group descriptors at 24578-24578
-  Reserved GDT blocks at 24579-24656
 Pass 1: Checking inodes, blocks, and sizes
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity

tests/r_move_itable/script +4 -4

index e567888..3f02a79 100644
@@ -33,7 +33,7 @@ $DEBUGFS -R "set_super_value mkfs_time 0" -w $TMPFILE >/dev/null 2>&1
 $TUNE2FS -c 20 -U clear $TMPFILE  >/dev/null 2>&1
 
 echo dumpe2fs test.img >> $OUT
-$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
+$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed -e '/Block bitmap.*$/N;s/\n  Inode bitmap/, Inode bitmap/g' >> $OUT
 
 echo "--------------------------------" >> $OUT

@@ -53,7 +53,7 @@ $DEBUGFS -R "set_super_value mkfs_time 0" -w $TMPFILE >/dev/null 2>&1
 $TUNE2FS -c 20 -U clear $TMPFILE  >/dev/null 2>&1
 
 echo dumpe2fs test.img >> $OUT
-$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
+$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed -e '/Block bitmap.*$/N;s/\n  Inode bitmap/, Inode bitmap/g' >> $OUT
 
 echo "--------------------------------" >> $OUT

@@ -73,7 +73,7 @@ $DEBUGFS -R "set_super_value mkfs_time 0" -w $TMPFILE >/dev/null 2>&1
 $TUNE2FS -c 20 -U clear $TMPFILE  >/dev/null 2>&1
 
 echo dumpe2fs test.img >> $OUT
-$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
+$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed -e '/Block bitmap.*$/N;s/\n  Inode bitmap/, Inode bitmap/g' >> $OUT
 
 echo "--------------------------------" >> $OUT

@@ -93,7 +93,7 @@ $DEBUGFS -R "set_super_value mkfs_time 0" -w $TMPFILE >/dev/null 2>&1
 $TUNE2FS -c 20 -U clear $TMPFILE  >/dev/null 2>&1
 
 echo dumpe2fs test.img >> $OUT
-$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
+$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed -e '/Block bitmap.*$/N;s/\n  Inode bitmap/, Inode bitmap/g' >> $OUT
 
 rm -f $TMPFILE

tests/r_resize_inode/script +3 -3

index 0f12138..4e3eb19 100644
@@ -33,7 +33,7 @@ $DEBUGFS -R "set_super_value mkfs_time 0" -w $TMPFILE >/dev/null 2>&1
 $TUNE2FS -c 20 -U clear $TMPFILE  >/dev/null 2>&1
 
 echo dumpe2fs test.img >> $OUT
-$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
+$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed -e '/Block bitmap.*$/N;s/\n  Inode bitmap/, Inode bitmap/g' >> $OUT
 
 echo "--------------------------------" >> $OUT

@@ -59,7 +59,7 @@ $DEBUGFS -R "set_super_value mkfs_time 0" -w $TMPFILE >/dev/null 2>&1
 $TUNE2FS -c 20 -U clear $TMPFILE  >/dev/null 2>&1
 
 echo dumpe2fs test.img >> $OUT
-$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
+$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed -e '/Block bitmap.*$/N;s/\n  Inode bitmap/, Inode bitmap/g' >> $OUT
 
 echo "--------------------------------" >> $OUT

@@ -79,7 +79,7 @@ $DEBUGFS -R "set_super_value mkfs_time 0" -w $TMPFILE >/dev/null 2>&1
 $TUNE2FS -c 20 -U clear $TMPFILE  >/dev/null 2>&1
 
 echo dumpe2fs test.img >> $OUT
-$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed >> $OUT
+$DUMPE2FS $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed -e '/Block bitmap.*$/N;s/\n  Inode bitmap/, Inode bitmap/g' >> $OUT
 
 rm -f $TMPFILE
 cmp -s $OUT $EXP

Darrick J. Wong

author

2014-10-25 20:58 UTC #25 lore raw

PATCH 16/39 dumpe2fs: output cleanup

Don't display unused inodes twice, and make it clear that we're printing a descriptor checksum.

Cc TR Reardon <[email protected]>

---
 misc/dumpe2fs.c                 |    8 +++-----
 tests/filter.sed                |    3 ++-
 tests/r_fixup_lastbg_big/expect |    2 ++
 3 files changed, 7 insertions(+), 6 deletions(-)

misc/dumpe2fs.c +3 -5

index f39cb3c..6751bff 100644
@@ -218,20 +218,18 @@ static void list_desc(ext2_filsys fs, int grp_only)
 			continue;
 		}
 
-		printf (_("Group %lu: (Blocks "), i);
+		printf(_("Group %lu: (Blocks "), i);
 		print_range(first_block, last_block);
 		fputs(")", stdout);
-		print_bg_opts(fs, i);
 		if (ext2fs_has_group_desc_csum(fs)) {
 			unsigned csum = ext2fs_bg_checksum(fs, i);
 			unsigned exp_csum = ext2fs_group_desc_csum(fs, i);
 
-			printf(_("  Checksum 0x%04x"), csum);
+			printf(_(" csum 0x%04x"), csum);
 			if (csum != exp_csum)
 				printf(_(" (EXPECTED 0x%04x)"), exp_csum);
-			printf(_(", unused inodes %u\n"),
-			       ext2fs_bg_itable_unused(fs, i));
 		}
+		print_bg_opts(fs, i);
 		has_super = ((i==0) || super_blk);
 		if (has_super) {
 			printf (_("  %s superblock at "),

tests/filter.sed +2 -1

index 59fad4e..d9a336c 100644
@@ -21,4 +21,5 @@ s/\\015//g
 /Reserved blocks uid:/s/ (user .*)//
 /Reserved blocks gid:/s/ (group .*)//
 /whichever comes first/d
-/^  Checksum /d
+s/, csum 0x\([0-9a-f]*\)//g
+s/ csum 0x\([0-9a-f]*\)//g

tests/r_fixup_lastbg_big/expect +2 -0

index badf6cf..8f302a3 100644
@@ -33,6 +33,8 @@ Group 2: (Blocks 16385-24576) [INODE_UNINIT, BLOCK_UNINIT]
   Free inodes: 833-1248
 Group 3: (Blocks 24577-32768) [INODE_UNINIT]
   Backup superblock at 24577, Group descriptors at 24578-24578
+  Reserved GDT blocks at 24579-24656
+  Block bitmap at 413 (bg #0 + 412)
 Pass 1: Checking inodes, blocks, and sizes
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity

Darrick J. Wong

author

2014-10-25 20:58 UTC #26 lore raw

PATCH 17/39 e2fsck: fix dangling pointer when dir_info array is resized

e2fsck uses an array to store directory usage information during pass 3; the usage context also contains a pointer to the last directory looked up. When expanding the dir_info array, this cache pointer needs to be cleared if the array resize changed the pointer location, or else we'll later walk off the end of this dead pointer.

---
 e2fsck/dirinfo.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

e2fsck/dirinfo.c +4 -1

index 4a9019b..dab5a13 100644
@@ -121,7 +121,7 @@ static void setup_db(e2fsck_t ctx)
 void e2fsck_add_dir_info(e2fsck_t ctx, ext2_ino_t ino, ext2_ino_t parent)
 {
 	struct dir_info_db 	*db;
-	struct dir_info 	*dir, ent;
+	struct dir_info		*dir, ent, *old_array;
 	int			i, j;
 	errcode_t		retval;
 	unsigned long		old_size;

@@ -136,6 +136,7 @@ void e2fsck_add_dir_info(e2fsck_t ctx, ext2_ino_t ino, ext2_ino_t parent)
 	if (ctx->dir_info->count >= ctx->dir_info->size) {
 		old_size = ctx->dir_info->size * sizeof(struct dir_info);
 		ctx->dir_info->size += 10;
+		old_array = ctx->dir_info->array;
 		retval = ext2fs_resize_mem(old_size, ctx->dir_info->size *
 					   sizeof(struct dir_info),
 					   &ctx->dir_info->array);

@@ -147,6 +148,8 @@ void e2fsck_add_dir_info(e2fsck_t ctx, ext2_ino_t ino, ext2_ino_t parent)
 			ctx->dir_info->size -= 10;
 			return;
 		}
+		if (old_array != ctx->dir_info->array)
+			ctx->dir_info->last_lookup = NULL;
 	}
 
 	ent.ino = ino;

Theodore Ts'o

Applied

2014-11-05 16:12 UTC #27 lore raw

Re: [PATCH 17/39] e2fsck: fix dangling pointer when dir_info array is resized

Quoted 8 lines · Darrick J. Wong

e2fsck uses an array to store directory usage information during pass 3; the usage context also contains a pointer to the last directory looked up. When expanding the dir_info array, this cache pointer needs to be cleared if the array resize changed the pointer location, or else we'll later walk off the end of this dead pointer.
Signed-off-by Darrick J. Wong <[email protected]>
Reported-by Sami Liedes <[email protected]>

Applied, thanks.

- Ted

Darrick J. Wong

author

2014-10-25 20:58 UTC #28 lore raw

PATCH 18/39 e2fsck: opportunistically recalculate unused inode count and inode_uninit after pass 5

At the end of pass5, update the unused inode count to cover any newly unused inodes and set inode_uninit if none of the inodes in the group are used. This will save us time on future fsck runs since we don't need to scan those inodes any more.

Since it's not an error for the unused inode count to be smaller than necessary, don't bother if the fs is mounted and we're not fixing anything else.

On an aged filesystem that has experienced at least one massive inode die-off, this has been shown to reduce future e2fsck times by around 10%. The effect is diluted on less heavily used filesystems.

---
 e2fsck/pass5.c                          |   38 +++++++++++++++++++++++++++++++
 e2fsck/problem.c                        |    5 ++++
 e2fsck/problem.h                        |    3 ++
 tests/f_bad_bbitmap/expect.1            |    3 ++
 tests/f_bad_bmap_csum/expect.1          |    3 ++
 tests/f_bad_gdt_csum/expect.1           |    7 +++++-
 tests/f_bad_ibitmap/expect.1            |    3 ++
 tests/f_bad_inode_csum/expect.1         |    3 ++
 tests/f_idata_and_extents/expect.1      |    3 ++
 tests/f_illitable_flexbg/expect.1       |   12 ++++++++++
 tests/f_inlinedata_repair/expect.1      |    3 ++
 tests/f_invalid_extent_symlink/expect.1 |    3 ++
 tests/f_jnl_64bit/expect.1              |    6 +++++
 tests/f_super_bad_csum/expect.1         |    8 +++++--
 tests/f_unused_itable/expect.1          |    6 +++++
 15 files changed, 103 insertions(+), 3 deletions(-)

e2fsck/pass5.c +38 -0

index 64fb7fe..1a225fb 100644
@@ -821,6 +821,44 @@ do_counts:
 			} else
 				ext2fs_unmark_valid(fs);
 		}
+
+		/*
+		 * Opportunistically update the unused inodes count
+		 * and set inode_uninit so we can skip scanning unused
+		 * inodes during future fsck runs.  However, it's not
+		 * an error if the unused inode count is smaller
+		 * than necessary, so don't bother the user if FS is
+		 * mounted and we haven't fixed anything else, to
+		 * minimize unnecessary reboots.
+		 */
+		if (((ctx->options & E2F_OPT_NO) ||
+		     !(ctx->mount_flags & EXT2_MF_MOUNTED) ||
+		     ext2fs_test_changed(fs)) &&
+		    ext2fs_has_group_desc_csum(fs) &&
+		    !ext2fs_bg_flags_test(fs, i, EXT2_BG_INODE_UNINIT)) {
+			ext2_ino_t start, end, j;
+
+			start = (fs->super->s_inodes_per_group * i) + 1;
+			end = (fs->super->s_inodes_per_group * (i + 1));
+			pctx.group = i;
+			pctx.ino = ext2fs_bg_itable_unused(fs, i);
+			end -= pctx.ino;
+			for (j = end; j >= start; j--)
+				if (ext2fs_test_inode_bitmap2(ctx->inode_used_map, j))
+					break;
+
+			pctx.ino2 = fs->super->s_inodes_per_group -
+						(j - start + 1);
+			if (pctx.ino != pctx.ino2 &&
+			    fix_problem(ctx, PR_5_UNUSED_INODES_COUNT_GROUP,
+					&pctx)) {
+				if (pctx.ino2 == fs->super->s_inodes_per_group)
+					ext2fs_bg_flags_set(fs, i,
+							EXT2_BG_INODE_UNINIT);
+				ext2fs_bg_itable_unused_set(fs, i, pctx.ino2);
+				ext2fs_mark_super_dirty(fs);
+			}
+		}
 	}
 	if (free_inodes != fs->super->s_free_inodes_count) {
 		pctx.group = -1;

e2fsck/problem.c +5 -0

index a4da64b..9c2b2e6 100644
@@ -1899,6 +1899,11 @@ static struct e2fsck_problem problem_table[] = {
 	  N_("@g %g @b @B does not match checksum.\n"),
 	  PROMPT_FIX, PR_LATCH_BBITMAP | PR_PREEN_OK },
 
+	/* Unused inode count for group wrong */
+	{ PR_5_UNUSED_INODES_COUNT_GROUP,
+	  N_("Unused @i count wrong for @g #%g (%i, counted=%j).\n"),
+	  PROMPT_FIX, PR_PREEN_OK | PR_NO_OK },
+
 	/* Post-Pass 5 errors */
 
 	/* Recreate journal if E2F_FLAG_JOURNAL_INODE flag is set */

e2fsck/problem.h +3 -0

index 3c28166..6a71fc8 100644
@@ -1139,6 +1139,9 @@ struct problem_context {
 /* Block bitmap checksum does not match */
 #define PR_5_BLOCK_BITMAP_CSUM_INVALID	0x05001B
 
+/* Unused inode count for group wrong */
+#define PR_5_UNUSED_INODES_COUNT_GROUP	0x05001C
+
 /*
  * Post-Pass 5 errors
  */

tests/f_bad_bbitmap/expect.1 +3 -0

index 71ad1bb..af3f632 100644
@@ -9,6 +9,9 @@ Pass 5: Checking group summary information
 Block bitmap differences:  -(8--10) -(12--17) -(19--31)
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=117).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/128 files (0.0% non-contiguous), 18/512 blocks

tests/f_bad_bmap_csum/expect.1 +3 -0

index ca8f77f..7616751 100644
@@ -6,6 +6,9 @@ Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
+Unused inode count wrong for group #0 (0, counted=117).
+Fix? yes
+
 Inode bitmap differences: Group 0 inode bitmap does not match checksum.
 FIXED.
 Block bitmap differences: Group 0 block bitmap does not match checksum.

tests/f_bad_gdt_csum/expect.1 +6 -1

index e14c897..0f496e0 100644
@@ -6,5 +6,10 @@ Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
+Unused inode count wrong for group #0 (0, counted=117).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/128 files (0.0% non-contiguous), 18/512 blocks
-Exit status is 0
+Exit status is 1

tests/f_bad_ibitmap/expect.1 +3 -0

index ea17523..81689e0 100644
@@ -9,6 +9,9 @@ Pass 5: Checking group summary information
 Inode bitmap differences:  -(12--32)
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=117).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/128 files (0.0% non-contiguous), 18/512 blocks

tests/f_bad_inode_csum/expect.1 +3 -0

index b3c628d..b28a349 100644
@@ -117,6 +117,9 @@ Fix? yes
 Free inodes count wrong for group #0 (0, counted=32).
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=32).
+Fix? yes
+
 Free inodes count wrong (0, counted=32).
 Fix? yes

tests/f_idata_and_extents/expect.1 +3 -0

index 7f7fbf3..aba063a 100644
@@ -26,6 +26,9 @@ Fix? yes
 Free inodes count wrong for group #0 (105, counted=106).
 Fix? yes
 
+Unused inode count wrong for group #0 (103, counted=105).
+Fix? yes
+
 Free inodes count wrong (105, counted=106).
 Fix? yes

tests/f_illitable_flexbg/expect.1 +12 -0

index fa42a0f..04ce65b 100644
@@ -18,6 +18,18 @@ Pass 5: Checking group summary information
 Inode bitmap differences:  -(65--128)
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=52).
+Fix? yes
+
+Unused inode count wrong for group #1 (0, counted=64).
+Fix? yes
+
+Unused inode count wrong for group #2 (0, counted=64).
+Fix? yes
+
+Unused inode count wrong for group #3 (0, counted=64).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 12/256 files (0.0% non-contiguous), 31163/32768 blocks

tests/f_inlinedata_repair/expect.1 +3 -0

index faba192..f7eebd3 100644
@@ -69,6 +69,9 @@ Pass 5: Checking group summary information
 Directories count wrong for group #0 (7, counted=8).
 Fix? yes
 
+Unused inode count wrong for group #0 (90, counted=91).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 28/128 files (0.0% non-contiguous), 18/512 blocks

tests/f_invalid_extent_symlink/expect.1 +3 -0

index 7bda0b7..cbb2ac5 100644
@@ -6,6 +6,9 @@ Clear? yes
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
+Unused inode count wrong for group #0 (4, counted=5).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/16 files (9.1% non-contiguous), 21/100 blocks

tests/f_jnl_64bit/expect.1 +6 -0

index e360e2f..915076b 100644
@@ -7,6 +7,12 @@ Pass 5: Checking group summary information
 Free blocks count wrong (14059, counted=12712).
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=2876).
+Fix? yes
+
+Unused inode count wrong for group #1 (1395, counted=3958).
+Fix? yes
+
 Free inodes count wrong (8181, counted=6834).
 Fix? yes

tests/f_super_bad_csum/expect.1 +6 -2

index 25ced5c..a606fab 100644
@@ -5,8 +5,12 @@ Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
-Inode bitmap differences: Group 1 inode bitmap does not match checksum.
-FIXED.
+Unused inode count wrong for group #0 (0, counted=501).
+Fix? yes
+
+Unused inode count wrong for group #1 (0, counted=512).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/1024 files (0.0% non-contiguous), 1557/16384 blocks

tests/f_unused_itable/expect.1 +6 -0

index a4da987..fbc6538 100644
@@ -19,9 +19,15 @@ Pass 5: Checking group summary information
 Free inodes count wrong for group #0 (53, counted=51).
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=51).
+Fix? yes
+
 Free inodes count wrong for group #1 (64, counted=58).
 Fix? yes
 
+Unused inode count wrong for group #1 (0, counted=58).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 19/128 files (0.0% non-contiguous), 165/1000 blocks

Darrick J. Wong

author

2014-10-27 20:18 UTC #29 lore raw

Re: [PATCH 18/39] e2fsck: opportunistically recalculate unused inode count and inode_uninit after pass 5

Quoted 5 lines · Darrick J. Wong

At the end of pass5, update the unused inode count to cover any newly unused inodes and set inode_uninit if none of the inodes in the group are used. This will save us time on future fsck runs since we don't need to scan those inodes any more.

Quoted 7 lines

Since it's not an error for the unused inode count to be smaller than necessary, don't bother if the fs is mounted and we're not fixing anything else.
On an aged filesystem that has experienced at least one massive inode die-off, this has been shown to reduce future e2fsck times by around 10%. The effect is diluted on less heavily used filesystems.

I didn't notice that ext2fs_set_gdt_csum() quietly takes care of this if the filesystem's open for writes. Therefore, I withdraw this patch and apologize for the noise.

However, the BLOCK_UNINIT piece is still needed, so I'll issue a new patch #19.

--D

Quoted 298 lines

---
 e2fsck/pass5.c                          |   38 +++++++++++++++++++++++++++++++
 e2fsck/problem.c                        |    5 ++++
 e2fsck/problem.h                        |    3 ++
 tests/f_bad_bbitmap/expect.1            |    3 ++
 tests/f_bad_bmap_csum/expect.1          |    3 ++
 tests/f_bad_gdt_csum/expect.1           |    7 +++++-
 tests/f_bad_ibitmap/expect.1            |    3 ++
 tests/f_bad_inode_csum/expect.1         |    3 ++
 tests/f_idata_and_extents/expect.1      |    3 ++
 tests/f_illitable_flexbg/expect.1       |   12 ++++++++++
 tests/f_inlinedata_repair/expect.1      |    3 ++
 tests/f_invalid_extent_symlink/expect.1 |    3 ++
 tests/f_jnl_64bit/expect.1              |    6 +++++
 tests/f_super_bad_csum/expect.1         |    8 +++++--
 tests/f_unused_itable/expect.1          |    6 +++++
 15 files changed, 103 insertions(+), 3 deletions(-)

e2fsck/pass5.c +38 -0

index 64fb7fe..1a225fb 100644
@@ -821,6 +821,44 @@ do_counts:
 			} else
 				ext2fs_unmark_valid(fs);
 		}
+
+		/*
+		 * Opportunistically update the unused inodes count
+		 * and set inode_uninit so we can skip scanning unused
+		 * inodes during future fsck runs.  However, it's not
+		 * an error if the unused inode count is smaller
+		 * than necessary, so don't bother the user if FS is
+		 * mounted and we haven't fixed anything else, to
+		 * minimize unnecessary reboots.
+		 */
+		if (((ctx->options & E2F_OPT_NO) ||
+		     !(ctx->mount_flags & EXT2_MF_MOUNTED) ||
+		     ext2fs_test_changed(fs)) &&
+		    ext2fs_has_group_desc_csum(fs) &&
+		    !ext2fs_bg_flags_test(fs, i, EXT2_BG_INODE_UNINIT)) {
+			ext2_ino_t start, end, j;
+
+			start = (fs->super->s_inodes_per_group * i) + 1;
+			end = (fs->super->s_inodes_per_group * (i + 1));
+			pctx.group = i;
+			pctx.ino = ext2fs_bg_itable_unused(fs, i);
+			end -= pctx.ino;
+			for (j = end; j >= start; j--)
+				if (ext2fs_test_inode_bitmap2(ctx->inode_used_map, j))
+					break;
+
+			pctx.ino2 = fs->super->s_inodes_per_group -
+						(j - start + 1);
+			if (pctx.ino != pctx.ino2 &&
+			    fix_problem(ctx, PR_5_UNUSED_INODES_COUNT_GROUP,
+					&pctx)) {
+				if (pctx.ino2 == fs->super->s_inodes_per_group)
+					ext2fs_bg_flags_set(fs, i,
+							EXT2_BG_INODE_UNINIT);
+				ext2fs_bg_itable_unused_set(fs, i, pctx.ino2);
+				ext2fs_mark_super_dirty(fs);
+			}
+		}
 	}
 	if (free_inodes != fs->super->s_free_inodes_count) {
 		pctx.group = -1;

e2fsck/problem.c +5 -0

index a4da64b..9c2b2e6 100644
@@ -1899,6 +1899,11 @@ static struct e2fsck_problem problem_table[] = {
 	  N_("@g %g @b @B does not match checksum.\n"),
 	  PROMPT_FIX, PR_LATCH_BBITMAP | PR_PREEN_OK },
 
+	/* Unused inode count for group wrong */
+	{ PR_5_UNUSED_INODES_COUNT_GROUP,
+	  N_("Unused @i count wrong for @g #%g (%i, counted=%j).\n"),
+	  PROMPT_FIX, PR_PREEN_OK | PR_NO_OK },
+
 	/* Post-Pass 5 errors */
 
 	/* Recreate journal if E2F_FLAG_JOURNAL_INODE flag is set */

e2fsck/problem.h +3 -0

index 3c28166..6a71fc8 100644
@@ -1139,6 +1139,9 @@ struct problem_context {
 /* Block bitmap checksum does not match */
 #define PR_5_BLOCK_BITMAP_CSUM_INVALID	0x05001B
 
+/* Unused inode count for group wrong */
+#define PR_5_UNUSED_INODES_COUNT_GROUP	0x05001C
+
 /*
  * Post-Pass 5 errors
  */

tests/f_bad_bbitmap/expect.1 +3 -0

index 71ad1bb..af3f632 100644
@@ -9,6 +9,9 @@ Pass 5: Checking group summary information
 Block bitmap differences:  -(8--10) -(12--17) -(19--31)
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=117).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/128 files (0.0% non-contiguous), 18/512 blocks

tests/f_bad_bmap_csum/expect.1 +3 -0

index ca8f77f..7616751 100644
@@ -6,6 +6,9 @@ Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
+Unused inode count wrong for group #0 (0, counted=117).
+Fix? yes
+
 Inode bitmap differences: Group 0 inode bitmap does not match checksum.
 FIXED.
 Block bitmap differences: Group 0 block bitmap does not match checksum.

tests/f_bad_gdt_csum/expect.1 +6 -1

index e14c897..0f496e0 100644
@@ -6,5 +6,10 @@ Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
+Unused inode count wrong for group #0 (0, counted=117).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/128 files (0.0% non-contiguous), 18/512 blocks
-Exit status is 0
+Exit status is 1

tests/f_bad_ibitmap/expect.1 +3 -0

index ea17523..81689e0 100644
@@ -9,6 +9,9 @@ Pass 5: Checking group summary information
 Inode bitmap differences:  -(12--32)
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=117).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/128 files (0.0% non-contiguous), 18/512 blocks

tests/f_bad_inode_csum/expect.1 +3 -0

index b3c628d..b28a349 100644
@@ -117,6 +117,9 @@ Fix? yes
 Free inodes count wrong for group #0 (0, counted=32).
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=32).
+Fix? yes
+
 Free inodes count wrong (0, counted=32).
 Fix? yes

tests/f_idata_and_extents/expect.1 +3 -0

index 7f7fbf3..aba063a 100644
@@ -26,6 +26,9 @@ Fix? yes
 Free inodes count wrong for group #0 (105, counted=106).
 Fix? yes
 
+Unused inode count wrong for group #0 (103, counted=105).
+Fix? yes
+
 Free inodes count wrong (105, counted=106).
 Fix? yes

tests/f_illitable_flexbg/expect.1 +12 -0

index fa42a0f..04ce65b 100644
@@ -18,6 +18,18 @@ Pass 5: Checking group summary information
 Inode bitmap differences:  -(65--128)
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=52).
+Fix? yes
+
+Unused inode count wrong for group #1 (0, counted=64).
+Fix? yes
+
+Unused inode count wrong for group #2 (0, counted=64).
+Fix? yes
+
+Unused inode count wrong for group #3 (0, counted=64).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 12/256 files (0.0% non-contiguous), 31163/32768 blocks

tests/f_inlinedata_repair/expect.1 +3 -0

index faba192..f7eebd3 100644
@@ -69,6 +69,9 @@ Pass 5: Checking group summary information
 Directories count wrong for group #0 (7, counted=8).
 Fix? yes
 
+Unused inode count wrong for group #0 (90, counted=91).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 28/128 files (0.0% non-contiguous), 18/512 blocks

tests/f_invalid_extent_symlink/expect.1 +3 -0

index 7bda0b7..cbb2ac5 100644
@@ -6,6 +6,9 @@ Clear? yes
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
+Unused inode count wrong for group #0 (4, counted=5).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/16 files (9.1% non-contiguous), 21/100 blocks

tests/f_jnl_64bit/expect.1 +6 -0

index e360e2f..915076b 100644
@@ -7,6 +7,12 @@ Pass 5: Checking group summary information
 Free blocks count wrong (14059, counted=12712).
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=2876).
+Fix? yes
+
+Unused inode count wrong for group #1 (1395, counted=3958).
+Fix? yes
+
 Free inodes count wrong (8181, counted=6834).
 Fix? yes

tests/f_super_bad_csum/expect.1 +6 -2

index 25ced5c..a606fab 100644
@@ -5,8 +5,12 @@ Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
-Inode bitmap differences: Group 1 inode bitmap does not match checksum.
-FIXED.
+Unused inode count wrong for group #0 (0, counted=501).
+Fix? yes
+
+Unused inode count wrong for group #1 (0, counted=512).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 11/1024 files (0.0% non-contiguous), 1557/16384 blocks

tests/f_unused_itable/expect.1 +6 -1

index a4da987..fbc6538 100644
@@ -19,9 +19,15 @@ Pass 5: Checking group summary information
 Free inodes count wrong for group #0 (53, counted=51).
 Fix? yes
 
+Unused inode count wrong for group #0 (0, counted=51).
+Fix? yes
+
 Free inodes count wrong for group #1 (64, counted=58).
 Fix? yes
 
+Unused inode count wrong for group #1 (0, counted=58).
+Fix? yes
+
 
 test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
 test_filesys: 19/128 files (0.0% non-contiguous), 165/1000 blocks
 
--

To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Darrick J. Wong

author

2014-10-25 20:58 UTC #30 lore raw

PATCH 19/39 e2fsck: opportunistically set block_uninit after pass 5

At the end of pass5, set block_uninit on any group that's empty. For meta_bg filesystems this might save a small amount of disk accesses at load time.

Since it's not an error for the flag not to be set even when it could, don't bother if the fs is mounted and we're not fixing anything else.

---
 e2fsck/pass5.c                        |   22 ++++++++++++++++++++++
 e2fsck/problem.c                      |    5 +++++
 e2fsck/problem.h                      |    3 +++
 tests/j_long_trans/expect             |    3 +++
 tests/j_long_trans_mcsum_32bit/expect |    6 ++++++
 tests/j_long_trans_mcsum_64bit/expect |    6 ++++++
 6 files changed, 45 insertions(+)

e2fsck/pass5.c +22 -0

index 1a225fb..92ab633 100644
@@ -544,6 +544,28 @@ redo_counts:
 			} else
 				ext2fs_unmark_valid(fs);
 		}
+
+		/*
+		 * Opportunistically set block_uninit if none of the
+		 * blocks are in use.  However, it's not an error if
+		 * the flag could be set but isn't, so don't bother
+		 * the user if the FS is mounted and we haven't fixed
+		 * anything else, to minimize unnecessary reboots.
+		 */
+		if (((ctx->options & E2F_OPT_NO) ||
+		     !(ctx->mount_flags & EXT2_MF_MOUNTED) ||
+		     ext2fs_test_changed(fs)) &&
+		    ext2fs_has_group_desc_csum(fs) &&
+		    !ext2fs_bg_flags_test(fs, g, EXT2_BG_BLOCK_UNINIT) &&
+		    (g != fs->group_desc_count - 1) &&
+		    free_array[g] == fs->super->s_blocks_per_group) {
+			pctx.group = g;
+			if (fix_problem(ctx, PR_5_BLOCK_UNINIT_UNSET, &pctx)) {
+				ext2fs_bg_flags_set(fs, g,
+							EXT2_BG_BLOCK_UNINIT);
+				ext2fs_mark_super_dirty(fs);
+			}
+		}
 	}
 	free_blocks = EXT2FS_C2B(fs, free_blocks);
 	if (free_blocks != ext2fs_free_blocks_count(fs->super)) {

e2fsck/problem.c +5 -0

index 9c2b2e6..6a5f814 100644
@@ -1904,6 +1904,11 @@ static struct e2fsck_problem problem_table[] = {
 	  N_("Unused @i count wrong for @g #%g (%i, counted=%j).\n"),
 	  PROMPT_FIX, PR_PREEN_OK | PR_NO_OK },
 
+	/* block_uninit flag can be set for group */
+	{ PR_5_BLOCK_UNINIT_UNSET,
+	  N_("BLOCK_UNINIT flag can be set for @g #%g.\n"),
+	  PROMPT_FIX, PR_PREEN_OK | PR_NO_OK },
+
 	/* Post-Pass 5 errors */
 
 	/* Recreate journal if E2F_FLAG_JOURNAL_INODE flag is set */

e2fsck/problem.h +3 -0

index 6a71fc8..345dd0a 100644
@@ -1142,6 +1142,9 @@ struct problem_context {
 /* Unused inode count for group wrong */
 #define PR_5_UNUSED_INODES_COUNT_GROUP	0x05001C
 
+/* Unused inode count for group wrong */
+#define PR_5_BLOCK_UNINIT_UNSET		0x05001D
+
 /*
  * Post-Pass 5 errors
  */

tests/j_long_trans/expect +3 -0

index 7638ef1..2cc1dab 100644
@@ -81,6 +81,9 @@ Fix? yes
 Free blocks count wrong for group #14 (0, counted=8192).
 Fix? yes
 
+BLOCK_UNINIT flag can be set for group #14.
+Fix? yes
+
 Free blocks count wrong (247711, counted=255916).
 Fix? yes

tests/j_long_trans_mcsum_32bit/expect +6 -0

index 0d141c1..39b6e6d 100644
@@ -117,9 +117,15 @@ Fix? yes
 Free blocks count wrong for group #17 (0, counted=8192).
 Fix? yes
 
+BLOCK_UNINIT flag can be set for group #17.
+Fix? yes
+
 Free blocks count wrong for group #18 (0, counted=8192).
 Fix? yes
 
+BLOCK_UNINIT flag can be set for group #18.
+Fix? yes
+
 Free blocks count wrong (497236, counted=513633).
 Fix? yes

tests/j_long_trans_mcsum_64bit/expect +6 -0

index 94e9925..edb7776 100644
@@ -116,9 +116,15 @@ Fix? yes
 Free blocks count wrong for group #17 (0, counted=8192).
 Fix? yes
 
+BLOCK_UNINIT flag can be set for group #17.
+Fix? yes
+
 Free blocks count wrong for group #18 (0, counted=8192).
 Fix? yes
 
+BLOCK_UNINIT flag can be set for group #18.
+Fix? yes
+
 Free blocks count wrong (497218, counted=513615).
 Fix? yes

Darrick J. Wong

author

2014-10-27 23:28 UTC #31 lore raw

PATCH 19/39 libext2fs: set BLOCK_UNINIT for non-last blockgroups if all blocks are free

Set BLOCK_UNINIT in any group whose blocks are all unused, so long as it isn't the last group. This helps us speed up future e2fsck runs and mounts because we don't need to read or checksum block bitmaps for these groups.

v2: Take care of this in the library, since e2fsck always updates these summary counts.

---
 lib/ext2fs/csum.c |    5 +++++
 1 file changed, 5 insertions(+)

lib/ext2fs/csum.c +5 -0

index 12f6fe9..6dcefb9 100644
@@ -854,6 +854,11 @@ errcode_t ext2fs_set_gdt_csum(ext2_filsys fs)
 		__u32 old_unused = ext2fs_bg_itable_unused(fs, i);
 		__u32 old_flags = ext2fs_bg_flags(fs, i);
 		__u32 old_free_inodes_count = ext2fs_bg_free_inodes_count(fs, i);
+		__u32 old_free_blocks_count = ext2fs_bg_free_blocks_count(fs, i);
+
+		if (old_free_blocks_count == sb->s_blocks_per_group &&
+		    i != fs->group_desc_count - 1)
+			ext2fs_bg_flags_set(fs, i, EXT2_BG_BLOCK_UNINIT);
 
 		if (old_free_inodes_count == sb->s_inodes_per_group) {
 			ext2fs_bg_flags_set(fs, i, EXT2_BG_INODE_UNINIT);

Darrick J. Wong

author

2014-10-25 20:59 UTC #32 lore raw

PATCH 20/39 libext2fs/e2fsck: provide routines to read-ahead metadata

This patch adds to e2fsck the ability to pre-fetch metadata into the page cache in the hopes of speeding up fsck runs. There are two new functions -- the first allows a caller to readahead a list of blocks, and the second is a helper function that uses that first mechanism to load group data (bitmaps, inode tables).

These new e2fsck routines require the addition of a dblist API to allow us to iterate a subset of a dblist. This will enable incremental directory block readahead in e2fsck pass 2.

There's also a function to estimate the readahead given a FS.

v2: Add an API to create a dblist with a given number of list elements pre-allocated. This enables us to save ~2ms per call to e2fsck_readahead() (assuming a 2MB RA buffer) by not having to repeatedly call ext2_resize_mem as we add blocks to the list.

---
 configure           |    2 
 configure.in        |    1 
 e2fsck/Makefile.in  |    9 +-
 e2fsck/e2fsck.h     |   18 ++++
 e2fsck/readahead.c  |  252 +++++++++++++++++++++++++++++++++++++++++++++++++++
 e2fsck/util.c       |   51 ++++++++++
 lib/config.h.in     |    3 +
 lib/ext2fs/dblist.c |   21 ++++
 lib/ext2fs/ext2fs.h |   10 ++
 9 files changed, 359 insertions(+), 8 deletions(-)
 create mode 100644 e2fsck/readahead.c

configure +1 -1

index 2bc1275..a65d3e6 100755
@@ -12414,7 +12414,7 @@ fi
 done
 
 fi
-for ac_header in  	dirent.h 	errno.h 	execinfo.h 	getopt.h 	malloc.h 	mntent.h 	paths.h 	semaphore.h 	setjmp.h 	signal.h 	stdarg.h 	stdint.h 	stdlib.h 	termios.h 	termio.h 	unistd.h 	utime.h 	attr/xattr.h 	linux/falloc.h 	linux/fd.h 	linux/major.h 	linux/loop.h 	net/if_dl.h 	netinet/in.h 	sys/disklabel.h 	sys/disk.h 	sys/file.h 	sys/ioctl.h 	sys/mkdev.h 	sys/mman.h 	sys/mount.h 	sys/prctl.h 	sys/resource.h 	sys/select.h 	sys/socket.h 	sys/sockio.h 	sys/stat.h 	sys/syscall.h 	sys/sysmacros.h 	sys/time.h 	sys/types.h 	sys/un.h 	sys/wait.h
+for ac_header in  	dirent.h 	errno.h 	execinfo.h 	getopt.h 	malloc.h 	mntent.h 	paths.h 	semaphore.h 	setjmp.h 	signal.h 	stdarg.h 	stdint.h 	stdlib.h 	termios.h 	termio.h 	unistd.h 	utime.h 	attr/xattr.h 	linux/falloc.h 	linux/fd.h 	linux/major.h 	linux/loop.h 	net/if_dl.h 	netinet/in.h 	sys/disklabel.h 	sys/disk.h 	sys/file.h 	sys/ioctl.h 	sys/mkdev.h 	sys/mman.h 	sys/mount.h 	sys/prctl.h 	sys/resource.h 	sys/select.h 	sys/socket.h 	sys/sockio.h 	sys/stat.h 	sys/syscall.h 	sys/sysctl.h 	sys/sysmacros.h 	sys/time.h 	sys/types.h 	sys/un.h 	sys/wait.h
 do :
   as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh`
 ac_fn_c_check_header_mongrel "$LINENO" "$ac_header" "$as_ac_Header" "$ac_includes_default"

configure.in +1 -0

index e809324..d1be5b9 100644
@@ -949,6 +949,7 @@ AC_CHECK_HEADERS(m4_flatten([
 	sys/sockio.h
 	sys/stat.h
 	sys/syscall.h
+	sys/sysctl.h
 	sys/sysmacros.h
 	sys/time.h
 	sys/types.h

e2fsck/Makefile.in +7 -2

index d0e64eb..e40e51b 100644
@@ -62,7 +62,7 @@ OBJS= dict.o unix.o e2fsck.o super.o pass1.o pass1b.o pass2.o \
 	pass3.o pass4.o pass5.o journal.o badblocks.o util.o dirinfo.o \
 	dx_dirinfo.o ehandler.o problem.o message.o quota.o recovery.o \
 	region.o revoke.o ea_refcount.o rehash.o profile.o prof_err.o \
-	logfile.o sigcatcher.o $(MTRACE_OBJ) plausible.o
+	logfile.o sigcatcher.o $(MTRACE_OBJ) plausible.o readahead.o
 
 PROFILED_OBJS= profiled/dict.o profiled/unix.o profiled/e2fsck.o \
 	profiled/super.o profiled/pass1.o profiled/pass1b.o \

@@ -73,7 +73,8 @@ PROFILED_OBJS= profiled/dict.o profiled/unix.o profiled/e2fsck.o \
 	profiled/recovery.o profiled/region.o profiled/revoke.o \
 	profiled/ea_refcount.o profiled/rehash.o profiled/profile.o \
 	profiled/prof_err.o profiled/logfile.o \
-	profiled/sigcatcher.o profiled/plausible.o
+	profiled/sigcatcher.o profiled/plausible.o \
+	profiled/sigcatcher.o profiled/readahead.o
 
 SRCS= $(srcdir)/e2fsck.c \
 	$(srcdir)/dict.c \

@@ -97,6 +98,7 @@ SRCS= $(srcdir)/e2fsck.c \
 	$(srcdir)/message.c \
 	$(srcdir)/ea_refcount.c \
 	$(srcdir)/rehash.c \
+	$(srcdir)/readahead.c \
 	$(srcdir)/region.c \
 	$(srcdir)/profile.c \
 	$(srcdir)/sigcatcher.c \

@@ -541,3 +543,6 @@ plausible.o: $(srcdir)/../misc/plausible.c $(top_builddir)/lib/config.h \
  $(top_builddir)/lib/ext2fs/ext2_err.h \
  $(top_srcdir)/lib/ext2fs/ext2_ext_attr.h $(top_srcdir)/lib/ext2fs/bitops.h \
  $(srcdir)/../misc/nls-enable.h $(srcdir)/../misc/plausible.h
+readahead.o: $(srcdir)/readahead.c $(top_builddir)/lib/config.h \
+ $(top_srcdir)/lib/ext2fs/ext2fs.h $(top_srcdir)/lib/ext2fs/ext2_fs.h \
+ $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/e2fsck.h prof_err.h

e2fsck/e2fsck.h +18 -0

index 6ca3a6a..8837af9 100644
@@ -490,6 +490,23 @@ extern ext2_ino_t e2fsck_get_lost_and_found(e2fsck_t ctx, int fix);
 extern errcode_t e2fsck_adjust_inode_count(e2fsck_t ctx, ext2_ino_t ino,
 					   int adj);
 
+/* readahead.c */
+#define E2FSCK_READA_SUPER	(0x01)
+#define E2FSCK_READA_GDT	(0x02)
+#define E2FSCK_READA_BBITMAP	(0x04)
+#define E2FSCK_READA_IBITMAP	(0x08)
+#define E2FSCK_READA_ITABLE	(0x10)
+#define E2FSCK_READA_ALL_FLAGS	(0x1F)
+errcode_t e2fsck_readahead(ext2_filsys fs, int flags, dgrp_t start,
+			   dgrp_t ngroups);
+#define E2FSCK_RA_DBLIST_IGNORE_BLOCKCNT	(0x01)
+#define E2FSCK_RA_DBLIST_ALL_FLAGS		(0x01)
+errcode_t e2fsck_readahead_dblist(ext2_filsys fs, int flags,
+				  ext2_dblist dblist,
+				  unsigned long long start,
+				  unsigned long long count);
+int e2fsck_can_readahead(ext2_filsys fs);
+unsigned long long e2fsck_guess_readahead(ext2_filsys fs);
 
 /* region.c */
 extern region_t region_create(region_addr_t min, region_addr_t max);

@@ -579,6 +596,7 @@ extern errcode_t e2fsck_allocate_subcluster_bitmap(ext2_filsys fs,
 						   int default_type,
 						   const char *profile_name,
 						   ext2fs_block_bitmap *ret);
+unsigned long long get_memory_size(void);
 
 /* unix.c */
 extern void e2fsck_clear_progbar(e2fsck_t ctx);

e2fsck/readahead.c new file +252 -0

252 lines not on this page · raw message

e2fsck/util.c +51 -0

60 lines not on this page · raw message

lib/config.h.in +3 -0

index e318969..92fca3e 100644
@@ -509,6 +509,9 @@
 /* Define to 1 if you have the <sys/syscall.h> header file. */
 #undef HAVE_SYS_SYSCALL_H
 
+/* Define to 1 if you have the <sys/sysctl.h> header file. */
+#undef HAVE_SYS_SYSCTL_H
+
 /* Define to 1 if you have the <sys/sysmacros.h> header file. */
 #undef HAVE_SYS_SYSMACROS_H

lib/ext2fs/dblist.c +18 -3

index 942c4f0..bbdb221 100644
@@ -194,20 +194,25 @@ void ext2fs_dblist_sort2(ext2_dblist dblist,
 /*
  * This function iterates over the directory block list
  */
-errcode_t ext2fs_dblist_iterate2(ext2_dblist dblist,
+errcode_t ext2fs_dblist_iterate3(ext2_dblist dblist,
 				 int (*func)(ext2_filsys fs,
 					     struct ext2_db_entry2 *db_info,
 					     void	*priv_data),
+				 unsigned long long start,
+				 unsigned long long count,
 				 void *priv_data)
 {
-	unsigned long long	i;
+	unsigned long long	i, end;
 	int		ret;
 
 	EXT2_CHECK_MAGIC(dblist, EXT2_ET_MAGIC_DBLIST);
 
+	end = start + count;
 	if (!dblist->sorted)
 		ext2fs_dblist_sort2(dblist, 0);
-	for (i=0; i < dblist->count; i++) {
+	if (end > dblist->count)
+		end = dblist->count;
+	for (i = start; i < end; i++) {
 		ret = (*func)(dblist->fs, &dblist->list[i], priv_data);
 		if (ret & DBLIST_ABORT)
 			return 0;

@@ -215,6 +220,16 @@ errcode_t ext2fs_dblist_iterate2(ext2_dblist dblist,
 	return 0;
 }
 
+errcode_t ext2fs_dblist_iterate2(ext2_dblist dblist,
+				 int (*func)(ext2_filsys fs,
+					     struct ext2_db_entry2 *db_info,
+					     void	*priv_data),
+				 void *priv_data)
+{
+	return ext2fs_dblist_iterate3(dblist, func, 0, dblist->count,
+				      priv_data);
+}
+
 static EXT2_QSORT_TYPE dir_block_cmp2(const void *a, const void *b)
 {
 	const struct ext2_db_entry2 *db_a =

lib/ext2fs/ext2fs.h +8 -2

index fef6910..21fe45c 100644
@@ -1055,11 +1055,17 @@ extern void ext2fs_dblist_sort2(ext2_dblist dblist,
 extern errcode_t ext2fs_dblist_iterate(ext2_dblist dblist,
 	int (*func)(ext2_filsys fs, struct ext2_db_entry *db_info,
 		    void	*priv_data),
-       void *priv_data);
+	void *priv_data);
 extern errcode_t ext2fs_dblist_iterate2(ext2_dblist dblist,
 	int (*func)(ext2_filsys fs, struct ext2_db_entry2 *db_info,
 		    void	*priv_data),
-       void *priv_data);
+	void *priv_data);
+extern errcode_t ext2fs_dblist_iterate3(ext2_dblist dblist,
+	int (*func)(ext2_filsys fs, struct ext2_db_entry2 *db_info,
+		    void	*priv_data),
+	unsigned long long start,
+	unsigned long long count,
+	void *priv_data);
 extern errcode_t ext2fs_set_dir_block(ext2_dblist dblist, ext2_ino_t ino,
 				      blk_t blk, int blockcnt);
 extern errcode_t ext2fs_set_dir_block2(ext2_dblist dblist, ext2_ino_t ino,

Darrick J. Wong

author

2014-10-25 20:59 UTC #33 lore raw

PATCH 21/39 e2fsck: read-ahead metadata during passes 1, 2, and 4

e2fsck pass1 is modified to use the block group data prefetch function to try to fetch the inode tables into the pagecache before it is needed. We iterate through the blockgroups until we have enough inode tables that need reading such that we can issue readahead; then we sit and wait until the last inode table block read of the last group to start fetching the next bunch.

pass2 is modified to use the dirblock prefetching function to prefetch the list of directory blocks that are assembled in pass1. We use the "iterate a subset of a dblist" and avoid copying the dblist. Directory blocks are fetched incrementally as we walk through the directory block list. In previous iterations of this patch we would free the directory blocks after processing, but the performance hit to e2fsck itself wasn't worth it. Furthermore, it is anticipated that most users will then mount the FS and start using the directories, so they may as well remain in the page cache.

pass4 is modified to prefetch the block and inode bitmaps in anticipation of pass 5, because pass4 is entirely CPU bound.

In general, these mechanisms can decrease fsck time by 10-40%, if the host system has sufficient memory and the storage system can provide a lot of IOPs. Pretty much any storage system capable of handling multiple IOs in-flight at any time will see a fairly large performance boost. (Single-issue USB mass storage disks seem to suffer badly.)

By default, the readahead buffer size will be set to the size of a block group's inode table (which is 2MiB for a regular ext4 FS). The -E readahead_kb= option can be given to specify the amount of memory to use for readahead or zero to disable it entirely; or an option can be given in e2fsck.conf.

v2: Fix an off-by-one error in the pass1 readahead which made the readahead trigger one inode too late if the block groups are full.

---
 e2fsck/e2fsck.8.in      |    7 +++++
 e2fsck/e2fsck.conf.5.in |   15 +++++++++++
 e2fsck/e2fsck.h         |    3 ++
 e2fsck/pass1.c          |   65 +++++++++++++++++++++++++++++++++++++++++++++++
 e2fsck/pass2.c          |   38 +++++++++++++++++++++++++++
 e2fsck/pass4.c          |    9 +++++++
 e2fsck/unix.c           |   28 ++++++++++++++++++++
 lib/ext2fs/ext2fs.h     |    1 +
 lib/ext2fs/inode.c      |    3 +-
 9 files changed, 167 insertions(+), 2 deletions(-)

e2fsck/e2fsck.8.in +7 -0

index f5ed758..84ae50f 100644
@@ -207,6 +207,13 @@ option may prevent you from further manual data recovery.
 .BI nodiscard
 Do not attempt to discard free blocks and unused inode blocks. This option is
 exactly the opposite of discard option. This is set as default.
+.TP
+.BI readahead_kb
+Use this many KiB of memory to pre-fetch metadata in the hopes of reducing
+e2fsck runtime.  By default, this is set to the size of a block group's inode
+table (typically 2MiB on a regular ext4 filesystem); if this amount is more
+than 1/100 of total physical memory, readahead is disabled.  Set this to zero
+to disable readahead entirely.
 .RE
 .TP
 .B \-f

e2fsck/e2fsck.conf.5.in +15 -0

index 9ebfbbf..e1d0518 100644
@@ -205,6 +205,21 @@ of that type are squelched.  This can be useful if the console is slow
 (i.e., connected to a serial port) and so a large amount of output could
 end up delaying the boot process for a long time (potentially hours).
 .TP
+.I readahead_mem_pct
+Use this percentage of memory to try to read in metadata blocks ahead of the
+main e2fsck thread.  This should reduce run times, depending on the speed of
+the underlying storage and the amount of free memory.  There is no default, but
+see
+.B readahead_mem_pct
+for more details.
+.TP
+.I readahead_kb
+Use this amount of memory to read in metadata blocks ahead of the main checking
+thread.  Setting this value to zero disables readahead entirely.  By default,
+this is set the size of one block group's inode table (typically 2MiB on a
+regular ext4 filesystem); if this amount is more than 1/100th of total physical
+memory, readahead is disabled.
+.TP
 .I report_features
 If this boolean relation is true, e2fsck will print the file system
 features as part of its verbose reporting (i.e., if the

e2fsck/e2fsck.h +3 -0

index 8837af9..b2654ef 100644
@@ -378,6 +378,9 @@ struct e2fsck_struct {
 	 */
 	void *priv_data;
 	ext2fs_block_bitmap block_metadata_map; /* Metadata blocks */
+
+	/* How much are we allowed to readahead? */
+	unsigned long long readahead_kb;
 };
 
 /* Used by the region allocation code */

e2fsck/pass1.c +65 -0

87 lines not on this page · raw message

e2fsck/pass2.c +37 -1

index 0b9c5c5..2060ed2 100644
@@ -61,6 +61,9 @@
  * Keeps track of how many times an inode is referenced.
  */
 static void deallocate_inode(e2fsck_t ctx, ext2_ino_t ino, char* block_buf);
+static int check_dir_block2(ext2_filsys fs,
+			   struct ext2_db_entry2 *dir_blocks_info,
+			   void *priv_data);
 static int check_dir_block(ext2_filsys fs,
 			   struct ext2_db_entry2 *dir_blocks_info,
 			   void *priv_data);

@@ -77,6 +80,9 @@ struct check_dir_struct {
 	struct problem_context	pctx;
 	int	count, max;
 	e2fsck_t ctx;
+	unsigned long long list_offset;
+	unsigned long long ra_entries;
+	unsigned long long next_ra_off;
 };
 
 void e2fsck_pass2(e2fsck_t ctx)

@@ -96,6 +102,9 @@ void e2fsck_pass2(e2fsck_t ctx)
 	int			i, depth;
 	problem_t		code;
 	int			bad_dir;
+	int (*check_dir_func)(ext2_filsys fs,
+			      struct ext2_db_entry2 *dir_blocks_info,
+			      void *priv_data);
 
 	init_resource_track(&rtrack, ctx->fs->io);
 	clear_problem_context(&cd.pctx);

@@ -139,6 +148,9 @@ void e2fsck_pass2(e2fsck_t ctx)
 	cd.ctx = ctx;
 	cd.count = 1;
 	cd.max = ext2fs_dblist_count2(fs->dblist);
+	cd.list_offset = 0;
+	cd.ra_entries = ctx->readahead_kb * 1024 / ctx->fs->blocksize;
+	cd.next_ra_off = 0;
 
 	if (ctx->progress)
 		(void) (ctx->progress)(ctx, 2, 0, cd.max);

@@ -146,7 +158,8 @@ void e2fsck_pass2(e2fsck_t ctx)
 	if (fs->super->s_feature_compat & EXT2_FEATURE_COMPAT_DIR_INDEX)
 		ext2fs_dblist_sort2(fs->dblist, special_dir_block_cmp);
 
-	cd.pctx.errcode = ext2fs_dblist_iterate2(fs->dblist, check_dir_block,
+	check_dir_func = cd.ra_entries ? check_dir_block2 : check_dir_block;
+	cd.pctx.errcode = ext2fs_dblist_iterate2(fs->dblist, check_dir_func,
 						 &cd);
 	if (ctx->flags & E2F_FLAG_SIGNAL_MASK || ctx->flags & E2F_FLAG_RESTART)
 		return;

@@ -824,6 +837,29 @@ err:
 	return retval;
 }
 
+static int check_dir_block2(ext2_filsys fs,
+			   struct ext2_db_entry2 *db,
+			   void *priv_data)
+{
+	int err;
+	struct check_dir_struct *cd = priv_data;
+
+	if (cd->ra_entries && cd->list_offset >= cd->next_ra_off) {
+		err = e2fsck_readahead_dblist(fs,
+					E2FSCK_RA_DBLIST_IGNORE_BLOCKCNT,
+					fs->dblist,
+					cd->list_offset + cd->ra_entries / 8,
+					cd->ra_entries);
+		if (err)
+			cd->ra_entries = 0;
+		cd->next_ra_off = cd->list_offset + (cd->ra_entries * 7 / 8);
+	}
+
+	err = check_dir_block(fs, db, priv_data);
+	cd->list_offset++;
+	return err;
+}
+
 static int check_dir_block(ext2_filsys fs,
 			   struct ext2_db_entry2 *db,
 			   void *priv_data)

e2fsck/pass4.c +9 -0

index 21d93f0..bc9a2c4 100644
@@ -106,6 +106,15 @@ void e2fsck_pass4(e2fsck_t ctx)
 #ifdef MTRACE
 	mtrace_print("Pass 4");
 #endif
+	/*
+	 * Since pass4 is mostly CPU bound, start readahead of bitmaps
+	 * ahead of pass 5 if we haven't already loaded them.
+	 */
+	if (ctx->readahead_kb &&
+	    (fs->block_map == NULL || fs->inode_map == NULL))
+		e2fsck_readahead(fs, E2FSCK_READA_BBITMAP |
+				     E2FSCK_READA_IBITMAP,
+				 0, fs->group_desc_count);
 
 	clear_problem_context(&pctx);

e2fsck/unix.c +28 -0

index 615d690..f3672c0 100644
@@ -650,6 +650,7 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts)
 	char	*buf, *token, *next, *p, *arg;
 	int	ea_ver;
 	int	extended_usage = 0;
+	unsigned long long reada_kb;
 
 	buf = string_copy(ctx, opts, 0);
 	for (token = buf; token && *token; token = next) {

@@ -678,6 +679,15 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts)
 				continue;
 			}
 			ctx->ext_attr_ver = ea_ver;
+		} else if (strcmp(token, "readahead_kb") == 0) {
+			reada_kb = strtoull(arg, &p, 0);
+			if (*p) {
+				fprintf(stderr, "%s",
+					_("Invalid readahead buffer size.\n"));
+				extended_usage++;
+				continue;
+			}
+			ctx->readahead_kb = reada_kb;
 		} else if (strcmp(token, "fragcheck") == 0) {
 			ctx->options |= E2F_OPT_FRAGCHECK;
 			continue;

@@ -717,6 +727,7 @@ static void parse_extended_opts(e2fsck_t ctx, const char *opts)
 		fputs(("\tjournal_only\n"), stderr);
 		fputs(("\tdiscard\n"), stderr);
 		fputs(("\tnodiscard\n"), stderr);
+		fputs(("\treadahead_kb=<buffer size>\n"), stderr);
 		fputc('\n', stderr);
 		exit(1);
 	}

@@ -750,6 +761,7 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 #ifdef CONFIG_JBD_DEBUG
 	char 		*jbd_debug;
 #endif
+	unsigned long long phys_mem_kb;
 
 	retval = e2fsck_allocate_context(&ctx);
 	if (retval)

@@ -777,6 +789,8 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 	else
 		ctx->program_name = "e2fsck";
 
+	phys_mem_kb = get_memory_size() / 1024;
+	ctx->readahead_kb = ~0ULL;
 	while ((c = getopt (argc, argv, "panyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDk")) != EOF)
 		switch (c) {
 		case 'C':

@@ -961,6 +975,20 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 	if (c)
 		verbose = 1;
 
+	if (ctx->readahead_kb == ~0ULL) {
+		profile_get_integer(ctx->profile, "options",
+				    "readahead_mem_pct", 0, -1, &c);
+		if (c >= 0 && c <= 100)
+			ctx->readahead_kb = phys_mem_kb * c / 100;
+		profile_get_integer(ctx->profile, "options",
+				    "readahead_kb", 0, -1, &c);
+		if (c >= 0)
+			ctx->readahead_kb = c;
+		if (ctx->readahead_kb != ~0ULL &&
+		    ctx->readahead_kb > phys_mem_kb)
+			ctx->readahead_kb = phys_mem_kb;
+	}
+
 	/* Turn off discard in read-only mode */
 	if ((ctx->options & E2F_OPT_NO) &&
 	    (ctx->options & E2F_OPT_DISCARD))

lib/ext2fs/ext2fs.h +1 -0

index 21fe45c..689f6a6 100644
@@ -1421,6 +1421,7 @@ extern errcode_t ext2fs_get_next_inode_full(ext2_inode_scan scan,
 					    ext2_ino_t *ino,
 					    struct ext2_inode *inode,
 					    int bufsize);
+#define EXT2_INODE_SCAN_DEFAULT_BUFFER_BLOCKS	8
 extern errcode_t ext2fs_open_inode_scan(ext2_filsys fs, int buffer_blocks,
 				  ext2_inode_scan *ret_scan);
 extern void ext2fs_close_inode_scan(ext2_inode_scan scan);

lib/ext2fs/inode.c +2 -1

index 4310b82..4b3e14e 100644
@@ -175,7 +175,8 @@ errcode_t ext2fs_open_inode_scan(ext2_filsys fs, int buffer_blocks,
 	scan->bytes_left = 0;
 	scan->current_group = 0;
 	scan->groups_left = fs->group_desc_count - 1;
-	scan->inode_buffer_blocks = buffer_blocks ? buffer_blocks : 8;
+	scan->inode_buffer_blocks = buffer_blocks ? buffer_blocks :
+				    EXT2_INODE_SCAN_DEFAULT_BUFFER_BLOCKS;
 	scan->current_block = ext2fs_inode_table_loc(scan->fs,
 						     scan->current_group);
 	scan->inodes_left = EXT2_INODES_PER_GROUP(scan->fs->super);

Darrick J. Wong

author

2014-10-25 20:59 UTC #34 lore raw

PATCH 22/39 libext2fs: ext2fs_new_block2() should call alloc_block hook

If ext2fs_new_block2() is called without a specific block map, we should call the alloc_block hook before checking fs->block_map. This helps us to avoid a bug in e2fsck where we need to allocate a block but instead of consulting block_found_map, we use the FS bitmaps, which (prior to pass 5) could be wrong.

---
 e2fsck/pass1.c     |    2 +-
 lib/ext2fs/alloc.c |   15 +++++++++++++++
 2 files changed, 16 insertions(+), 1 deletion(-)

e2fsck/pass1.c +1 -1

index 36eea16..d4760ef 100644
@@ -3790,7 +3790,7 @@ static errcode_t e2fsck_get_alloc_block(ext2_filsys fs, blk64_t goal,
 				return retval;
 		}
 
-		retval = ext2fs_new_block2(fs, goal, 0, &new_block);
+		retval = ext2fs_new_block2(fs, goal, fs->block_map, &new_block);
 		if (retval)
 			return retval;
 	}

lib/ext2fs/alloc.c +15 -0

index 578fd7f..d1c1a84 100644
@@ -137,9 +137,23 @@ errcode_t ext2fs_new_block2(ext2_filsys fs, blk64_t goal,
 {
 	errcode_t retval;
 	blk64_t	b = 0;
+	errcode_t (*gab)(ext2_filsys fs, blk64_t goal, blk64_t *ret);
 
 	EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS);
 
+	if (!map && fs->get_alloc_block) {
+		/*
+		 * In case there are clients out there whose get_alloc_block
+		 * handlers call ext2fs_new_block2 with a NULL block map,
+		 * temporarily swap out the function pointer so that we don't
+		 * end up in an infinite loop.
+		 */
+		gab = fs->get_alloc_block;
+		fs->get_alloc_block = NULL;
+		retval = gab(fs, goal, &b);
+		fs->get_alloc_block = gab;
+		goto allocated;
+	}
 	if (!map)
 		map = fs->block_map;
 	if (!map)

@@ -153,6 +167,7 @@ errcode_t ext2fs_new_block2(ext2_filsys fs, blk64_t goal,
 	if ((retval == ENOENT) && (goal != fs->super->s_first_data_block))
 		retval = ext2fs_find_first_zero_block_bitmap2(map,
 			fs->super->s_first_data_block, goal - 1, &b);
+allocated:
 	if (retval == ENOENT)
 		return EXT2_ET_BLOCK_ALLOC_FAIL;
 	if (retval)

Darrick J. Wong

author

2014-10-25 20:59 UTC #35 lore raw

PATCH 23/39 libext2fs: support BLKZEROOUT/FALLOC_FL_ZERO_RANGE in ext2fs_zero_blocks

Plumb a new call into the IO manager to support translating ext2fs_zero_blocks calls into the equivalent kernel-level BLKZEROOUT ioctl or FALLOC_FL_ZERO_RANGE fallocate flag primitives when possible.

---
 lib/ext2fs/ext2_io.h    |    7 ++-
 lib/ext2fs/io_manager.c |   11 ++++
 lib/ext2fs/mkjournal.c  |    6 ++
 lib/ext2fs/test_io.c    |   21 ++++++++
 lib/ext2fs/unix_io.c    |  125 +++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 169 insertions(+), 1 deletion(-)

lib/ext2fs/ext2_io.h +6 -1

index 4c5a5c5..1faa720 100644
@@ -93,7 +93,9 @@ struct struct_io_manager {
 	errcode_t (*cache_readahead)(io_channel channel,
 				     unsigned long long block,
 				     unsigned long long count);
-	long	reserved[15];
+	errcode_t (*zeroout)(io_channel channel, unsigned long long block,
+			     unsigned long long count);
+	long	reserved[14];
 };
 
 #define IO_FLAG_RW		0x0001

@@ -125,6 +127,9 @@ extern errcode_t io_channel_write_blk64(io_channel channel,
 extern errcode_t io_channel_discard(io_channel channel,
 				    unsigned long long block,
 				    unsigned long long count);
+extern errcode_t io_channel_zeroout(io_channel channel,
+				    unsigned long long block,
+				    unsigned long long count);
 extern errcode_t io_channel_alloc_buf(io_channel channel,
 				      int count, void *ptr);
 extern errcode_t io_channel_cache_readahead(io_channel io,

lib/ext2fs/io_manager.c +11 -0

index dc5888d..c395d61 100644
@@ -112,6 +112,17 @@ errcode_t io_channel_discard(io_channel channel, unsigned long long block,
 	return EXT2_ET_UNIMPLEMENTED;
 }
 
+errcode_t io_channel_zeroout(io_channel channel, unsigned long long block,
+			     unsigned long long count)
+{
+	EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL);
+
+	if (channel->manager->zeroout)
+		return (channel->manager->zeroout)(channel, block, count);
+
+	return EXT2_ET_UNIMPLEMENTED;
+}
+
 errcode_t io_channel_alloc_buf(io_channel io, int count, void *ptr)
 {
 	size_t	size;

lib/ext2fs/mkjournal.c +6 -0

index 6f3a862..5be425c 100644
@@ -164,6 +164,12 @@ errcode_t ext2fs_zero_blocks2(ext2_filsys fs, blk64_t blk, int num,
 		}
 		return 0;
 	}
+
+	/* Try a zero out command, if supported */
+	retval = io_channel_zeroout(fs->io, blk, num);
+	if (retval == 0)
+		return 0;
+
 	/* Allocate the zeroizing buffer if necessary */
 	if (!buf) {
 		buf = malloc(fs->blocksize * STRIDE_LENGTH);

lib/ext2fs/test_io.c +21 -0

index b03a939..f7c50d1 100644
@@ -86,6 +86,7 @@ void (*test_io_cb_write_byte)
 #define TEST_FLAG_SET_OPTION		0x20
 #define TEST_FLAG_DISCARD		0x40
 #define TEST_FLAG_READAHEAD		0x80
+#define TEST_FLAG_ZEROOUT		0x100
 
 static void test_dump_block(io_channel channel,
 			    struct test_private_data *data,

@@ -507,6 +508,25 @@ static errcode_t test_cache_readahead(io_channel channel,
 	return retval;
 }
 
+static errcode_t test_zeroout(io_channel channel, unsigned long long block,
+			      unsigned long long count)
+{
+	struct test_private_data *data;
+	errcode_t	retval = 0;
+
+	EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL);
+	data = (struct test_private_data *) channel->private_data;
+	EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_TEST_IO_CHANNEL);
+
+	if (data->real)
+		retval = io_channel_zeroout(data->real, block, count);
+	if (data->flags & TEST_FLAG_ZEROOUT)
+		fprintf(data->outfile,
+			"Test_io: zeroout(%llu, %llu) returned %s\n",
+			block, count, retval ? error_message(retval) : "OK");
+	return retval;
+}
+
 static struct struct_io_manager struct_test_manager = {
 	.magic		= EXT2_ET_MAGIC_IO_MANAGER,
 	.name		= "Test I/O Manager",

@@ -523,6 +543,7 @@ static struct struct_io_manager struct_test_manager = {
 	.write_blk64	= test_write_blk64,
 	.discard	= test_discard,
 	.cache_readahead	= test_cache_readahead,
+	.zeroout	= test_zeroout,
 };
 
 io_manager test_io_manager = &struct_test_manager;

lib/ext2fs/unix_io.c +125 -0

137 lines not on this page · raw message

Darrick J. Wong

author

2014-10-25 20:59 UTC #36 lore raw

PATCH 24/39 libext2fs/e2fsck: refactor everyone who writes zero blocks to disk

Convert all call sites that write zero blocks to disk to use ext2fs_zero_blocks2() since it can use Linux's zero out feature to do the writes more quickly. Reclaim the zero buffer at freefs time and make the write-zeroes fallback use a larger buffer.

---
 e2fsck/e2fsck.h        |    2 -
 e2fsck/pass1.c         |   13 +++++---
 e2fsck/pass3.c         |   13 ++------
 e2fsck/util.c          |   53 ----------------------------------
 lib/ext2fs/alloc.c     |   14 ++-------
 lib/ext2fs/expanddir.c |   13 ++------
 lib/ext2fs/ext2fs.h    |    3 ++
 lib/ext2fs/freefs.c    |    1 +
 lib/ext2fs/mkjournal.c |   75 +++++++++++++++++++++++++++++++++++-------------
 misc/mke2fs.c          |    2 -
 resize/resize2fs.c     |   25 ++++++----------
 11 files changed, 84 insertions(+), 130 deletions(-)

e2fsck/e2fsck.h +0 -2

index b2654ef..e359515 100644
@@ -544,8 +544,6 @@ extern void e2fsck_read_bitmaps(e2fsck_t ctx);
 extern void e2fsck_write_bitmaps(e2fsck_t ctx);
 extern void preenhalt(e2fsck_t ctx);
 extern char *string_copy(e2fsck_t ctx, const char *str, int len);
-extern errcode_t e2fsck_zero_blocks(ext2_filsys fs, blk_t blk, int num,
-				    blk_t *ret_blk, int *ret_count);
 extern int fs_proc_check(const char *fs_name);
 extern int check_for_modules(const char *fs_name);
 #ifdef RESOURCE_TRACK

e2fsck/pass1.c +8 -5

index d4760ef..a963849 100644
@@ -3576,12 +3576,15 @@ static void new_table_block(e2fsck_t ctx, blk64_t first_block, dgrp_t group,
 				   old_block + i, 1, buf);
 			if (pctx.errcode)
 				fix_problem(ctx, PR_1_RELOC_READ_ERR, &pctx);
-		} else
-			memset(buf, 0, fs->blocksize);
+			pctx.blk = (*new_block) + i;
+			pctx.errcode = io_channel_write_blk64(fs->io, pctx.blk,
+							      1, buf);
+		} else {
+			pctx.blk = (*new_block) + i;
+			pctx.errcode = ext2fs_zero_blocks2(fs, pctx.blk, 1,
+							   NULL, NULL);
+		}
 
-		pctx.blk = (*new_block) + i;
-		pctx.errcode = io_channel_write_blk64(fs->io, pctx.blk,
-					      1, buf);
 		if (pctx.errcode)
 			fix_problem(ctx, PR_1_RELOC_WRITE_ERR, &pctx);
 	}

e2fsck/pass3.c +3 ~~-10~~

index f03c7ae..2d94ece 100644
@@ -809,20 +809,13 @@ static int expand_dir_proc(ext2_filsys fs,
 		es->num--;
 		retval = ext2fs_write_dir_block4(fs, new_blk, block, 0,
 						 es->dir);
-	} else {
-		retval = ext2fs_get_mem(fs->blocksize, &block);
-		if (retval) {
-			es->err = retval;
-			return BLOCK_ABORT;
-		}
-		memset(block, 0, fs->blocksize);
-		retval = io_channel_write_blk64(fs->io, new_blk, 1, block);
-	}
+		ext2fs_free_mem(&block);
+	} else
+		retval = ext2fs_zero_blocks2(fs, new_blk, 1, NULL, NULL);
 	if (retval) {
 		es->err = retval;
 		return BLOCK_ABORT;
 	}
-	ext2fs_free_mem(&block);
 	*blocknr = new_blk;
 	ext2fs_mark_block_bitmap2(ctx->block_found_map, new_blk);

e2fsck/util.c +0 ~~-53~~

59 lines not on this page · raw message

lib/ext2fs/alloc.c +3 ~~-11~~

index d1c1a84..82af18e 100644
@@ -198,15 +198,9 @@ errcode_t ext2fs_alloc_block2(ext2_filsys fs, blk64_t goal,
 {
 	errcode_t	retval;
 	blk64_t		block;
-	char		*buf = 0;
 
-	if (!block_buf) {
-		retval = ext2fs_get_mem(fs->blocksize, &buf);
-		if (retval)
-			return retval;
-		block_buf = buf;
-	}
-	memset(block_buf, 0, fs->blocksize);
+	if (block_buf)
+		memset(block_buf, 0, fs->blocksize);
 
 	if (fs->get_alloc_block) {
 		retval = (fs->get_alloc_block)(fs, goal, &block);

@@ -224,7 +218,7 @@ errcode_t ext2fs_alloc_block2(ext2_filsys fs, blk64_t goal,
 			goto fail;
 	}
 
-	retval = io_channel_write_blk64(fs->io, block, 1, block_buf);
+	retval = ext2fs_zero_blocks3(fs, block, 1, NULL, NULL, block_buf);
 	if (retval)
 		goto fail;

@@ -232,8 +226,6 @@ errcode_t ext2fs_alloc_block2(ext2_filsys fs, blk64_t goal,
 	*ret = block;
 
 fail:
-	if (buf)
-		ext2fs_free_mem(&buf);
 	return retval;
 }

lib/ext2fs/expanddir.c +3 ~~-10~~

index d0f7287..ecc13ae 100644
@@ -67,22 +67,15 @@ static int expand_dir_proc(ext2_filsys	fs,
 		es->done = 1;
 		retval = ext2fs_write_dir_block4(fs, new_blk, block, 0,
 						 es->dir);
-	} else {
-		retval = ext2fs_get_mem(fs->blocksize, &block);
-		if (retval) {
-			es->err = retval;
-			return BLOCK_ABORT;
-		}
-		memset(block, 0, fs->blocksize);
-		retval = io_channel_write_blk64(fs->io, new_blk, 1, block);
-	}
+		ext2fs_free_mem(&block);
+	} else
+		retval = ext2fs_zero_blocks2(fs, new_blk, 1, NULL, NULL);
 	if (blockcnt >= 0)
 		es->goal = new_blk;
 	if (retval) {
 		es->err = retval;
 		return BLOCK_ABORT;
 	}
-	ext2fs_free_mem(&block);
 	*blocknr = new_blk;
 
 	if (es->done)

lib/ext2fs/ext2fs.h +3 -0

index 689f6a6..5ecaa85 100644
@@ -1504,6 +1504,9 @@ extern errcode_t ext2fs_zero_blocks(ext2_filsys fs, blk_t blk, int num,
 				    blk_t *ret_blk, int *ret_count);
 extern errcode_t ext2fs_zero_blocks2(ext2_filsys fs, blk64_t blk, int num,
 				     blk64_t *ret_blk, int *ret_count);
+extern errcode_t ext2fs_zero_blocks3(ext2_filsys fs, blk64_t blk, int num,
+				     blk64_t *ret_blk, int *ret_count,
+				     const void *block_buf);
 extern errcode_t ext2fs_create_journal_superblock(ext2_filsys fs,
 						  __u32 num_blocks, int flags,
 						  char  **ret_jsb);

lib/ext2fs/freefs.c +1 -0

index 89a157b..ea9742e 100644
@@ -61,6 +61,7 @@ void ext2fs_free(ext2_filsys fs)
 
 	fs->magic = 0;
 
+	ext2fs_zero_blocks2(NULL, 0, 0, NULL, NULL);
 	ext2fs_free_mem(&fs);
 }

lib/ext2fs/mkjournal.c +54 ~~-21~~

139 lines not on this page · raw message

misc/mke2fs.c +0 -2

index c6a2c2f..149330c 100644
@@ -437,7 +437,6 @@ static void write_inode_tables(ext2_filsys fs, int lazy_flag, int itable_zeroed)
 				sync();
 		}
 	}
-	ext2fs_zero_blocks2(0, 0, 0, 0, 0);
 	ext2fs_numeric_progress_close(fs, &progress,
 				      _("done                            \n"));

@@ -626,7 +625,6 @@ static void create_journal_dev(ext2_filsys fs)
 		count -= c;
 		ext2fs_numeric_progress_update(fs, &progress, blk);
 	}
-	ext2fs_zero_blocks2(0, 0, 0, 0, 0);
 
 	ext2fs_numeric_progress_close(fs, &progress, NULL);
 write_superblock:

resize/resize2fs.c +5 ~~-10~~

index e71acf3..93df070 100644
@@ -762,11 +762,11 @@ static errcode_t adjust_superblock(ext2_resize_t rfs, blk64_t new_size)
 		/*
 		 * Write out the new inode table
 		 */
-		retval = io_channel_write_blk64(fs->io,
-						ext2fs_inode_table_loc(fs, i),
-						fs->inode_blocks_per_group,
-						rfs->itable_buf);
-		if (retval) goto errout;
+		retval = ext2fs_zero_blocks2(fs, ext2fs_inode_table_loc(fs, i),
+					     fs->inode_blocks_per_group, NULL,
+					     NULL);
+		if (retval)
+			goto errout;
 
 		io_channel_flush(fs->io);
 		if (rfs->progress) {

@@ -2244,15 +2244,11 @@ static errcode_t fix_resize_inode(ext2_filsys fs)
 {
 	struct ext2_inode	inode;
 	errcode_t		retval;
-	char			*block_buf = NULL;
 
 	if (!(fs->super->s_feature_compat &
 	      EXT2_FEATURE_COMPAT_RESIZE_INODE))
 		return 0;
 
-	retval = ext2fs_get_mem(fs->blocksize, &block_buf);
-	if (retval) goto errout;
-
 	retval = ext2fs_read_inode(fs, EXT2_RESIZE_INO, &inode);
 	if (retval) goto errout;

@@ -2272,19 +2268,16 @@ static errcode_t fix_resize_inode(ext2_filsys fs)
 		exit(1);
 	}
 
-	memset(block_buf, 0, fs->blocksize);

Darrick J. Wong

author

2014-10-25 20:59 UTC #37 lore raw

PATCH 25/39 libext2fs: support allocating uninit blocks in bmap2()

As part of supporting fallocate-like functionality, extend ext2fs_bmap() with two flags -- BMAP_UNINIT and BMAP_ZERO. The first will cause it to mark/set a block uninitialized, if it's part of an extent based file. For a block mapped file, the mapping is put in, but there is no way to remember the uninitialized status. The second flag causes the block to be zeroed to support the use case of emulating uninitialized blocks on a block-map file by zeroing them.

Eventually fallocate or fuse2fs or somebody will use these.

---
 lib/ext2fs/bmap.c   |    9 +++++++--
 lib/ext2fs/ext2fs.h |    2 ++
 2 files changed, 9 insertions(+), 2 deletions(-)

lib/ext2fs/bmap.c +7 -2

index c1d0e6f..e936ec1 100644
@@ -214,10 +214,13 @@ static errcode_t extent_bmap(ext2_filsys fs, ext2_ino_t ino,
 	errcode_t		retval = 0;
 	blk64_t			blk64 = 0;
 	int			alloc = 0;
+	int			set_flags;
+
+	set_flags = bmap_flags & BMAP_UNINIT ? EXT2_EXTENT_SET_BMAP_UNINIT : 0;
 
 	if (bmap_flags & BMAP_SET) {
 		retval = ext2fs_extent_set_bmap(handle, block,
-						*phys_blk, 0);
+						*phys_blk, set_flags);
 		return retval;
 	}
 	retval = ext2fs_extent_goto(handle, block);

@@ -254,7 +257,7 @@ got_block:
 		alloc++;
 	set_extent:
 		retval = ext2fs_extent_set_bmap(handle, block,
-						blk64, 0);
+						blk64, set_flags);
 		if (retval) {
 			ext2fs_block_alloc_stats2(fs, blk64, -1);
 			return retval;

@@ -440,6 +443,8 @@ errcode_t ext2fs_bmap2(ext2_filsys fs, ext2_ino_t ino, struct ext2_inode *inode,
 	if (retval == 0)
 		*phys_blk = blk32;
 done:
+	if (*phys_blk && retval == 0 && (bmap_flags & BMAP_ZERO))
+		retval = ext2fs_zero_blocks2(fs, *phys_blk, 1, NULL, NULL);
 	if (buf)
 		ext2fs_free_mem(&buf);
 	if (handle)

lib/ext2fs/ext2fs.h +2 -0

index 5ecaa85..664c3c0 100644
@@ -533,6 +533,8 @@ typedef struct ext2_icount *ext2_icount_t;
  */
 #define BMAP_ALLOC	0x0001
 #define BMAP_SET	0x0002
+#define BMAP_UNINIT	0x0004
+#define BMAP_ZERO	0x0008
 
 /*
  * Returned flags from ext2fs_bmap

Darrick J. Wong

author

2014-10-25 20:59 UTC #38 lore raw

PATCH 26/39 libext2fs: file IO routines should handle uninit blocks

The file IO routines do not handle uninit blocks at all. The read method should check for the uninit flag and return a buffer of zeroes, and the write routine should convert unwritten extents.

---
 lib/ext2fs/fileio.c         |   24 ++++++++++++++++++++++--
 tests/f_uninit_cat/expect   |  Bin
 tests/f_uninit_cat/image.gz |  Bin
 tests/f_uninit_cat/name     |    1 +
 tests/f_uninit_cat/script   |   37 +++++++++++++++++++++++++++++++++++++
 5 files changed, 60 insertions(+), 2 deletions(-)
 create mode 100644 tests/f_uninit_cat/expect
 create mode 100644 tests/f_uninit_cat/image.gz
 create mode 100644 tests/f_uninit_cat/name
 create mode 100755 tests/f_uninit_cat/script

lib/ext2fs/fileio.c +22 -2

index 1d5032a..d0a05d6 100644
@@ -123,6 +123,8 @@ errcode_t ext2fs_file_flush(ext2_file_t file)
 {
 	errcode_t	retval;
 	ext2_filsys fs;
+	int		ret_flags;
+	blk64_t		dontcare;
 
 	EXT2_CHECK_MAGIC(file, EXT2_ET_MAGIC_EXT2_FILE);
 	fs = file->fs;

@@ -131,6 +133,22 @@ errcode_t ext2fs_file_flush(ext2_file_t file)
 	    !(file->flags & EXT2_FILE_BUF_DIRTY))
 		return 0;
 
+	/* Is this an uninit block? */
+	if (file->physblock && file->inode.i_flags & EXT4_EXTENTS_FL) {
+		retval = ext2fs_bmap2(fs, file->ino, &file->inode, BMAP_BUFFER,
+				      0, file->blockno, &ret_flags, &dontcare);
+		if (retval)
+			return retval;
+		if (ret_flags & BMAP_RET_UNINIT) {
+			retval = ext2fs_bmap2(fs, file->ino, &file->inode,
+					      BMAP_BUFFER, BMAP_SET,
+					      file->blockno, 0,
+					      &file->physblock);
+			if (retval)
+				return retval;
+		}
+	}
+
 	/*
 	 * OK, the physical block hasn't been allocated yet.
 	 * Allocate it.

@@ -185,15 +203,17 @@ static errcode_t load_buffer(ext2_file_t file, int dontfill)
 {
 	ext2_filsys	fs = file->fs;
 	errcode_t	retval;
+	int		ret_flags;
 
 	if (!(file->flags & EXT2_FILE_BUF_VALID)) {
 		retval = ext2fs_bmap2(fs, file->ino, &file->inode,
-				     BMAP_BUFFER, 0, file->blockno, 0,
+				     BMAP_BUFFER, 0, file->blockno, &ret_flags,
 				     &file->physblock);
 		if (retval)
 			return retval;
 		if (!dontfill) {
-			if (file->physblock) {
+			if (file->physblock &&
+			    !(ret_flags & BMAP_RET_UNINIT)) {
 				retval = io_channel_read_blk64(fs->io,
 							       file->physblock,
 							       1, file->buf);

tests/f_uninit_cat/expect new file binary +0 -0

new file mode 100644
index 0000000000000000000000000000000000000000..0c0a5cf84cc58483ca33e257259607e4ca54ebbe
GIT binary patch

tests/f_uninit_cat/image.gz new file binary +0 -0

new file mode 100644
index 0000000000000000000000000000000000000000..d2ae66cb35f93dd4022e85f8667365487a7b24ea
GIT binary patch

tests/f_uninit_cat/name new file +1 -0

new file mode 100644
index 0000000..f6b5674
@@ -0,0 +1 @@
+cat a file with uninit blocks

tests/f_uninit_cat/script new file +37 -0

new file mode 100755
index 0000000..11dbb6e
@@ -0,0 +1,37 @@
+#!/bin/bash
+
+if test -x $DEBUGFS_EXE; then
+FSCK_OPT=-fy
+IMAGE=$test_dir/image.gz
+
+gzip -d < $IMAGE > $TMPFILE
+#e2label $TMPFILE test_filesys
+
+# Run fsck to fix things?
+EXP=$test_dir/expect
+OUT=$test_name.log
+rm -rf $test_name.failed $test_name.ok
+
+$FSCK $FSCK_OPT -N test_filesys $TMPFILE 2>&1 | sed -f $cmd_dir/filter.sed > $OUT
+echo "Exit status is $?" >> $OUT
+
+echo "debugfs cat uninit file" >> $OUT
+echo "ex /a" > $TMPFILE.cmd
+echo "cat /a" >> $TMPFILE.cmd
+$DEBUGFS_EXE -w $TMPFILE -f $TMPFILE.cmd >> $OUT.new 2>&1
+echo >> $OUT.new
+sed -f $cmd_dir/filter.sed < $OUT.new >> $OUT
+rm -rf $OUT.new $TMPFILE
+
+# Figure out what happened
+if cmp -s $EXP $OUT; then
+	echo "$test_name: $test_description: ok"
+	touch $test_name.ok
+else
+	echo "$test_name: $test_description: failed"
+	diff -u $EXP $OUT >> $test_name.failed
+fi
+unset EXP OUT FSCK_OPT IMAGE
+else #if test -a -x $DEBUGFS_EXE; then
+        echo "$test_name: $test_description: skipped"
+fi

Darrick J. Wong

author

2014-10-25 20:59 UTC #39 lore raw

PATCH 27/39 resize2fs: convert fs to and from 64bit mode

resize2fs does its magic by loading a filesystem, duplicating the in-memory image of that fs, moving relevant blocks out of the way of whatever new metadata get created, and finally writing everything back out to disk. Enabling 64bit mode enlarges the group descriptors, which makes resize2fs a reasonable vehicle for taking care of the rest of the bookkeeping requirements, so add to resize2fs the ability to convert a filesystem to 64bit mode and back.

Cc TR Reardon <[email protected]>

---
 resize/main.c         |   39 +++++
 resize/resize2fs.8.in |   18 ++
 resize/resize2fs.c    |  363 +++++++++++++++++++++++++++++++++++++++++++++++--
 resize/resize2fs.h    |    3 
 4 files changed, 402 insertions(+), 21 deletions(-)

resize/main.c +36 -3

index e017792..425ee63 100644
@@ -42,7 +42,7 @@ static char *device_name, *io_options;
 static void usage (char *prog)
 {
 	fprintf (stderr, _("Usage: %s [-d debug_flags] [-f] [-F] [-M] [-P] "
-			   "[-p] device [new_size]\n\n"), prog);
+			   "[-p] device [-b|-s|new_size]\n\n"), prog);
 
 	exit (1);
 }

@@ -203,7 +203,7 @@ int main (int argc, char ** argv)
 	if (argc && *argv)
 		program_name = *argv;
 
-	while ((c = getopt (argc, argv, "d:fFhMPpS:")) != EOF) {
+	while ((c = getopt(argc, argv, "d:fFhMPpS:bs")) != EOF) {
 		switch (c) {
 		case 'h':
 			usage(program_name);

@@ -229,6 +229,12 @@ int main (int argc, char ** argv)
 		case 'S':
 			use_stride = atoi(optarg);
 			break;
+		case 'b':
+			flags |= RESIZE_ENABLE_64BIT;
+			break;
+		case 's':
+			flags |= RESIZE_DISABLE_64BIT;
+			break;
 		default:
 			usage(program_name);
 		}

@@ -393,6 +399,10 @@ int main (int argc, char ** argv)
 		if (sys_page_size > blocksize)
 			new_size &= ~((sys_page_size / blocksize)-1);
 	}
+	/* If changing 64bit, don't change the filesystem size. */
+	if (flags & (RESIZE_DISABLE_64BIT | RESIZE_ENABLE_64BIT)) {
+		new_size = ext2fs_blocks_count(fs->super);
+	}
 	if (!EXT2_HAS_INCOMPAT_FEATURE(fs->super,
 				       EXT4_FEATURE_INCOMPAT_64BIT)) {
 		/* Take 16T down to 2^32-1 blocks */

@@ -444,7 +454,30 @@ int main (int argc, char ** argv)
 			blocksize / 1024, new_size);
 		exit(1);
 	}
-	if (new_size == ext2fs_blocks_count(fs->super)) {
+	if ((flags & RESIZE_DISABLE_64BIT) && (flags & RESIZE_ENABLE_64BIT)) {
+		fprintf(stderr, _("Cannot set and unset 64bit feature.\n"));
+		exit(1);
+	} else if (flags & (RESIZE_DISABLE_64BIT | RESIZE_ENABLE_64BIT)) {
+		if (new_size >= (1ULL << 32)) {
+			fprintf(stderr, _("Cannot change the 64bit feature "
+				"on a filesystem that is larger than "
+				"2^32 blocks.\n"));
+			exit(1);
+		}
+		if (mount_flags & EXT2_MF_MOUNTED) {
+			fprintf(stderr, _("Cannot change the 64bit feature "
+				"while the filesystem is mounted.\n"));
+			exit(1);
+		}
+		if (flags & RESIZE_ENABLE_64BIT &&
+		    !EXT2_HAS_INCOMPAT_FEATURE(fs->super,
+				EXT3_FEATURE_INCOMPAT_EXTENTS)) {
+			fprintf(stderr, _("Please enable the extents feature "
+				"with tune2fs before enabling the 64bit "
+				"feature.\n"));
+			exit(1);
+		}
+	} else if (new_size == ext2fs_blocks_count(fs->super)) {
 		fprintf(stderr, _("The filesystem is already %llu (%dk) "
 			"blocks long.  Nothing to do!\n\n"), new_size,
 			blocksize / 1024);

resize/resize2fs.8.in +17 -1

index 86495c6..0129bfc 100644
@@ -8,7 +8,7 @@ resize2fs \- ext2/ext3/ext4 file system resizer
 .SH SYNOPSIS
 .B resize2fs
 [
-.B \-fFpPM
+.B \-fFpPMbs
 ]
 [
 .B \-d

@@ -86,8 +86,21 @@ to shrink the size of filesystem.  Then you may use
 to shrink the size of the partition.  When shrinking the size of
 the partition, make sure you do not make it smaller than the new size
 of the ext2 filesystem!
+.PP
+The
+.B \-b
+and
+.B \-s
+options enable and disable the 64bit feature, respectively.  The resize2fs
+program will, of course, take care of resizing the block group descriptors
+and moving other data blocks out of the way, as needed.  It is not possible
+to resize the filesystem concurrent with changing the 64bit status.
 .SH OPTIONS
 .TP
+.B \-b
+Turns on the 64bit feature, resizes the group descriptors as necessary, and
+moves other metadata out of the way.
+.TP
 .B \-d \fIdebug-flags
 Turns on various resize2fs debugging features, if they have been compiled
 into the binary.

@@ -127,6 +140,9 @@ of what the program is doing.
 .B \-P
 Print the minimum size of the filesystem and exit.
 .TP
+.B \-s
+Turns off the 64bit feature and frees blocks that are no longer in use.
+.TP
 .B \-S \fIRAID-stride
 The
 .B resize2fs

resize/resize2fs.c +346 ~~-17~~

412 lines not on this page · raw message

resize/resize2fs.h +3 -0

index 7aeab91..829fcd8 100644
@@ -82,6 +82,9 @@ typedef struct ext2_sim_progress *ext2_sim_progmeter;
 #define RESIZE_PERCENT_COMPLETE		0x0100
 #define RESIZE_VERBOSE			0x0200
 
+#define RESIZE_ENABLE_64BIT		0x0400
+#define RESIZE_DISABLE_64BIT		0x0800
+
 /*
  * This structure is used for keeping track of how much resources have
  * been used for a particular resize2fs pass.

Darrick J. Wong

author

2014-10-25 20:59 UTC #40 lore raw

PATCH 28/39 tests: test resize2fs 32->64 and 64->32bit conversion code

Add some simple tests to check that flex_bg and meta_bg filesystems can be converted between 32 and 64bit layouts.

---
 tests/r_32to64bit/expect.gz      |  Bin
 tests/r_32to64bit/name           |    1 +
 tests/r_32to64bit/script         |   75 ++++++++++++++++++++++++++++++++++++++
 tests/r_32to64bit_meta/expect.gz |  Bin
 tests/r_32to64bit_meta/name      |    1 +
 tests/r_32to64bit_meta/script    |   75 ++++++++++++++++++++++++++++++++++++++
 tests/r_64to32bit/expect.gz      |  Bin
 tests/r_64to32bit/name           |    1 +
 tests/r_64to32bit/script         |   75 ++++++++++++++++++++++++++++++++++++++
 tests/r_64to32bit_meta/expect.gz |  Bin
 tests/r_64to32bit_meta/name      |    1 +
 tests/r_64to32bit_meta/script    |   75 ++++++++++++++++++++++++++++++++++++++
 12 files changed, 304 insertions(+)
 create mode 100644 tests/r_32to64bit/expect.gz
 create mode 100644 tests/r_32to64bit/name
 create mode 100644 tests/r_32to64bit/script
 create mode 100644 tests/r_32to64bit_meta/expect.gz
 create mode 100644 tests/r_32to64bit_meta/name
 create mode 100644 tests/r_32to64bit_meta/script
 create mode 100644 tests/r_64to32bit/expect.gz
 create mode 100644 tests/r_64to32bit/name
 create mode 100644 tests/r_64to32bit/script
 create mode 100644 tests/r_64to32bit_meta/expect.gz
 create mode 100644 tests/r_64to32bit_meta/name
 create mode 100644 tests/r_64to32bit_meta/script

tests/r_32to64bit/expect.gz new file binary +0 -0

new file mode 100644
index 0000000000000000000000000000000000000000..4ba35c8ffb7ca9ef3159f1c40dd507c5fdcff180
GIT binary patch

tests/r_32to64bit/name new file +1 -0

new file mode 100644
index 0000000..fb45fab
@@ -0,0 +1 @@
+convert flex_bg 32bit fs to 64bit fs

tests/r_32to64bit/script new file +75 -0

75 lines not on this page · raw message

tests/r_32to64bit_meta/expect.gz new file binary +0 -0

new file mode 100644
index 0000000000000000000000000000000000000000..d6985ecb82b5b7bd9f5046ba99a7e6cf2d60963a
GIT binary patch

tests/r_32to64bit_meta/name new file +1 -0

new file mode 100644
index 0000000..d83492e
@@ -0,0 +1 @@
+convert meta_bg 32bit fs to 64bit fs

tests/r_32to64bit_meta/script new file +75 -0

75 lines not on this page · raw message

tests/r_64to32bit/expect.gz new file binary +0 -0

new file mode 100644
index 0000000000000000000000000000000000000000..375a749e42b2a49724e874761048ac4cb91e010b
GIT binary patch

tests/r_64to32bit/name new file +1 -0

new file mode 100644
index 0000000..4c82371
@@ -0,0 +1 @@
+convert flex_bg 64bit fs to 32bit fs

tests/r_64to32bit/script new file +75 -0

75 lines not on this page · raw message

tests/r_64to32bit_meta/expect.gz new file binary +0 -0

new file mode 100644
index 0000000000000000000000000000000000000000..e39d01f1bc21038a73c3678f11dd66fba8c25aeb
GIT binary patch

tests/r_64to32bit_meta/name new file +1 -0

new file mode 100644
index 0000000..e99ed8b
@@ -0,0 +1 @@
+convert meta_bg 64bit fs to 32bit fs

tests/r_64to32bit_meta/script new file +75 -0

75 lines not on this page · raw message

Darrick J. Wong

author

2014-10-25 21:00 UTC #41 lore raw

PATCH 29/39 libext2fs: find inode goal when allocating blocks

Try to be a little smarter about where we go to allocate blocks for a inode.

---
 e2fsck/pass2.c         |    3 ++-
 lib/ext2fs/alloc.c     |   10 ++++++++++
 lib/ext2fs/bmap.c      |    5 +++--
 lib/ext2fs/expanddir.c |    2 +-
 lib/ext2fs/ext2fs.h    |    1 +
 lib/ext2fs/ext_attr.c  |    4 +---
 lib/ext2fs/extent.c    |   10 ++--------
 lib/ext2fs/mkdir.c     |    3 ++-
 lib/ext2fs/symlink.c   |    3 ++-
 9 files changed, 24 insertions(+), 17 deletions(-)

e2fsck/pass2.c +2 -1

index 2060ed2..fa17f20 100644
@@ -1807,7 +1807,8 @@ static int allocate_dir_block(e2fsck_t ctx,
 	pctx->errcode = ext2fs_map_cluster_block(fs, db->ino, &inode,
 						 db->blockcnt, &blk);
 	if (pctx->errcode || blk == 0) {
-		pctx->errcode = ext2fs_new_block2(fs, 0,
+		blk = ext2fs_find_inode_goal(fs, db->ino);
+		pctx->errcode = ext2fs_new_block2(fs, blk,
 						  ctx->block_found_map, &blk);
 		if (pctx->errcode) {
 			pctx->str = "ext2fs_new_block";

lib/ext2fs/alloc.c +10 -0

index 82af18e..00cc9ba 100644
@@ -303,3 +303,13 @@ void ext2fs_set_alloc_block_callback(ext2_filsys fs,
 
 	fs->get_alloc_block = func;
 }
+
+blk64_t ext2fs_find_inode_goal(ext2_filsys fs, ext2_ino_t ino)
+{
+	dgrp_t	group = ext2fs_group_of_ino(fs, ino);
+	__u8	log_flex = fs->super->s_log_groups_per_flex;
+
+	if (log_flex)
+		group = group & ~((1 << (log_flex)) - 1);
+	return ext2fs_group_first_block2(fs, group);
+}

lib/ext2fs/bmap.c +3 -2

index e936ec1..b0d9087 100644
@@ -247,7 +247,7 @@ got_block:
 		retval = extent_bmap(fs, ino, inode, handle, block_buf,
 				     0, block-1, 0, blocks_alloc, &blk64);
 		if (retval)
-			blk64 = 0;
+			blk64 = ext2fs_find_inode_goal(fs, ino);
 		retval = ext2fs_alloc_block2(fs, blk64, block_buf,
 					     &blk64);
 		if (retval)

@@ -357,7 +357,8 @@ errcode_t ext2fs_bmap2(ext2_filsys fs, ext2_ino_t ino, struct ext2_inode *inode,
 		}
 
 		*phys_blk = inode_bmap(inode, block);
-		b = block ? inode_bmap(inode, block-1) : 0;
+		b = block ? inode_bmap(inode, block-1) :
+			    ext2fs_find_inode_goal(fs, ino);
 
 		if ((*phys_blk == 0) && (bmap_flags & BMAP_ALLOC)) {
 			retval = ext2fs_alloc_block(fs, b, block_buf, &b);

lib/ext2fs/expanddir.c +1 -1

index ecc13ae..e8dff30 100644
@@ -104,7 +104,7 @@ errcode_t ext2fs_expand_dir(ext2_filsys fs, ext2_ino_t dir)
 
 	es.done = 0;
 	es.err = 0;
-	es.goal = 0;
+	es.goal = ext2fs_find_inode_goal(fs, dir);
 	es.newblocks = 0;
 	es.dir = dir;

lib/ext2fs/ext2fs.h +1 -0

index 664c3c0..7941c5d 100644
@@ -697,6 +697,7 @@ extern void ext2fs_set_alloc_block_callback(ext2_filsys fs,
 					    errcode_t (**old)(ext2_filsys fs,
 							      blk64_t goal,
 							      blk64_t *ret));
+blk64_t ext2fs_find_inode_goal(ext2_filsys fs, ext2_ino_t ino);
 
 /* alloc_sb.c */
 extern int ext2fs_reserve_super_and_bgd(ext2_filsys fs,

lib/ext2fs/ext_attr.c +1 -3

index c397e00..43c19c5 100644
@@ -374,7 +374,6 @@ static errcode_t prep_ea_block_for_write(ext2_filsys fs, ext2_ino_t ino,
 {
 	struct ext2_ext_attr_header *header;
 	void *block_buf = NULL;
-	dgrp_t grp;
 	blk64_t blk, goal;
 	errcode_t err;

@@ -420,8 +419,7 @@ static errcode_t prep_ea_block_for_write(ext2_filsys fs, ext2_ino_t ino,
 	}
 
 	/* Allocate a block */
-	grp = ext2fs_group_of_ino(fs, ino);
-	goal = ext2fs_inode_table_loc(fs, grp);
+	goal = ext2fs_find_inode_goal(fs, ino);
 	err = ext2fs_alloc_block2(fs, goal, NULL, &blk);
 	if (err)
 		goto out2;

lib/ext2fs/extent.c +2 -8

index c9ef701..4c6fbbf 100644
@@ -1012,14 +1012,8 @@ static errcode_t extent_node_split(ext2_extent_handle_t handle,
 		goto done;
 	}
 
-	if (!goal_blk) {
-		dgrp_t	group = ext2fs_group_of_ino(handle->fs, handle->ino);
-		__u8	log_flex = handle->fs->super->s_log_groups_per_flex;
-
-		if (log_flex)
-			group = group & ~((1 << (log_flex)) - 1);
-		goal_blk = ext2fs_group_first_block2(handle->fs, group);
-	}
+	if (!goal_blk)
+		goal_blk = ext2fs_find_inode_goal(handle->fs, handle->ino);
 	retval = ext2fs_alloc_block2(handle->fs, goal_blk, block_buf,
 				    &new_node_pblk);
 	if (retval)

lib/ext2fs/mkdir.c +2 -1

index c4c7967..c88ff9e 100644
@@ -69,7 +69,8 @@ errcode_t ext2fs_mkdir(ext2_filsys fs, ext2_ino_t parent, ext2_ino_t inum,
 	 * Allocate a data block for the directory
 	 */
 	if (!inline_data) {
-		retval = ext2fs_new_block2(fs, 0, 0, &blk);
+		retval = ext2fs_new_block2(fs, ext2fs_find_inode_goal(fs, ino),
+					   NULL, &blk);
 		if (retval)
 			goto cleanup;
 	}

lib/ext2fs/symlink.c +2 -1

index f6eb6b6..e268ed4 100644
@@ -53,7 +53,8 @@ errcode_t ext2fs_symlink(ext2_filsys fs, ext2_ino_t parent, ext2_ino_t ino,
 	 */
 	fastlink = (target_len < sizeof(inode.i_block));
 	if (!fastlink) {
-		retval = ext2fs_new_block2(fs, 0, 0, &blk);
+		retval = ext2fs_new_block2(fs, ext2fs_find_inode_goal(fs, ino),
+					   NULL, &blk);
 		if (retval)
 			goto cleanup;
 		retval = ext2fs_get_mem(fs->blocksize, &block_buf);

Darrick J. Wong

author

2014-10-25 21:00 UTC #42 lore raw

PATCH 30/39 libext2fs: find/alloc a range of empty blocks

Provide a function that, given a goal pblk and a range, will try to find a run of free blocks to satisfy the allocation. By default the function will look anywhere in the filesystem for the run, though this can be constrained with optional flags. One flag indicates that the range must start at the goal block; the other flag indicates that we should not return a range shorter than len.

v2: Add a second function to allocate a range of blocks.

---
 lib/ext2fs/alloc.c  |  141 +++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/ext2fs/ext2fs.h |   11 ++++
 2 files changed, 152 insertions(+)

lib/ext2fs/alloc.c +141 -0

index 00cc9ba..45791ff 100644
@@ -26,6 +26,16 @@
 #include "ext2_fs.h"
 #include "ext2fs.h"
 
+#define min(a, b) ((a) < (b) ? (a) : (b))
+
+#undef DEBUG
+
+#ifdef DEBUG
+# define dbg_printf(f, a...)  do {printf(f, ## a); fflush(stdout); } while (0)
+#else
+# define dbg_printf(f, a...)
+#endif
+
 /*
  * Clear the uninit block bitmap flag if necessary
  */

@@ -313,3 +323,134 @@ blk64_t ext2fs_find_inode_goal(ext2_filsys fs, ext2_ino_t ino)
 		group = group & ~((1 << (log_flex)) - 1);
 	return ext2fs_group_first_block2(fs, group);
 }
+
+/*
+ * Starting at _goal_, scan around the filesystem to find a run of free blocks
+ * that's at least _len_ blocks long.  Possible flags:
+ * - EXT2_NEWRANGE_EXACT_GOAL: The range of blocks must start at _goal_.
+ * - EXT2_NEWRANGE_MIN_LENGTH: do not return a allocation shorter than _len_.
+ * - EXT2_NEWRANGE_ZERO_BLOCKS: Zero blocks pblk to pblk+plen before returning.
+ *
+ * The starting block is returned in _pblk_ and the length is returned via
+ * _plen_.  The blocks are not marked in the bitmap; the caller must mark
+ * however much of the returned run they actually use, hopefully via
+ * ext2fs_block_alloc_stats_range().
+ *
+ * This function can return a range that is longer than what was requested.
+ */
+errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
+			   blk64_t len, ext2fs_block_bitmap map, blk64_t *pblk,
+			   blk64_t *plen)
+{
+	errcode_t retval;
+	blk64_t start, end, b;
+	int looped = 0;
+	blk64_t max_blocks = ext2fs_blocks_count(fs->super);
+
+	dbg_printf("%s: flags=0x%x goal=%llu len=%llu\n", __func__, flags,
+		   goal, len);
+	EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS);
+	if (len == 0 || (flags & ~EXT2_NEWRANGE_ALL_FLAGS))
+		return EXT2_ET_INVALID_ARGUMENT;
+	if (!map)
+		map = fs->block_map;
+	if (!map)
+		return EXT2_ET_NO_BLOCK_BITMAP;
+	if (!goal || goal >= ext2fs_blocks_count(fs->super))
+		goal = fs->super->s_first_data_block;
+
+	start = goal;
+	while (!looped || start <= goal) {
+		retval = ext2fs_find_first_zero_block_bitmap2(map, start,
+							      max_blocks - 1,
+							      &start);
+		if (retval == ENOENT) {
+			/*
+			 * If there are no free blocks beyond the starting
+			 * point, try scanning the whole filesystem, unless the
+			 * user told us only to allocate from _goal_, or if
+			 * we're already scanning the whole filesystem.
+			 */
+			if (flags & EXT2_NEWRANGE_FIXED_GOAL ||
+			    start == fs->super->s_first_data_block)
+				goto fail;
+			start = fs->super->s_first_data_block;
+			continue;
+		} else if (retval)
+			goto errout;
+
+		if (flags & EXT2_NEWRANGE_FIXED_GOAL && start != goal)
+			goto fail;
+
+		b = min(start + len - 1, max_blocks - 1);
+		retval =  ext2fs_find_first_set_block_bitmap2(map, start, b,
+							      &end);
+		if (retval == ENOENT)
+			end = b + 1;
+		else if (retval)
+			goto errout;
+
+		if (!(flags & EXT2_NEWRANGE_MIN_LENGTH) ||
+		    (end - start) >= len) {
+			/* Success! */
+			*pblk = start;
+			*plen = end - start;
+			dbg_printf("%s: new_range goal=%llu--%llu "
+				   "blk=%llu--%llu %llu\n",
+				   __func__, goal, goal + len - 1,
+				   *pblk, *pblk + *plen - 1, *plen);
+
+			for (b = start; b < end;
+			     b += fs->super->s_blocks_per_group)
+				clear_block_uninit(fs,
+						ext2fs_group_of_blk2(fs, b));
+			return 0;
+		}
+
+		if (flags & EXT2_NEWRANGE_FIXED_GOAL)
+			goto fail;
+		start = end;
+		if (start >= max_blocks) {
+			if (looped)
+				goto fail;
+			looped = 1;
+			start = fs->super->s_first_data_block;
+		}
+	}
+
+fail:
+	retval = EXT2_ET_BLOCK_ALLOC_FAIL;
+errout:
+	return retval;
+}
+
+errcode_t ext2fs_alloc_range(ext2_filsys fs, int flags, blk64_t goal,
+			     blk_t len, blk64_t *ret)
+{
+	int newr_flags = EXT2_NEWRANGE_MIN_LENGTH;
+	errcode_t retval;
+	blk64_t plen;
+
+	EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS);
+	if (len == 0 || (flags & ~EXT2_ALLOCRANGE_ALL_FLAGS))
+		return EXT2_ET_INVALID_ARGUMENT;
+
+	if (flags & EXT2_ALLOCRANGE_FIXED_GOAL)
+		newr_flags |= EXT2_NEWRANGE_FIXED_GOAL;
+
+	retval = ext2fs_new_range(fs, newr_flags, goal, len, NULL, ret, &plen);
+	if (retval)
+		return retval;
+
+	if (plen < len)
+		return EXT2_ET_BLOCK_ALLOC_FAIL;
+
+	if (flags & EXT2_ALLOCRANGE_ZERO_BLOCKS) {
+		retval = ext2fs_zero_blocks2(fs, *ret, len, NULL, NULL);
+		if (retval)
+			return retval;
+	}
+
+	ext2fs_block_alloc_stats_range(fs, *ret, len, +1);
+	return retval;
+}

lib/ext2fs/ext2fs.h +11 -0

index 7941c5d..91e1bef 100644
@@ -698,6 +698,17 @@ extern void ext2fs_set_alloc_block_callback(ext2_filsys fs,
 							      blk64_t goal,
 							      blk64_t *ret));
 blk64_t ext2fs_find_inode_goal(ext2_filsys fs, ext2_ino_t ino);
+#define EXT2_NEWRANGE_FIXED_GOAL	(0x1)
+#define EXT2_NEWRANGE_MIN_LENGTH	(0x2)
+#define EXT2_NEWRANGE_ALL_FLAGS		(0x3)
+errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
+			   blk64_t len, ext2fs_block_bitmap map, blk64_t *pblk,
+			   blk64_t *plen);
+#define EXT2_ALLOCRANGE_FIXED_GOAL	(0x1)
+#define EXT2_ALLOCRANGE_ZERO_BLOCKS	(0x2)
+#define EXT2_ALLOCRANGE_ALL_FLAGS	(0x3)
+errcode_t ext2fs_alloc_range(ext2_filsys fs, int flags, blk64_t goal,
+			     blk_t len, blk64_t *ret);
 
 /* alloc_sb.c */
 extern int ext2fs_reserve_super_and_bgd(ext2_filsys fs,

Darrick J. Wong

author

2014-10-25 21:00 UTC #43 lore raw

PATCH 31/39 libext2fs: add new hooks to support large allocations

Add a new get_alloc_blocks hook and a block_alloc_stats_range hook so that e2fsck can capture allocation requests spanning more than a block to its block_found_map.

---
 e2fsck/pass1.c           |   45 +++++++++++++++++++++++++++++++++++++++++++++
 lib/ext2fs/alloc.c       |   37 ++++++++++++++++++++++++++++++++++++-
 lib/ext2fs/alloc_stats.c |   16 ++++++++++++++++
 lib/ext2fs/ext2fs.h      |   16 ++++++++++++++++
 4 files changed, 113 insertions(+), 1 deletion(-)

e2fsck/pass1.c +45 -0

index a963849..2d59f63 100644
@@ -3802,6 +3802,26 @@ static errcode_t e2fsck_get_alloc_block(ext2_filsys fs, blk64_t goal,
 	return (0);
 }
 
+static errcode_t e2fsck_new_range(ext2_filsys fs, int flags, blk64_t goal,
+				  blk64_t len, blk64_t *pblk, blk64_t *plen)
+{
+	e2fsck_t ctx = (e2fsck_t) fs->priv_data;
+	errcode_t	retval;
+
+	if (ctx->block_found_map)
+		return ext2fs_new_range(fs, flags, goal, len,
+					ctx->block_found_map, pblk, plen);
+
+	if (!fs->block_map) {
+		retval = ext2fs_read_block_bitmap(fs);
+		if (retval)
+			return retval;
+	}
+
+	return ext2fs_new_range(fs, flags, goal, len, fs->block_map,
+				pblk, plen);
+}
+
 static void e2fsck_block_alloc_stats(ext2_filsys fs, blk64_t blk, int inuse)
 {
 	e2fsck_t ctx = (e2fsck_t) fs->priv_data;

@@ -3821,6 +3841,28 @@ static void e2fsck_block_alloc_stats(ext2_filsys fs, blk64_t blk, int inuse)
 	}
 }
 
+static void e2fsck_block_alloc_stats_range(ext2_filsys fs, blk64_t blk,
+					   blk_t num, int inuse)
+{
+	e2fsck_t ctx = (e2fsck_t) fs->priv_data;
+
+	/* Never free a critical metadata block */
+	if (ctx->block_found_map &&
+	    ctx->block_metadata_map &&
+	    inuse < 0 &&
+	    ext2fs_test_block_bitmap_range2(ctx->block_metadata_map, blk, num))
+		return;
+
+	if (ctx->block_found_map) {
+		if (inuse > 0)
+			ext2fs_mark_block_bitmap_range2(ctx->block_found_map,
+							blk, num);
+		else
+			ext2fs_unmark_block_bitmap_range2(ctx->block_found_map,
+							blk, num);
+	}
+}
+
 void e2fsck_use_inode_shortcuts(e2fsck_t ctx, int use_shortcuts)
 {
 	ext2_filsys fs = ctx->fs;

@@ -3844,4 +3886,7 @@ void e2fsck_intercept_block_allocations(e2fsck_t ctx)
 	ext2fs_set_alloc_block_callback(ctx->fs, e2fsck_get_alloc_block, 0);
 	ext2fs_set_block_alloc_stats_callback(ctx->fs,
 						e2fsck_block_alloc_stats, 0);
+	ext2fs_set_new_range_callback(ctx->fs, e2fsck_new_range, NULL);
+	ext2fs_set_block_alloc_stats_range_callback(ctx->fs,
+					e2fsck_block_alloc_stats_range, NULL);
 }

lib/ext2fs/alloc.c +36 -1

index 45791ff..8070801 100644
@@ -346,12 +346,32 @@ errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
 	blk64_t start, end, b;
 	int looped = 0;
 	blk64_t max_blocks = ext2fs_blocks_count(fs->super);
+	errcode_t (*nrf)(ext2_filsys fs, int flags, blk64_t goal,
+			 blk64_t len, blk64_t *pblk, blk64_t *plen);
 
 	dbg_printf("%s: flags=0x%x goal=%llu len=%llu\n", __func__, flags,
 		   goal, len);
 	EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS);
 	if (len == 0 || (flags & ~EXT2_NEWRANGE_ALL_FLAGS))
 		return EXT2_ET_INVALID_ARGUMENT;
+
+	if (!map && fs->new_range) {
+		/*
+		 * In case there are clients out there whose new_range
+		 * handlers call ext2fs_new_range with a NULL block map,
+		 * temporarily swap out the function pointer so that we don't
+		 * end up in an infinite loop.
+		 */
+		nrf = fs->new_range;
+		fs->new_range = NULL;
+		retval = nrf(fs, flags, goal, len, pblk, plen);
+		fs->new_range = nrf;
+		if (retval)
+			return retval;
+		start = *pblk;
+		end = *pblk + *plen;
+		goto allocated;
+	}
 	if (!map)
 		map = fs->block_map;
 	if (!map)

@@ -399,7 +419,7 @@ errcode_t ext2fs_new_range(ext2_filsys fs, int flags, blk64_t goal,
 				   "blk=%llu--%llu %llu\n",
 				   __func__, goal, goal + len - 1,
 				   *pblk, *pblk + *plen - 1, *plen);
-
+allocated:
 			for (b = start; b < end;
 			     b += fs->super->s_blocks_per_group)
 				clear_block_uninit(fs,

@@ -424,6 +444,21 @@ errout:
 	return retval;
 }
 
+void ext2fs_set_new_range_callback(ext2_filsys fs,
+	errcode_t (*func)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen),
+	errcode_t (**old)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen))
+{
+	if (!fs || fs->magic != EXT2_ET_MAGIC_EXT2FS_FILSYS)
+		return;
+
+	if (old)
+		*old = fs->new_range;
+
+	fs->new_range = func;
+}
+
 errcode_t ext2fs_alloc_range(ext2_filsys fs, int flags, blk64_t goal,
 			     blk_t len, blk64_t *ret)
 {

lib/ext2fs/alloc_stats.c +16 -0

index 3d3697c..5999082 100644
@@ -145,4 +145,20 @@ void ext2fs_block_alloc_stats_range(ext2_filsys fs, blk64_t blk,
 	}
 	ext2fs_mark_super_dirty(fs);
 	ext2fs_mark_bb_dirty(fs);
+	if (fs->block_alloc_stats_range)
+		(fs->block_alloc_stats_range)(fs, blk, num, inuse);
+}
+
+void ext2fs_set_block_alloc_stats_range_callback(ext2_filsys fs,
+	void (*func)(ext2_filsys fs, blk64_t blk,
+				    blk_t num, int inuse),
+	void (**old)(ext2_filsys fs, blk64_t blk,
+				    blk_t num, int inuse))
+{
+	if (!fs || fs->magic != EXT2_ET_MAGIC_EXT2FS_FILSYS)
+		return;
+	if (old)
+		*old = fs->block_alloc_stats_range;
+
+	fs->block_alloc_stats_range = func;
 }

lib/ext2fs/ext2fs.h +16 -0

index 91e1bef..b1169cd 100644
@@ -285,6 +285,12 @@ struct struct_ext2_filsys {
 
 	io_channel			journal_io;
 	char				*journal_name;
+
+	/* New block range allocation hooks */
+	errcode_t (*new_range)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen);
+	void (*block_alloc_stats_range)(ext2_filsys fs, blk64_t blk, blk_t num,
+					int inuse);
 };
 
 #if EXT2_FLAT_INCLUDES

@@ -697,6 +703,16 @@ extern void ext2fs_set_alloc_block_callback(ext2_filsys fs,
 					    errcode_t (**old)(ext2_filsys fs,
 							      blk64_t goal,
 							      blk64_t *ret));
+extern void ext2fs_set_new_range_callback(ext2_filsys fs,
+	errcode_t (*func)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen),
+	errcode_t (**old)(ext2_filsys fs, int flags, blk64_t goal,
+			       blk64_t len, blk64_t *pblk, blk64_t *plen));
+extern void ext2fs_set_block_alloc_stats_range_callback(ext2_filsys fs,
+	void (*func)(ext2_filsys fs, blk64_t blk,
+				    blk_t num, int inuse),
+	void (**old)(ext2_filsys fs, blk64_t blk,
+				    blk_t num, int inuse));
 blk64_t ext2fs_find_inode_goal(ext2_filsys fs, ext2_ino_t ino);
 #define EXT2_NEWRANGE_FIXED_GOAL	(0x1)
 #define EXT2_NEWRANGE_MIN_LENGTH	(0x2)

Darrick J. Wong

author

2014-10-25 21:00 UTC #44 lore raw

PATCH 32/39 libext2fs: implement fallocate

Create a library function to perform fallocation on arbitrary files, and wire up a few users for this function. This is a bit more intense than Ted's original mk_hugefiles implementation since we have to honor any blocks that may already be allocated to the file.

---
 lib/ext2fs/Makefile.in |    8 
 lib/ext2fs/bmap.c      |    2 
 lib/ext2fs/ext2fs.h    |   10 +
 lib/ext2fs/fallocate.c |  853 ++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 872 insertions(+), 1 deletion(-)
 create mode 100644 lib/ext2fs/fallocate.c

lib/ext2fs/Makefile.in +8 -0

index dc3e144..cd7dcc5 100644
@@ -79,6 +79,7 @@ OBJS= $(DEBUGFS_LIB_OBJS) $(RESIZE_LIB_OBJS) $(E2IMAGE_LIB_OBJS) \
 	expanddir.o \
 	ext_attr.o \
 	extent.o \
+	fallocate.o \
 	fileio.o \
 	finddev.o \
 	flushb.o \

@@ -763,6 +764,13 @@ extent.o: $(srcdir)/extent.c $(top_builddir)/lib/config.h \
  $(top_srcdir)/lib/et/com_err.h $(srcdir)/ext2_io.h \
  $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/ext2_ext_attr.h \
  $(srcdir)/bitops.h $(srcdir)/e2image.h
+fallocate.o: $(srcdir)/fallocate.c $(top_builddir)/lib/config.h \
+ $(top_builddir)/lib/dirpaths.h $(srcdir)/ext2_fs.h \
+ $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fsP.h \
+ $(srcdir)/ext2fs.h $(srcdir)/ext2_fs.h $(srcdir)/ext3_extents.h \
+ $(top_srcdir)/lib/et/com_err.h $(srcdir)/ext2_io.h \
+ $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/ext2_ext_attr.h \
+ $(srcdir)/bitops.h $(srcdir)/e2image.h
 fileio.o: $(srcdir)/fileio.c $(top_builddir)/lib/config.h \
  $(top_builddir)/lib/dirpaths.h $(srcdir)/ext2_fs.h \
  $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \

lib/ext2fs/bmap.c +1 -1

index b0d9087..0cb3cc8 100644
@@ -67,7 +67,7 @@ static _BMAP_INLINE_ errcode_t block_ind_bmap(ext2_filsys fs, int flags,
 #endif
 
 	if (!b && (flags & BMAP_ALLOC)) {
-		b = nr ? ((blk_t *) block_buf)[nr-1] : 0;
+		b = nr ? ((blk_t *) block_buf)[nr-1] : ind;
 		retval = ext2fs_alloc_block(fs, b,
 					    block_buf + fs->blocksize, &b);
 		if (retval)

lib/ext2fs/ext2fs.h +10 -0

index b1169cd..1b97c93 100644
@@ -1259,6 +1259,16 @@ extern errcode_t ext2fs_extent_goto2(ext2_extent_handle_t handle,
 				     int leaf_level, blk64_t blk);
 extern errcode_t ext2fs_extent_fix_parents(ext2_extent_handle_t handle);
 
+/* fallocate.c */
+#define EXT2_FALLOCATE_ZERO_BLOCKS	(0x1)
+#define EXT2_FALLOCATE_FORCE_INIT	(0x2)
+#define EXT2_FALLOCATE_FORCE_UNINIT	(0x4)
+#define EXT2_FALLOCATE_INIT_BEYOND_EOF	(0x8)
+#define EXT2_FALLOCATE_ALL_FLAGS	(0xF)
+errcode_t ext2fs_fallocate(ext2_filsys fs, int flags, ext2_ino_t ino,
+			   struct ext2_inode *inode, blk64_t goal,
+			   blk64_t start, blk64_t len);
+
 /* fileio.c */
 extern errcode_t ext2fs_file_open2(ext2_filsys fs, ext2_ino_t ino,
 				   struct ext2_inode *inode,

lib/ext2fs/fallocate.c new file +853 -0

853 lines not on this page · raw message

Darrick J. Wong

author

2014-10-25 21:00 UTC #45 lore raw

PATCH 33/39 libext2fs: use fallocate for creating journals and hugefiles

Use the new fallocate API for creating the journal and the mk_hugefile feature.

---
 lib/ext2fs/mkjournal.c |  134 +++++++-----------------------------------------
 misc/mk_hugefiles.c    |   96 ++++------------------------------
 2 files changed, 30 insertions(+), 200 deletions(-)

lib/ext2fs/mkjournal.c +20 ~~-114~~

index 2cd7ca5..9378314 100644
@@ -250,89 +250,6 @@ errcode_t ext2fs_zero_blocks(ext2_filsys fs, blk_t blk, int num,
 }
 
 /*
- * Helper function for creating the journal using direct I/O routines
- */
-struct mkjournal_struct {
-	int		num_blocks;
-	int		newblocks;
-	blk64_t		goal;
-	blk64_t		blk_to_zero;
-	int		zero_count;
-	int		flags;
-	char		*buf;
-	errcode_t	err;
-};
-
-static int mkjournal_proc(ext2_filsys	fs,
-			  blk64_t	*blocknr,
-			  e2_blkcnt_t	blockcnt,
-			  blk64_t	ref_block EXT2FS_ATTR((unused)),
-			  int		ref_offset EXT2FS_ATTR((unused)),
-			  void		*priv_data)
-{
-	struct mkjournal_struct *es = (struct mkjournal_struct *) priv_data;
-	blk64_t	new_blk;
-	errcode_t	retval;
-
-	if (*blocknr) {
-		es->goal = *blocknr;
-		return 0;
-	}
-	if (blockcnt &&
-	    (EXT2FS_B2C(fs, es->goal) == EXT2FS_B2C(fs, es->goal+1)))
-		new_blk = es->goal+1;
-	else {
-		es->goal &= ~EXT2FS_CLUSTER_MASK(fs);
-		retval = ext2fs_new_block2(fs, es->goal, 0, &new_blk);
-		if (retval) {
-			es->err = retval;
-			return BLOCK_ABORT;
-		}
-		ext2fs_block_alloc_stats2(fs, new_blk, +1);
-		es->newblocks++;
-	}
-	if (blockcnt >= 0)
-		es->num_blocks--;
-
-	retval = 0;
-	if (blockcnt <= 0)
-		retval = io_channel_write_blk64(fs->io, new_blk, 1, es->buf);
-	else if (!(es->flags & EXT2_MKJOURNAL_LAZYINIT)) {
-		if (es->zero_count) {
-			if ((es->blk_to_zero + es->zero_count == new_blk) &&
-			    (es->zero_count < 1024))
-				es->zero_count++;
-			else {
-				retval = ext2fs_zero_blocks2(fs,
-							     es->blk_to_zero,
-							     es->zero_count,
-							     0, 0);
-				es->zero_count = 0;
-			}
-		}
-		if (es->zero_count == 0) {
-			es->blk_to_zero = new_blk;
-			es->zero_count = 1;
-		}
-	}
-
-	if (blockcnt == 0)
-		memset(es->buf, 0, fs->blocksize);
-
-	if (retval) {
-		es->err = retval;
-		return BLOCK_ABORT;
-	}
-	*blocknr = es->goal = new_blk;
-
-	if (es->num_blocks == 0)
-		return (BLOCK_CHANGED | BLOCK_ABORT);
-	else
-		return BLOCK_CHANGED;
-
-}
-
-/*
  * Calculate the initial goal block to be roughly at the middle of the
  * filesystem.  Pick a group that has the largest number of free
  * blocks.

@@ -373,7 +290,8 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 	errcode_t		retval;
 	struct ext2_inode	inode;
 	unsigned long long	inode_size;
-	struct mkjournal_struct	es;
+	int			falloc_flags = EXT2_FALLOCATE_FORCE_INIT;
+	blk64_t			zblk;
 
 	if ((retval = ext2fs_create_journal_superblock(fs, num_blocks, flags,
 						       &buf)))

@@ -390,40 +308,16 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 		goto out2;
 	}
 
-	es.num_blocks = num_blocks;
-	es.newblocks = 0;
-	es.buf = buf;
-	es.err = 0;
-	es.flags = flags;
-	es.zero_count = 0;
-	es.goal = (goal != ~0ULL) ? goal : get_midpoint_journal_block(fs);
+	if (goal == ~0ULL)
+		goal = get_midpoint_journal_block(fs);
 
-	if (fs->super->s_feature_incompat & EXT3_FEATURE_INCOMPAT_EXTENTS) {
+	if (fs->super->s_feature_incompat & EXT3_FEATURE_INCOMPAT_EXTENTS)
 		inode.i_flags |= EXT4_EXTENTS_FL;
-		if ((retval = ext2fs_write_inode(fs, journal_ino, &inode)))
-			goto out2;
-	}
 
-	retval = ext2fs_block_iterate3(fs, journal_ino, BLOCK_FLAG_APPEND,
-				       0, mkjournal_proc, &es);
-	if (retval)
-		goto out2;
-	if (es.err) {
-		retval = es.err;
-		goto out2;
-	}
-	if (es.zero_count) {
-		retval = ext2fs_zero_blocks2(fs, es.blk_to_zero,
-					    es.zero_count, 0, 0);
-		if (retval)
-			goto out2;
-	}
-
-	if ((retval = ext2fs_read_inode(fs, journal_ino, &inode)))
-		goto out2;
+	if (!(flags & EXT2_MKJOURNAL_LAZYINIT))
+		falloc_flags |= EXT2_FALLOCATE_ZERO_BLOCKS;
 
 	inode_size = (unsigned long long)fs->blocksize * num_blocks;
-	ext2fs_iblk_add_blocks(fs, &inode, es.newblocks);
 	inode.i_mtime = inode.i_ctime = fs->now ? fs->now : time(0);
 	inode.i_links_count = 1;
 	inode.i_mode = LINUX_S_IFREG | 0600;

@@ -431,9 +325,21 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino,
 	if (retval)
 		goto out2;
 
+	retval = ext2fs_fallocate(fs, falloc_flags, journal_ino,
+				  &inode, goal, 0, num_blocks);
+	if (retval)
+		goto out2;
+
 	if ((retval = ext2fs_write_new_inode(fs, journal_ino, &inode)))
 		goto out2;
-	retval = 0;
+
+	retval = ext2fs_bmap2(fs, journal_ino, &inode, NULL, 0, 0, NULL, &zblk);
+	if (retval)
+		goto out2;
+
+	retval = io_channel_write_blk64(fs->io, zblk, 1, buf);
+	if (retval)
+		goto out2;
 
 	memcpy(fs->super->s_jnl_blocks, inode.i_block, EXT2_N_BLOCKS*4);
 	fs->super->s_jnl_blocks[15] = inode.i_size_high;

misc/mk_hugefiles.c +10 ~~-85~~

116 lines not on this page · raw message

Darrick J. Wong

author

2014-10-25 21:00 UTC #46 lore raw

PATCH 34/39 debugfs: implement fallocate

Implement a fallocate function for debugfs, and add some tests to demonstrate that it works (more or less).

---
 debugfs/debug_cmds.ct                |    3 +
 debugfs/debugfs.c                    |   36 +++++++
 debugfs/debugfs.h                    |    1 
 tests/d_fallocate/expect.gz          |  Bin
 tests/d_fallocate/name               |    1 
 tests/d_fallocate/script             |  175 ++++++++++++++++++++++++++++++++++
 tests/d_fallocate_bigalloc/expect.gz |  Bin
 tests/d_fallocate_bigalloc/name      |    1 
 tests/d_fallocate_bigalloc/script    |  176 ++++++++++++++++++++++++++++++++++
 tests/d_fallocate_blkmap/expect      |   58 +++++++++++
 tests/d_fallocate_blkmap/name        |    1 
 tests/d_fallocate_blkmap/script      |   85 ++++++++++++++++
 12 files changed, 537 insertions(+)
 create mode 100644 tests/d_fallocate/expect.gz
 create mode 100644 tests/d_fallocate/name
 create mode 100644 tests/d_fallocate/script
 create mode 100644 tests/d_fallocate_bigalloc/expect.gz
 create mode 100644 tests/d_fallocate_bigalloc/name
 create mode 100644 tests/d_fallocate_bigalloc/script
 create mode 100644 tests/d_fallocate_blkmap/expect
 create mode 100644 tests/d_fallocate_blkmap/name
 create mode 100644 tests/d_fallocate_blkmap/script

debugfs/debug_cmds.ct +3 -0

index c6f6d6c..34dad9e 100644
@@ -157,6 +157,9 @@ request do_dirsearch, "Search a directory for a particular filename",
 request do_bmap, "Calculate the logical->physical block mapping for an inode",
 	bmap;
 
+request do_fallocate, "Allocate uninitialized blocks to an inode",
+	fallocate;
+
 request do_punch, "Punch (or truncate) blocks from an inode by deallocating them",
 	punch, truncate;

debugfs/debugfs.c +36 -0

index db85028..b30a5ab 100644
@@ -2073,6 +2073,42 @@ void do_punch(int argc, char *argv[])
 		return;
 	}
 }
+
+void do_fallocate(int argc, char *argv[])
+{
+	ext2_ino_t	ino;
+	blk64_t		start, end;
+	int		err;
+	errcode_t	errcode;
+
+	if (common_args_process(argc, argv, 3, 4, argv[0],
+				"<file> start_blk [end_blk]",
+				CHECK_FS_RW | CHECK_FS_BITMAPS))
+		return;
+
+	ino = string_to_inode(argv[1]);
+	if (!ino)
+		return;
+	err = strtoblk(argv[0], argv[2], "logical block", &start);
+	if (err)
+		return;
+	if (argc == 4) {
+		err = strtoblk(argv[0], argv[3], "logical block", &end);
+		if (err)
+			return;
+	} else
+		end = ~0;
+
+	errcode = ext2fs_fallocate(current_fs, EXT2_FALLOCATE_INIT_BEYOND_EOF,
+				   ino, NULL, ~0ULL, start, end - start + 1);
+
+	if (errcode) {
+		com_err(argv[0], errcode,
+			"while fallocating inode %u from %llu to %llu\n", ino,
+			(unsigned long long) start, (unsigned long long) end);
+		return;
+	}
+}
 #endif /* READ_ONLY */
 
 void do_symlink(int argc, char *argv[])

debugfs/debugfs.h +1 -0

index e163d0a..76bb22c 100644
@@ -166,6 +166,7 @@ extern void do_imap(int argc, char **argv);
 extern void do_set_current_time(int argc, char **argv);
 extern void do_supported_features(int argc, char **argv);
 extern void do_punch(int argc, char **argv);
+extern void do_fallocate(int argc, char **argv);
 extern void do_symlink(int argc, char **argv);
 
 extern void do_dump_mmp(int argc, char **argv);

tests/d_fallocate/expect.gz new file binary +0 -0

new file mode 100644
index 0000000000000000000000000000000000000000..59d6263ab66ce86f46d3a617269427f13b9ddfde
GIT binary patch

tests/d_fallocate/name new file +1 -0

new file mode 100644
index 0000000..72d0ed3
@@ -0,0 +1 @@
+fallocate sparse files and big files

tests/d_fallocate/script new file +175 -0

175 lines not on this page · raw message

tests/d_fallocate_bigalloc/expect.gz new file binary +0 -0

new file mode 100644
index 0000000000000000000000000000000000000000..1f88dcdc60b63e9984e21d50ecdee982bd327aed
GIT binary patch

tests/d_fallocate_bigalloc/name new file +1 -0

new file mode 100644
index 0000000..915645c
@@ -0,0 +1 @@
+fallocate sparse files and big files with bigalloc

tests/d_fallocate_bigalloc/script new file +176 -0

176 lines not on this page · raw message

tests/d_fallocate_blkmap/expect new file +38 -0

new file mode 100644
index 0000000..f7ae606
@@ -0,0 +1,58 @@
+Creating filesystem with 65536 1k blocks and 4096 inodes
+Superblock backups stored on blocks: 
+	8193, 24577, 40961, 57345
+
+Allocating group tables:    done                            
+Writing inode tables:    done                            
+Writing superblocks and filesystem accounting information:    done
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 11/4096 files (0.0% non-contiguous), 2340/65536 blocks
+Exit status is 0
+debugfs write files
+debugfs: stat /a
+Inode: 12   Type: regular    Mode:  0666   Flags: 0x0
+Generation: 0    Version: 0x00000000:00000000
+User:     0   Group:     0   Size: 40960
+File ACL: 0    Directory ACL: 0
+Links: 1   Blockcount: 82
+Fragment:  Address: 0    Number: 0    Size: 0
+Size of extra inode fields: 28
+BLOCKS:
+(0-1):1312-1313, (2-11):8000-8009, (IND):8010, (12-39):8011-8038
+TOTAL: 41
+
+debugfs: stat /b
+Inode: 13   Type: regular    Mode:  0666   Flags: 0x0
+Generation: 0    Version: 0x00000000:00000000
+User:     0   Group:     0   Size: 10240000
+File ACL: 0    Directory ACL: 0
+Links: 1   Blockcount: 20082
+Fragment:  Address: 0    Number: 0    Size: 0
+Size of extra inode fields: 28
+BLOCKS:
+(0-11):10000-10011, (IND):10012, (12-267):10013-10268, (DIND):10269, (IND):10270, (268-523):10271-10526, (IND):10527, (524-779):10528-10783, (IND):10784, (780-1035):10785-11040, (IND):11041, (1036-1291):11042-11297, (IND):11298, (1292-1547):11299-11554, (IND):11555, (1548-1803):11556-11811, (IND):11812, (1804-2059):11813-12068, (IND):12069, (2060-2315):12070-12325, (IND):12326, (2316-2571):12327-12582, (IND):12583, (2572-2827):12584-12839, (IND):12840, (2828-3083):12841-13096, (IND):13097, (3084-3339):13098-13353, (IND):13354, (3340-3595):13355-13610, (IND):13611, (3596-3851):13612-13867, (IND):13868, (3852-4107):13869-14124, (IND):14125, (4108-4363):14126-14381, (IND):14382, (4364-4619):14383-14638, (IND):14639, (4620-4875):14640-14895, (IND):14896, (4876-5131):14897-15152, (IND):15153,

(5132-5387):15154-15409, (IND):15410, (5388-5643):15411-15666, (IND):15667, (5644-5899):15668-15923, (IND):15924, (5900-6155):15925-16180, (IND):16181, (6156-6411):16182-16437, (IND):16438,!
(6412-6667):16439-16694, (IND):16695, (6668-6923):16696-16951, (IND):16952, (6924-7179):16953-17208, (IND):17209, (7180-7435):17210-17465, (IND):17466, (7436-7691):17467-17722, (IND):17723, (7692-7947):17724-17979, (IND):17980, (7948-8203):17981-18236, (IND):18237, (8204-8459):18238-18493, (IND):18494, (8460-8715):18495-18750, (IND):18751, (8716-8971):18752-19007, (IND):19008, (8972-9227):19009-19264, (IND):19265, (9228-9483):19266-19521, (IND):19522, (9484-9739):19523-19778, (IND):19779, (9740-9995):19780-20035, (IND):20036, (9996-9999):20037-20040
+TOTAL: 10041
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+Free blocks count wrong for group #0 (6841, counted=6840).
+Fix? yes
+
+Free blocks count wrong for group #1 (1551, counted=1550).
+Fix? yes
+
+Free blocks count wrong (53116, counted=53114).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 13/4096 files (7.7% non-contiguous), 12422/65536 blocks
+Exit status is 1

tests/d_fallocate_blkmap/name new file +1 -0

new file mode 100644
index 0000000..ba2b61d
@@ -0,0 +1 @@
+fallocate sparse files and big files on a blockmap fs

tests/d_fallocate_blkmap/script new file +85 -0

85 lines not on this page · raw message

Darrick J. Wong

author

2014-10-25 21:00 UTC #47 lore raw

PATCH 35/39 tests: test debugfs punch command

Test punching out various parts of sparse files.

---
 tests/d_punch/expect          |  143 +++++++++++++++++++++++++++++++++++++++++
 tests/d_punch/name            |    1 
 tests/d_punch/script          |  129 +++++++++++++++++++++++++++++++++++++
 tests/d_punch_bigalloc/expect |  142 +++++++++++++++++++++++++++++++++++++++++
 tests/d_punch_bigalloc/name   |    1 
 tests/d_punch_bigalloc/script |  130 +++++++++++++++++++++++++++++++++++++
 6 files changed, 546 insertions(+)
 create mode 100644 tests/d_punch/expect
 create mode 100644 tests/d_punch/name
 create mode 100644 tests/d_punch/script
 create mode 100644 tests/d_punch_bigalloc/expect
 create mode 100644 tests/d_punch_bigalloc/name
 create mode 100644 tests/d_punch_bigalloc/script

tests/d_punch/expect new file +143 -0

new file mode 100644
index 0000000..a5b7fd8
@@ -0,0 +1,143 @@
+Creating filesystem with 65536 1k blocks and 4096 inodes
+Superblock backups stored on blocks: 
+	8193, 24577, 40961, 57345
+
+Allocating group tables:    done                            
+Writing inode tables:    done                            
+Writing superblocks and filesystem accounting information:    done
+
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+test_filesys: 11/4096 files (0.0% non-contiguous), 2345/65536 blocks
+Exit status is 0
+debugfs write files
+debugfs: ex /a
+Level Entries       Logical      Physical Length Flags
+debugfs: ex /sample
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1323              0
+ 1/ 1   1/  5     0 -     9  1313 -  1322     10 Uninit
+ 1/ 1   2/  5    11 -    12  1324 -  1325      2 Uninit
+ 1/ 1   3/  5    14 -    25  1327 -  1338     12 Uninit
+ 1/ 1   4/  5    27 -    28  1340 -  1341      2 Uninit
+ 1/ 1   5/  5    30 -    39  1343 -  1352     10 Uninit
+debugfs: ex /b8
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1390              0
+ 1/ 1   1/  4     0 -     0  1326 -  1326      1 Uninit
+ 1/ 1   2/  4     1 -     1  1339 -  1339      1 Uninit
+ 1/ 1   3/  4     2 -     2  1342 -  1342      1 Uninit
+ 1/ 1   4/  4     3 -     7  1353 -  1357      5 Uninit
+debugfs: ex /b9
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1368              0
+ 1/ 1   1/  1     0 -     8  1358 -  1366      9 Uninit
+debugfs: ex /b10
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1378              0
+ 1/ 1   1/  2     0 -     0  1367 -  1367      1 Uninit
+ 1/ 1   2/  2     1 -     9  1369 -  1377      9 Uninit
+debugfs: ex /b11
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1389              0
+ 1/ 1   1/  1     0 -     9  1379 -  1388     10 Uninit
+debugfs: ex /b12
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1401              0
+ 1/ 1   1/  2     0 -     9  1391 -  1400     10 Uninit
+ 1/ 1   2/  2    11 -    11  1402 -  1402      1 Uninit
+debugfs: ex /b13
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1413              0
+ 1/ 1   1/  2     0 -     9  1403 -  1412     10 Uninit
+ 1/ 1   2/  2    11 -    12  1414 -  1415      2 Uninit
+debugfs: ex /b14
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1426              0
+ 1/ 1   1/  2     0 -     9  1416 -  1425     10 Uninit
+ 1/ 1   2/  2    11 -    12  1427 -  1428      2 Uninit
+debugfs: ex /b15
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1439              0
+ 1/ 1   1/  3     0 -     9  1429 -  1438     10 Uninit
+ 1/ 1   2/  3    11 -    12  1440 -  1441      2 Uninit
+ 1/ 1   3/  3    14 -    14  1443 -  1443      1 Uninit
+debugfs: ex /c24
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    25 - 4294967295  1453         4294967271
+ 1/ 1   1/  3    25 -    25  1468 -  1468      1 Uninit
+ 1/ 1   2/  3    27 -    28  1470 -  1471      2 Uninit
+ 1/ 1   3/  3    30 -    39  1473 -  1482     10 Uninit
+debugfs: ex /c25
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    27 - 4294967295  1483         4294967269
+ 1/ 1   1/  2    27 -    28  1485 -  1486      2 Uninit
+ 1/ 1   2/  2    30 -    39  1488 -  1497     10 Uninit
+debugfs: ex /c26
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    27 - 4294967295  1484         4294967269
+ 1/ 1   1/  2    27 -    28  1498 -  1499      2 Uninit
+ 1/ 1   2/  2    30 -    39  1501 -  1510     10 Uninit
+debugfs: ex /c27
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    28 - 4294967295  1487         4294967268
+ 1/ 1   1/  2    28 -    28  1512 -  1512      1 Uninit
+ 1/ 1   2/  2    30 -    39  1514 -  1523     10 Uninit
+debugfs: ex /c28
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    30 - 4294967295  1500         4294967266
+ 1/ 1   1/  1    30 -    39  1526 -  1535     10 Uninit
+debugfs: ex /c29
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    30 - 4294967295  1511         4294967266
+ 1/ 1   1/  1    30 -    39  1537 -  1546     10 Uninit
+debugfs: ex /c30
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    31 - 4294967295  1513         4294967265
+ 1/ 1   1/  1    31 -    39  1549 -  1557      9 Uninit
+debugfs: ex /c31
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1    32 - 4294967295  1524         4294967264
+ 1/ 1   1/  1    32 -    39  1560 -  1567      8 Uninit
+debugfs: ex /d
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1525              0
+ 1/ 1   1/  3     0 -     0  1442 -  1442      1 Uninit
+ 1/ 1   2/  3     1 -     3  1444 -  1446      3 Uninit
+ 1/ 1   3/  3    36 -    39  1573 -  1576      4 Uninit
+debugfs: ex /e
+Level Entries       Logical      Physical Length Flags
+ 0/ 1   1/  1     0 - 4294967295  1547              0
+ 1/ 1   1/ 11     0 -     5  1447 -  1452      6 Uninit
+ 1/ 1   2/ 11     6 -     9  1454 -  1457      4 Uninit
+ 1/ 1   3/ 11    11 -    12  1459 -  1460      2 Uninit
+ 1/ 1   4/ 11    14 -    18  1462 -  1466      5 Uninit
+ 1/ 1   5/ 11    21 -    21  1472 -  1472      1 Uninit
+ 1/ 1   6/ 11    22 -    22  1536 -  1536      1 Uninit
+ 1/ 1   7/ 11    23 -    23  1548 -  1548      1 Uninit
+ 1/ 1   8/ 11    24 -    25  1558 -  1559      2 Uninit
+ 1/ 1   9/ 11    27 -    28  1569 -  1570      2 Uninit
+ 1/ 1  10/ 11    30 -    30  1572 -  1572      1 Uninit
+ 1/ 1  11/ 11    31 -    39  1577 -  1585      9 Uninit
+debugfs: ex /f
+Level Entries       Logical      Physical Length Flags
+ 0/ 0   1/  2     0 -     0  9000 -  9000      1 Uninit
+ 0/ 0   2/  2  8999 -  8999 17999 - 17999      1 Uninit
+Pass 1: Checking inodes, blocks, and sizes
+Pass 2: Checking directory structure
+Pass 3: Checking directory connectivity
+Pass 4: Checking reference counts
+Pass 5: Checking group summary information
+Free blocks count wrong for group #1 (7934, counted=7933).
+Fix? yes
+
+Free blocks count wrong (62923, counted=62922).
+Fix? yes
+
+
+test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
+test_filesys: 32/4096 files (43.8% non-contiguous), 2614/65536 blocks
+Exit status is 1

tests/d_punch/name new file +1 -0

new file mode 100644
index 0000000..724639f
@@ -0,0 +1 @@
+punch sparse files and big files

tests/d_punch/script new file +129 -0

129 lines not on this page · raw message

tests/d_punch_bigalloc/expect new file +142 -0

142 lines not on this page · raw message

tests/d_punch_bigalloc/name new file +1 -0

new file mode 100644
index 0000000..6d61ebe
@@ -0,0 +1 @@
+punch sparse files and big files with bigalloc

tests/d_punch_bigalloc/script new file +130 -0

130 lines not on this page · raw message

Darrick J. Wong

author

2014-10-25 21:00 UTC #48 lore raw

PATCH 37/39 fuse2fs: translate ACL structures

Translate "native" ACL structures into ext4 ACL structures when reading or writing the ACL EAs.

---
 configure       |    5 +
 configure.in    |    8 +-
 lib/config.h.in |    3 +
 misc/fuse2fs.c  |  263 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 4 files changed, 271 insertions(+), 8 deletions(-)

configure +4 -1

index 31efdc9..364cf5e 100755
@@ -12418,7 +12418,7 @@ fi
 done
 
 fi
-for ac_header in  	dirent.h 	errno.h 	execinfo.h 	getopt.h 	malloc.h 	mntent.h 	paths.h 	semaphore.h 	setjmp.h 	signal.h 	stdarg.h 	stdint.h 	stdlib.h 	termios.h 	termio.h 	unistd.h 	utime.h 	attr/xattr.h 	linux/falloc.h 	linux/fd.h 	linux/major.h 	linux/loop.h 	net/if_dl.h 	netinet/in.h 	sys/disklabel.h 	sys/disk.h 	sys/file.h 	sys/ioctl.h 	sys/mkdev.h 	sys/mman.h 	sys/mount.h 	sys/prctl.h 	sys/resource.h 	sys/select.h 	sys/socket.h 	sys/sockio.h 	sys/stat.h 	sys/syscall.h 	sys/sysctl.h 	sys/sysmacros.h 	sys/time.h 	sys/types.h 	sys/un.h 	sys/wait.h
+for ac_header in  	dirent.h 	errno.h 	execinfo.h 	getopt.h 	malloc.h 	mntent.h 	paths.h 	semaphore.h 	setjmp.h 	signal.h 	stdarg.h 	stdint.h 	stdlib.h 	termios.h 	termio.h 	unistd.h 	utime.h 	attr/xattr.h 	linux/falloc.h 	linux/fd.h 	linux/major.h 	linux/loop.h 	net/if_dl.h 	netinet/in.h 	sys/acl.h 	sys/disklabel.h 	sys/disk.h 	sys/file.h 	sys/ioctl.h 	sys/mkdev.h 	sys/mman.h 	sys/mount.h 	sys/prctl.h 	sys/resource.h 	sys/select.h 	sys/socket.h 	sys/sockio.h 	sys/stat.h 	sys/syscall.h 	sys/sysctl.h 	sys/sysmacros.h 	sys/time.h 	sys/types.h 	sys/un.h 	sys/wait.h
 do :
   as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh`
 ac_fn_c_check_header_mongrel "$LINENO" "$ac_header" "$as_ac_Header" "$ac_includes_default"

@@ -13209,6 +13209,7 @@ else
 do :
   as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh`
 ac_fn_c_check_header_compile "$LINENO" "$ac_header" "$as_ac_Header" "#define _FILE_OFFSET_BITS	64
+#define FUSE_USE_VERSION 29
 "
 if eval test \"x\$"$as_ac_Header"\" = x"yes"; then :
   cat >>confdefs.h <<_ACEOF

@@ -13227,6 +13228,7 @@ done
 
 	cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 /* end confdefs.h.  */
+#define FUSE_USE_VERSION 29
 #ifdef __linux__
 #include <linux/fs.h>
 #include <linux/falloc.h>

@@ -13346,6 +13348,7 @@ else
 do :
   as_ac_Header=`$as_echo "ac_cv_header_$ac_header" | $as_tr_sh`
 ac_fn_c_check_header_compile "$LINENO" "$ac_header" "$as_ac_Header" "#define _FILE_OFFSET_BITS	64
+#define FUSE_USE_VERSION 29
 #ifdef __linux__
 # include <linux/fs.h>
 # include <linux/falloc.h>

configure.in +6 -2

index eb4b6ad..9cb6aaf 100644
@@ -935,6 +935,7 @@ AC_CHECK_HEADERS(m4_flatten([
 	linux/loop.h
 	net/if_dl.h
 	netinet/in.h
+	sys/acl.h
 	sys/disklabel.h
 	sys/disk.h
 	sys/file.h

@@ -1177,10 +1178,12 @@ then
 else
 	AC_CHECK_HEADERS([pthread.h fuse.h], [],
 [AC_MSG_FAILURE([Cannot find fuse2fs headers.])],
-[#define _FILE_OFFSET_BITS	64])
+[#define _FILE_OFFSET_BITS	64
+#define FUSE_USE_VERSION 29])
 
 	AC_PREPROC_IFELSE(
-[AC_LANG_PROGRAM([[#ifdef __linux__
+[AC_LANG_PROGRAM([[#define FUSE_USE_VERSION 29
+#ifdef __linux__
 #include <linux/fs.h>
 #include <linux/falloc.h>
 #include <linux/xattr.h>

@@ -1195,6 +1198,7 @@ fi
 ,
 AC_CHECK_HEADERS([pthread.h fuse.h], [], [FUSE_CMT="#"],
 [#define _FILE_OFFSET_BITS	64
+#define FUSE_USE_VERSION 29
 #ifdef __linux__
 # include <linux/fs.h>
 # include <linux/falloc.h>

lib/config.h.in +3 -0

index b10e91d..7005940 100644
@@ -467,6 +467,9 @@
 /* Define to 1 if you have the `sysconf' function. */
 #undef HAVE_SYSCONF
 
+/* Define to 1 if you have the <sys/acl.h> header file. */
+#undef HAVE_SYS_ACL_H
+
 /* Define to 1 if you have the <sys/disklabel.h> header file. */
 #undef HAVE_SYS_DISKLABEL_H

misc/fuse2fs.c +258 -5

324 lines not on this page · raw message

Darrick J. Wong

author

2014-10-25 21:00 UTC #49 lore raw

PATCH 38/39 fuse2fs: handle 64-bit dates correctly

Fix fuse2fs' interpretation of 64-bit date quantities to match the kernel.

---
 misc/fuse2fs.c |   31 ++++++++++++++++++++++---------
 1 file changed, 22 insertions(+), 9 deletions(-)

misc/fuse2fs.c +22 -9

index ec0b743..2746a27 100644
@@ -353,15 +353,24 @@ static int __translate_error(ext2_filsys fs, errcode_t err, ext2_ino_t ino,
 
 static inline __u32 ext4_encode_extra_time(const struct timespec *time)
 {
-	return (sizeof(time->tv_sec) > 4 ?
-		(time->tv_sec >> 32) & EXT4_EPOCH_MASK : 0) |
-	       ((time->tv_nsec << EXT4_EPOCH_BITS) & EXT4_NSEC_MASK);
+	__u32 extra = sizeof(time->tv_sec) > 4 ?
+			((time->tv_sec - (__s32)time->tv_sec) >> 32) &
+			EXT4_EPOCH_MASK : 0;
+	return extra | (time->tv_nsec << EXT4_EPOCH_BITS);
 }
 
 static inline void ext4_decode_extra_time(struct timespec *time, __u32 extra)
 {
-	if (sizeof(time->tv_sec) > 4)
-		time->tv_sec |= (__u64)((extra) & EXT4_EPOCH_MASK) << 32;
+	if (sizeof(time->tv_sec) > 4 && (extra & EXT4_EPOCH_MASK)) {
+		__u64 extra_bits = extra & EXT4_EPOCH_MASK;
+		/*
+		 * Prior to kernel 3.14?, we had a broken decode function,
+		 * wherein we effectively did this:
+		 * if (extra_bits == 3)
+		 *     extra_bits = 0;
+		 */
+		time->tv_sec += extra_bits << 32;
+	}
 	time->tv_nsec = ((extra) & EXT4_NSEC_MASK) >> EXT4_EPOCH_BITS;
 }

@@ -387,7 +396,7 @@ do {									       \
 	(timespec)->tv_sec = (signed)((raw_inode)->xtime);		       \
 	if (EXT4_FITS_IN_INODE(raw_inode, xtime ## _extra))		       \
 		ext4_decode_extra_time((timespec),			       \
-				       raw_inode->xtime ## _extra);	       \
+				       (raw_inode)->xtime ## _extra);	       \
 	else								       \
 		(timespec)->tv_nsec = 0;				       \
 } while (0)

@@ -749,6 +758,7 @@ static int stat_inode(ext2_filsys fs, ext2_ino_t ino, struct stat *statbuf)
 	dev_t fakedev = 0;
 	errcode_t err;
 	int ret = 0;
+	struct timespec tv;
 
 	memset(&inode, 0, sizeof(inode));
 	err = ext2fs_read_inode_full(fs, ino, (struct ext2_inode *)&inode,

@@ -766,9 +776,12 @@ static int stat_inode(ext2_filsys fs, ext2_ino_t ino, struct stat *statbuf)
 	statbuf->st_size = EXT2_I_SIZE(&inode);
 	statbuf->st_blksize = fs->blocksize;
 	statbuf->st_blocks = blocks_from_inode(fs, &inode);
-	statbuf->st_atime = inode.i_atime;
-	statbuf->st_mtime = inode.i_mtime;
-	statbuf->st_ctime = inode.i_ctime;
+	EXT4_INODE_GET_XTIME(i_atime, &tv, &inode);
+	statbuf->st_atime = tv.tv_sec;
+	EXT4_INODE_GET_XTIME(i_mtime, &tv, &inode);
+	statbuf->st_mtime = tv.tv_sec;
+	EXT4_INODE_GET_XTIME(i_ctime, &tv, &inode);
+	statbuf->st_ctime = tv.tv_sec;
 	if (LINUX_S_ISCHR(inode.i_mode) ||
 	    LINUX_S_ISBLK(inode.i_mode)) {
 		if (inode.i_block[0])

Darrick J. Wong

author

2014-10-25 21:01 UTC #50 lore raw

PATCH 39/39 fuse2fs: implement fallocate

Use the (new) ext2fs_fallocate() to fallocate file space.

---
 misc/fuse2fs.c |   58 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)

misc/fuse2fs.c +57 -1

index 2746a27..466461e 100644
@@ -3300,7 +3300,63 @@ out:
 static int fallocate_helper(struct fuse_file_info *fp, int mode, off_t offset,
 			    off_t len)
 {
-	return -EOPNOTSUPP;
+	struct fuse_context *ctxt = fuse_get_context();
+	struct fuse2fs *ff = (struct fuse2fs *)ctxt->private_data;
+	struct fuse2fs_file_handle *fh = (struct fuse2fs_file_handle *)fp->fh;
+	ext2_filsys fs;
+	struct ext2_inode_large inode;
+	blk64_t start, end;
+	__u64 fsize;
+	errcode_t err;
+	int flags;
+
+	FUSE2FS_CHECK_CONTEXT(ff);
+	fs = ff->fs;
+	FUSE2FS_CHECK_MAGIC(fs, fh, FUSE2FS_FILE_MAGIC);
+	start = offset / fs->blocksize;
+	end = (offset + len - 1) / fs->blocksize;
+	dbg_printf("%s: ino=%d mode=0x%x start=%jd end=%llu\n", __func__,
+		   fh->ino, mode, offset / fs->blocksize, end);
+	if (!fs_can_allocate(ff, len / fs->blocksize))
+		return -ENOSPC;
+
+	memset(&inode, 0, sizeof(inode));
+	err = ext2fs_read_inode_full(fs, fh->ino, (struct ext2_inode *)&inode,
+				     sizeof(inode));
+	if (err)
+		return err;
+	fsize = EXT2_I_SIZE(&inode);
+
+	/* Allocate a bunch of blocks */
+	flags = (mode & FL_KEEP_SIZE_FLAG ? 0 :
+			EXT2_FALLOCATE_INIT_BEYOND_EOF);
+	err = ext2fs_fallocate(fs, flags, fh->ino,
+			       (struct ext2_inode *)&inode,
+			       ~0ULL, start, end - start + 1);
+	if (err && err != EXT2_ET_BLOCK_ALLOC_FAIL)
+		return translate_error(fs, fh->ino, err);
+
+	/* Update i_size */
+	if (!(mode & FL_KEEP_SIZE_FLAG)) {
+		if (offset + len > fsize) {
+			err = ext2fs_inode_size_set(fs,
+						(struct ext2_inode *)&inode,
+						offset + len);
+			if (err)
+				return translate_error(fs, fh->ino, err);
+		}
+	}
+
+	err = update_mtime(fs, fh->ino, &inode);
+	if (err)
+		return err;
+
+	err = ext2fs_write_inode_full(fs, fh->ino, (struct ext2_inode *)&inode,
+				      sizeof(inode));
+	if (err)
+		return translate_error(fs, fh->ino, err);
+
+	return err;
 }
 
 static errcode_t clean_block_middle(ext2_filsys fs, ext2_ino_t ino,

Darrick J. Wong

author

2014-10-27 23:31 UTC #51 lore raw

[PATCH 40/39] e2fsck: fix reporting of unknown htree block inode number

Sami Liedes reports that e2fsck fails to report the correct directory inode number during a pass2 check for unexpected HTREE blocks. Provide the inode number in the problem report.

---
 e2fsck/pass2.c |    1 +
 1 file changed, 1 insertion(+)

e2fsck/pass2.c +1 -0

index fa17f20..f645229 100644
@@ -1006,6 +1006,7 @@ inline_read_fail:
 	dx_dir = e2fsck_get_dx_dir_info(ctx, ino);
 	if (dx_dir && dx_dir->numblocks) {
 		if (db->blockcnt >= dx_dir->numblocks) {
+			pctx.dir = ino;
 			if (fix_problem(ctx, PR_2_UNEXPECTED_HTREE_BLOCK,
 					&pctx)) {
 				clear_htree(ctx, ino);

Theodore Ts'o

Applied

2014-11-05 16:11 UTC #52 lore raw

Re: [PATCH 40/39] e2fsck: fix reporting of unknown htree block inode number

Quoted 6 lines · Darrick J. Wong

Sami Liedes reports that e2fsck fails to report the correct directory inode number during a pass2 check for unexpected HTREE blocks. Provide the inode number in the problem report.
Signed-off-by Darrick J. Wong <[email protected]>
Reported-by Sami Liedes <[email protected]>

Applied, thanks.

- Ted

Darrick J. Wong

author

2014-10-27 23:32 UTC #53 lore raw

[PATCH 41/39] mke2fs: warn if enabling metadata_csum on a pre-3.18 kernel

The metadata_csum feature (really, the journal checksum disk format) didn't stabilize until the 3.18 kernel, at which point the companion journal_csum feature was turned on by default if metadata_csum was enabled. Therefore, warn the user if they try to create such a filesystem on a pre-3.18 kernel.