Skip to contentSkip to conversation

e2fsprogs: Add ext4migrate

RFC In review Aneesh Kumar K.V 3 participants 13 messages May 4, 2007 – May 7, 2007 1 message not in the archive Kbuild

lore
Share & tools
lore
#0 lore raw

This is work in progress

Changes from from my previous patches are as follows:
a) COW unix io manager using libtdb
b) add ext4replay to undo the file system changes done during migration

Changes from from my previous patches are as follows:
a) Mark the file system unclean if we fail migrating
b) support for migrating more than one file
c) Support for migrating all the ext3 inode in the file system
d) Fix the free block counts. ( Make sure fsck pass without errors after migration )
e) Mark the blocks used for extent idx as used so that we don't double allocate them
f) Don't mark the blocks used for indirect block map as free untill we fully migrate the inode.This make sure we don't use these blocks for extent idx and overwrite them.
g) Fix the migration when we end up with more than one extent idx block.

Changes from from my previous patches are as follows:
a) support for files with holes
b) use the block iterator present in libext2fs
c) don't mark the indirect blocks as unused early. If we mark the the blocks unused, they can be resued for extent index. If we later fail to migrate, the indirect blocks content would be overwritten.

ext4migrate command takes the below syntax ext4migrate --display | --migrate <image_name> [<filename>]

The --display option helps in displaying the block map details for an ext3 inode and extent map details for an ext4 inode. The --migrate option convert a block mapped ext3 inode to extent mapped ext4 inode.

This needs to be run on an unmounted file system (offline migration).

The extent insert code is derived out of the latest ext4 kernel source. I have tried to keep the code as close as possible to the kernel sources. This makes sure that any fixes for the tree building code in kernel should be easily applied to ext4migrate. The ext3_ext naming convention instead of ext4_ext found in kernel is to make sure we are in sync with rest of e2fsprogs source.

The inode modification is done only at the last stage. This is to make sure that if we fail at any intermediate stage, we exit without touching the disk.

The inode update is done as below
a) Walk the extent index blocks and write them to the disk. If failed exit
b) Writ the update block bitmap. if failed exit.
b) Write the inode. if failed Undo the write of block bitmap and exit.

The patch applies on top of e2fsprogs found at

http://www.kernel.org/pub/linux/kernel/people/tytso/e2fsprogs-interim/e2fsprogs-1.39-tyt1/e2fsprogs-1.39-tyt1.tar.bz2

Right now i am building ext4migrate and ext4replay by linking against libtd. This will not be needed once I rebase the patches against the latest e2fsprogs which has the tdb code available as a part libext2fs

-aneesh

#1 lore raw

PATCH 1/4 Add unix COW io manager

From: Aneesh Kumar K.V <[email protected]>

---
 lib/ext2fs/Makefile.in   |    6 +
 lib/ext2fs/ext2_io.h     |    3 +
 lib/ext2fs/unix_cow_io.c |  864 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 873 insertions(+), 0 deletions(-)
 create mode 100644 lib/ext2fs/unix_cow_io.c
lib/ext2fs/Makefile.in +6 -0
index 799659e..938f2d3 100644
@@ -64,6 +64,7 @@ OBJS= $(DEBUGFS_LIB_OBJS) $(RESIZE_LIB_OBJS) $(E2IMAGE_LIB_OBJS) \
 	rw_bitmaps.o \
 	swapfs.o \
 	unix_io.o \
+	unix_cow_io.o \
 	unlink.o \
 	valid_blk.o \
 	version.o
@@ -126,6 +127,7 @@ SRCS= ext2_err.c \
 	$(srcdir)/swapfs.c \
 	$(srcdir)/test_io.c \
 	$(srcdir)/unix_io.c \
+	$(srcdir)/unix_cow_io.c \
 	$(srcdir)/unlink.c \
 	$(srcdir)/valid_blk.c \
 	$(srcdir)/version.c \
@@ -512,6 +514,10 @@ unix_io.o: $(srcdir)/unix_io.c $(srcdir)/ext2_fs.h \
  $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \
  $(srcdir)/ext2_fs.h $(top_srcdir)/lib/et/com_err.h $(srcdir)/ext2_io.h \
  $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/bitops.h
+unix_cow_io.o: $(srcdir)/unix_cow_io.c $(srcdir)/ext2_fs.h \
+ $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \
+ $(srcdir)/ext2_fs.h $(top_srcdir)/lib/et/com_err.h $(srcdir)/ext2_io.h \
+ $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/bitops.h
 unlink.o: $(srcdir)/unlink.c $(srcdir)/ext2_fs.h \
  $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \
  $(srcdir)/ext2_fs.h $(top_srcdir)/lib/et/com_err.h $(srcdir)/ext2_io.h \
lib/ext2fs/ext2_io.h +3 -0
index eada278..cdfdd51 100644
@@ -96,6 +96,9 @@ extern errcode_t io_channel_write_byte(io_channel channel,
 /* unix_io.c */
 extern io_manager unix_io_manager;
 
+/* unix_cow_io.c */
+extern io_manager unix_cow_io_manager;
+
 /* test_io.c */
 extern io_manager test_io_manager, test_io_backing_manager;
 extern void (*test_io_cb_read_blk)
lib/ext2fs/unix_cow_io.c new file +864 -0

864 lines not on this page · raw message

Signature · 1 line
1.5.2.rc0.71.g4342-dirty
#2 lore raw

PATCH 2/4 Add extent related functions

From: Aneesh Kumar K.V <[email protected]>

The code is derived out of the latest ext4 kernel source. I have tried to keep the code as close as possible to the kernel sources. This makes sure that any fixes for the tree building code in kernel should be easily applied to ext4migrate. The ext3_ext naming convention instead of ext4_ext found in kernel is to make sure we are in sync with rest of e2fsprogs source

---
 ext4migrate/extents.c |  763 +++++++++++++++++++++++++++++++++++++++++++++++++
 ext4migrate/extents.h |   10 +
 2 files changed, 773 insertions(+), 0 deletions(-)
 create mode 100644 ext4migrate/extents.c
 create mode 100644 ext4migrate/extents.h
ext4migrate/extents.c new file +763 -0
new file mode 100644
index 0000000..d25ce36
@@ -0,0 +1,763 @@
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "migrate.h"
+#include "extents.h"
+
+struct ext4_ext_path {
+	ext4_fsblk_t			p_block;
+	__u16				p_depth;
+	struct ext3_extent		*p_ext;
+	struct ext3_extent_idx		*p_idx;
+	struct ext3_extent_header	*p_hdr;
+};
+
+/*
+ * ext_pblock:
+ * combine low and high parts of physical block number into ext4_fsblk_t
+ */
+ext4_fsblk_t ext_pblock(struct ext3_extent *ex)
+{
+	ext4_fsblk_t block;
+
+	block = ex->ee_start;
+	block |= ((ext4_fsblk_t) ex->ee_start_hi << 31) << 1;
+	return block;
+}
+
+/*
+ * idx_pblock:
+ * combine low and high parts of a leaf physical block number into ext4_fsblk_t
+ */
+ext4_fsblk_t idx_pblock(struct ext3_extent_idx *ix)
+{
+	ext4_fsblk_t block;
+
+	block = ix->ei_leaf;
+	block |= ((ext4_fsblk_t) ix->ei_leaf_hi << 31) << 1;
+	return block;
+}
+
+/*
+ * ext3_ext_store_pblock:
+ * stores a large physical block number into an extent struct,
+ * breaking it into parts
+ */
+void ext3_ext_store_pblock(struct ext3_extent *ex, ext4_fsblk_t pb)
+{
+	ex->ee_start = (unsigned long) (pb & 0xffffffff);
+	ex->ee_start_hi = (unsigned long) ((pb >> 31) >> 1) & 0xffff;
+}
+
+/*
+ * ext3_idx_store_pblock:
+ * stores a large physical block number into an index struct,
+ * breaking it into parts
+ */
+static void ext3_idx_store_pblock(struct ext3_extent_idx *ix, ext4_fsblk_t pb)
+{
+	ix->ei_leaf = (unsigned long) (pb & 0xffffffff);
+	ix->ei_leaf_hi = (unsigned long) ((pb >> 31) >> 1) & 0xffff;
+}
+
+
+static int __ext3_ext_space_block(ext2_filsys filesys)
+{
+	int size;
+
+	size = (filesys->blocksize - sizeof(struct ext3_extent_header))
+			/ sizeof(struct ext3_extent);
+	return size;
+}
+static int ext3_ext_space_block()
+{
+	return  __ext3_ext_space_block(current_fs);
+}
+
+static int __ext3_ext_space_block_idx(ext2_filsys filesys)
+{
+	int size;
+
+	size = (filesys->blocksize - sizeof(struct ext3_extent_header))
+			/ sizeof(struct ext3_extent_idx);
+	return size;
+}
+static int ext3_ext_space_block_idx()
+{
+	return __ext3_ext_space_block_idx(current_fs);
+
+}
+
+int ext3_ext_space_root(void)
+{
+	int size;
+
+	size = EXT2_N_BLOCKS*sizeof(blk_t);
+	size -= sizeof(struct ext3_extent_header);
+	size /= sizeof(struct ext3_extent);
+	return size;
+}
+
+static int ext3_ext_space_root_idx(void)
+{
+	int size;
+
+	size = EXT2_N_BLOCKS*sizeof(blk_t);
+	size -= sizeof(struct ext3_extent_header);
+	size /= sizeof(struct ext3_extent_idx);
+	return size;
+}
+static void ext3_ext_binsearch_idx(struct ext4_ext_path *path,
+						blk_t logical_blk)
+{
+	struct ext3_extent_header *eh = path->p_hdr;
+	struct ext3_extent_idx *r, *l, *m;
+
+	l = EXT_FIRST_INDEX(eh) + 1;
+	r = EXT_FIRST_INDEX(eh) + eh->eh_entries - 1;
+	while (l <= r) {
+		m = l + (r - l) / 2;
+		if (logical_blk < m->ei_block)
+			r = m - 1;
+		else
+			l = m + 1;
+	}
+	path->p_idx = l - 1;
+}
+
+static void ext3_ext_binsearch(struct ext4_ext_path *path, blk_t logical_blk)
+{
+	struct ext3_extent_header *eh = path->p_hdr;
+	struct ext3_extent *r, *l, *m;
+
+	if (eh->eh_entries == 0) {
+		/*
+		 * this leaf is empty:
+		 * we get such a leaf in split/add case
+		 */
+		return;
+	}
+
+	l = EXT_FIRST_EXTENT(eh) + 1;
+	r = EXT_FIRST_EXTENT(eh) + eh->eh_entries - 1;
+
+	while (l <= r) {
+		m = l + (r - l) / 2;
+		if (logical_blk < m->ee_block)
+			r = m - 1;
+		else
+			l = m + 1;
+	}
+
+	path->p_ext = l - 1;
+}
+
+/*
+ * ext3_ext_insert_index:
+ * insert new index [@logical;@ptr] into the block at @curp;
+ * check where to insert: before @curp or after @curp
+ */
+static int ext3_ext_insert_index(struct ext4_ext_path *curp,
+				int logical, ext4_fsblk_t ptr)
+{
+	struct ext3_extent_idx *ix;
+	int len, err = 0;
+
+	len = EXT_MAX_INDEX(curp->p_hdr) - curp->p_idx;
+	if (logical > curp->p_idx->ei_block) {
+		/* insert after */
+		if (curp->p_idx != EXT_LAST_INDEX(curp->p_hdr)) {
+			len = (len - 1) * sizeof(struct ext3_extent_idx);
+			len = len < 0 ? 0 : len;
+			memmove(curp->p_idx + 2, curp->p_idx + 1, len);
+		}
+		ix = curp->p_idx + 1;
+	} else {
+		/* insert before */
+		len = len * sizeof(struct ext3_extent_idx);
+		len = len < 0 ? 0 : len;
+		memmove(curp->p_idx + 1, curp->p_idx, len);
+		ix = curp->p_idx;
+	}
+
+	ix->ei_block = logical;
+	ext3_idx_store_pblock(ix, ptr);
+	curp->p_hdr->eh_entries = curp->p_hdr->eh_entries+1;
+
+	return err;
+}
+/*
+ * ext3_ext_split:
+ * inserts new subtree into the path, using free index entry
+ * at depth @at:
+ * - allocates all needed blocks (new leaf and all intermediate index blocks)
+ * - makes decision where to split
+ * - moves remaining extents and index entries (right to the split point)
+ *   into the newly allocated blocks
+ * - initializes subtree
+ */
+static int ext3_ext_split(struct ext3_extent_header *eh,
+				struct ext4_ext_path *path,
+				struct ext3_extent *newext, int at)
+{
+	int depth = eh->eh_depth;
+	struct ext3_extent_header *neh;
+	struct ext3_extent_idx *fidx;
+	struct ext3_extent *ex;
+	struct blk_cache *blkc;
+	int i = at, k, m, a;
+	ext4_fsblk_t newblock, oldblock;
+	int border;
+	ext4_fsblk_t *ablocks = NULL; /* array of allocated blocks */
+	int err = 0;
+
+	/* make decision: where to split? */
+	/* FIXME: now decision is simplest: at current extent */
+
+	/* if current leaf will be split, then we should use
+	 * border from split point */
+	if (path[depth].p_ext != EXT_MAX_EXTENT(path[depth].p_hdr)) {
+		border = path[depth].p_ext[1].ee_block;
+	} else {
+		border = newext->ee_block;
+	}
+
+	/*
+	 * If error occurs, then we break processing
+	 * and mark filesystem read-only. index won't
+	 * be inserted and tree will be in consistent
+	 * state. Next mount will repair buffers too.
+	 */
+
+	/*
+	 * Get array to track all allocated blocks.
+	 * We need this to handle errors and free blocks
+	 * upon them.
+	 */
+	ablocks = malloc(sizeof(ext4_fsblk_t) * depth);
+	if (!ablocks)
+		return -1;
+
+	memset(ablocks, 0, sizeof(ext4_fsblk_t) * depth);
+
+	/* allocate all needed blocks */
+	for (a = 0; a < depth - at; a++) {
+		blkc = get_block(0);
+		if (!blkc) {
+			err =  EXT2_ET_BLOCK_ALLOC_FAIL;
+			goto cleanup;
+		}
+		newblock = blkc->pblock;
+		if (newblock == 0)
+			goto cleanup;
+		ablocks[a] = newblock;
+	}
+
+	/* initialize new leaf */
+	newblock = ablocks[--a];
+
+	blkc = get_block(newblock);
+	if (!blkc) {
+		err =  EXT2_ET_BLOCK_ALLOC_FAIL;
+		goto cleanup;
+	}
+
+	neh = (struct ext3_extent_header *)blkc->block_mem_addr;
+	neh->eh_entries = 0;
+	neh->eh_max = ext3_ext_space_block();
+	neh->eh_magic = EXT4_EXT_MAGIC;
+	neh->eh_depth = 0;
+	ex = EXT_FIRST_EXTENT(neh);
+
+	/* move remainder of path[depth] to the new leaf */
+	/* start copy from next extent */
+	/* TODO: we could do it by single memmove */
+	m = 0;
+	path[depth].p_ext++;
+	while (path[depth].p_ext <=
+			EXT_MAX_EXTENT(path[depth].p_hdr)) {
+		/*memmove(ex++, path[depth].p_ext++,
+				sizeof(struct ext4_extent));
+		neh->eh_entries++;*/
+		path[depth].p_ext++;
+		m++;
+	}
+
+	/*
+	 * move the part of the old extent to the new allocated one
+	 */
+	if (m) {
+		memmove(ex, path[depth].p_ext-m, sizeof(struct ext3_extent)*m);
+		neh->eh_entries = neh->eh_entries+m;
+	}
+
+	/* correct old leaf */
+	if (m) {
+		path[depth].p_hdr->eh_entries = path[depth].p_hdr->eh_entries-m;
+
+	}
+
+	/* create intermediate indexes */
+	k = depth - at - 1;
+
+	/*create intermediate indices */
+	/* insert new index into current index block */
+	/* current depth stored in i var */
+	i = depth - 1;
+	while (k--) {
+		oldblock = newblock;
+		newblock = ablocks[--a];
+		blkc = get_block(newblock);
+		if (!blkc) {
+			err =  EXT2_ET_BLOCK_ALLOC_FAIL;
+			goto cleanup;
+		}
+
+		neh = (struct ext3_extent_header *)blkc->block_mem_addr;
+
+		neh->eh_entries = 1;
+		neh->eh_magic = EXT4_EXT_MAGIC;
+		neh->eh_max = ext3_ext_space_block_idx();
+		neh->eh_depth = depth - i;
+		fidx = EXT_FIRST_INDEX(neh);
+		fidx->ei_block = border;
+		ext3_idx_store_pblock(fidx, oldblock);
+
+		/* copy indexes */
+		m = 0;
+		path[i].p_idx++;
+
+		while (path[i].p_idx <= EXT_MAX_INDEX(path[i].p_hdr)) {
+			/*memmove(++fidx, path[i].p_idx++,
+					sizeof(struct ext4_extent_idx));
+			neh->eh_entries++;
+			BUG_ON(neh->eh_entries > neh->eh_max);*/
+			path[i].p_idx++;
+			m++;
+		}
+		if (m) {
+			memmove(++fidx, path[i].p_idx - m,
+				sizeof(struct ext3_extent_idx) * m);
+			neh->eh_entries = neh->eh_entries + m;
+		}
+
+
+		/* correct old index */
+		if (m) {
+			path[i].p_hdr->eh_entries = path[i].p_hdr->eh_entries-m;
+		}
+
+		i--;
+	}
+
+	/* insert new index */
+	err = ext3_ext_insert_index(path + at, border, newblock);
+
+cleanup:
+
+	free(ablocks);
+
+	return err;
+}
+
+/*
+ * ext3_ext_grow_indepth:
+ * implements tree growing procedure:
+ * - allocates new block
+ * - moves top-level data (index block or leaf) into the new block
+ * - initializes new top-level, creating index that points to the
+ *   just created block
+ */
+static int ext3_ext_grow_indepth(struct ext3_extent_header *eh,
+					struct ext4_ext_path *path)
+{
+	struct ext4_ext_path *curp = path;
+	struct ext3_extent_header *neh;
+	struct blk_cache *blkc;
+	ext4_fsblk_t newblock;
+	void *addr;
+
+	blkc = get_block(0);
+	if (!blkc) {
+		return  EXT2_ET_BLOCK_ALLOC_FAIL;
+
+	}
+	addr = blkc->block_mem_addr;
+	newblock = blkc->pblock;
+
+	/* move top-level index/leaf into new block */
+	memmove(addr, curp->p_hdr, EXT2_N_BLOCKS*sizeof(blk_t));
+
+	/* set size of new block */
+	neh =  (struct ext3_extent_header *)addr;
+	/* old root could have indexes or leaves
+	 * so calculate e_max right way */
+	if (eh->eh_depth) {
+		neh->eh_max = ext3_ext_space_block_idx();
+	} else {
+		neh->eh_max = ext3_ext_space_block();
+	}
+
+	neh->eh_magic = EXT4_EXT_MAGIC;
+
+	/* create index in new top-level index: num,max,pointer */
+
+	curp->p_hdr->eh_magic = EXT4_EXT_MAGIC;
+	curp->p_hdr->eh_max = ext3_ext_space_root_idx();
+	curp->p_hdr->eh_entries = 1;
+	curp->p_idx = EXT_FIRST_INDEX(curp->p_hdr);
+	/* FIXME: it works, but actually path[0] can be index */
+	curp->p_idx->ei_block = EXT_FIRST_EXTENT(path[0].p_hdr)->ee_block;
+	ext3_idx_store_pblock(curp->p_idx, newblock);
+
+	neh = eh;
+	//fidx = EXT_FIRST_INDEX(neh);
+
+	neh->eh_depth = path->p_depth + 1;
+
+	return 0;
+}
+struct ext4_ext_path * ext3_ext_find_extent(struct ext3_extent_header *eh,
+						blk_t logical_blk,
+						struct ext4_ext_path *path)
+{
+
+	struct blk_cache *blkc;
+	short int depth, i, ppos = 0;
+	int alloc = 0;
+
+	i = depth =  eh->eh_depth;
+
+	if (!path) {
+		/*
+		 * We may be called to rebuild the path while growing the
+		 * depth of the tree
+		 */
+		path = malloc(sizeof(struct ext4_ext_path)* (depth+2));
+		if (!path)
+			return NULL;
+
+		alloc = 1;
+		memset(path, 0, sizeof(struct ext4_ext_path)* (depth+2));
+	}
+
+
+	path[0].p_hdr = eh;
+
+	while (i) {
+
+		ext3_ext_binsearch_idx(path+ppos, logical_blk);
+
+		path[ppos].p_block = idx_pblock(path[ppos].p_idx);
+		path[ppos].p_depth = i;
+		path[ppos].p_ext = NULL;
+
+		blkc = get_block(path[ppos].p_block);
+
+		if (!blkc) {
+			if (alloc)
+				free(path);
+			return NULL;
+		}
+
+		eh = blkc->block_mem_addr;
+		if (!eh) {
+			if (alloc)
+				free(path);
+			return NULL;
+		}
+		if (eh->eh_max == 0) {
+			if (alloc)
+				free(path);
+			return NULL;
+		}
+		ppos++;
+		path[ppos].p_hdr = eh;
+		i--;
+	}
+	path[ppos].p_depth = i;
+	path[ppos].p_hdr = eh;
+	path[ppos].p_ext = NULL;
+	path[ppos].p_idx = NULL;
+
+	ext3_ext_binsearch(path + ppos, logical_blk);
+
+	return path;
+}
+
+static int ext3_ext_create_new_leaf(struct ext3_extent_header *eh,
+					struct ext4_ext_path *path,
+					struct ext3_extent *newext)
+{
+
+	struct ext4_ext_path *curp;
+	int depth, i, err = 0;
+
+repeat:
+	i = depth = eh->eh_depth;
+
+	/* walk up to the tree and look for free index entry */
+	curp = path + depth;
+	while (i > 0 && !EXT_HAS_FREE_INDEX(curp)) {
+		i--;
+		curp--;
+	}
+
+	if (EXT_HAS_FREE_INDEX(curp)) {
+
+		/* if we found index with free entry, then use that
+		 * entry: create all needed subtree and add new leaf */
+		err = ext3_ext_split(eh, path, newext, i);
+		if (err)
+			goto err_out;
+
+		/*
+		 * Recalculate the path since grow inode would
+		 * have changed it
+		 */
+		path = ext3_ext_find_extent(eh, newext->ee_block, path);
+		if (!path) {
+			err = EXT2_ET_BLOCK_ALLOC_FAIL;
+			goto err_out;
+		}
+
+	} else {
+
+		/* tree is full, time to grow in depth */
+		err = ext3_ext_grow_indepth(eh, path);
+		if (err)
+			goto err_out;
+
+		/*
+		 * recalculate the path since grow inode
+		 * would have changed it i
+		 */
+		path = ext3_ext_find_extent(eh, newext->ee_block, path);
+		if (!path) {
+			err = EXT2_ET_BLOCK_ALLOC_FAIL;
+			goto err_out;
+		}
+
+
+		/*
+		 * only first (depth 0 -> 1) produces free space;
+		 * in all other cases we have to split the grown tree
+		 */
+		depth = eh->eh_depth;
+		if (path[depth].p_hdr->eh_entries == path[depth].p_hdr->eh_max) {
+			/* now we need to split */
+			goto repeat;
+		}
+
+	}
+
+err_out:
+
+	return err;
+}
+
+static int ext3_can_extents_be_merged(struct ext3_extent *ex1,
+					struct ext3_extent *ex2)
+{
+	if (ex1->ee_block + ex1->ee_len != ex2->ee_block)
+		return 0;
+
+
+	if (ext_pblock(ex1) + ex1->ee_len == ext_pblock(ex2))
+		return 1;
+	return 0;
+}
+
+
+/*
+ * ext3_ext_next_leaf_block:
+ * returns first allocated block from next leaf or EXT_MAX_BLOCK
+ */
+static unsigned ext3_ext_next_leaf_block(struct ext4_ext_path *path)
+{
+	int depth;
+
+	depth = path->p_depth;
+
+	/* zero-tree has no leaf blocks at all */
+	if (depth == 0)
+		return EXT_MAX_BLOCK;
+
+	/* go to index block */
+	depth--;
+
+	while (depth >= 0) {
+		if (path[depth].p_idx != EXT_LAST_INDEX(path[depth].p_hdr)) {
+			return path[depth].p_idx[1].ei_block;
+		}
+		depth--;
+	}
+
+	return EXT_MAX_BLOCK;
+}
+
+/*
+ * ext3_ext_correct_indexes:
+ * if leaf gets modified and modified extent is first in the leaf,
+ * then we have to correct all indexes above.
+ * TODO: do we need to correct tree in all cases?
+ */
+static void ext3_ext_correct_indexes(struct ext3_extent_header *eh,
+					struct ext4_ext_path *path)
+{
+	int depth = eh->eh_depth;
+	struct ext3_extent *ex;
+	int border;
+	int k;
+
+
+	eh = path[depth].p_hdr;
+	ex = path[depth].p_ext;
+
+	if (depth == 0) {
+		/* there is no tree at all */
+		return;
+	}
+
+	if (ex != EXT_FIRST_EXTENT(eh)) {
+		/* we correct tree if first leaf got modified only */
+		return;
+	}
+
+	/*
+	 * TODO: we need correction if border is smaller than current one
+	 */
+	k = depth - 1;
+	border = path[depth].p_ext->ee_block;
+	path[k].p_idx->ei_block = border;
+
+	while (k--) {
+		/* change all left-side indexes */
+		if (path[k+1].p_idx != EXT_FIRST_INDEX(path[k+1].p_hdr))
+			break;
+		path[k].p_idx->ei_block = border;
+	}
+
+	return;
+}
+
+int ext3_ext_insert_extent(struct ext3_extent_header *eh,
+				struct ext4_ext_path *path,
+				struct ext3_extent *newext)
+{
+
+	struct ext3_extent *ex, *fex;
+	struct ext3_extent *nearex; /* nearest extent */
+	struct ext4_ext_path *npath = NULL;
+	int depth, len, err = 0, next;
+	struct ext3_extent_header *save_eh = eh;
+
+	depth = eh->eh_depth;
+	ex = path[depth].p_ext;
+
+	/* try to insert block into found extent and return */
+	if (ex && ext3_can_extents_be_merged(ex, newext)) {
+		ex->ee_len = ex->ee_len + newext->ee_len;
+		eh = path[depth].p_hdr;
+		nearex = ex;
+		goto merge;
+	}
+
+repeat:
+	/* recalculate depth here */
+	depth=save_eh->eh_depth;
+
+	eh = path[depth].p_hdr;
+	if (eh->eh_entries < eh->eh_max)
+		goto has_space;
+
+	/* probably next leaf has space for us? */
+	fex = EXT_LAST_EXTENT(eh);
+	next = ext3_ext_next_leaf_block(path);
+
+	if (newext->ee_block > fex->ee_block && next != EXT_MAX_BLOCK) {
+
+		npath = ext3_ext_find_extent(save_eh, next, NULL);
+		if (!npath) {
+			return EXT2_ET_BLOCK_ALLOC_FAIL;
+		}
+
+		eh = npath[depth].p_hdr;
+		if (eh->eh_entries < eh->eh_max) {
+			path = npath;
+			goto repeat;
+		}
+	}
+
+	/*
+	 * There is no free space in the found leaf.
+	 * We're gonna add a new leaf in the tree.
+	 */
+	err = ext3_ext_create_new_leaf(save_eh, path, newext);
+	if (err)
+		goto cleanup;
+
+	/* recalculate depth here */
+	depth=save_eh->eh_depth;
+	eh = path[depth].p_hdr;
+
+has_space:
+
+	nearex = path[depth].p_ext;
+
+	if (!nearex) {
+		/* there is no extent in this leaf, create first one */
+		path[depth].p_ext = EXT_FIRST_EXTENT(eh);
+
+	} else if (newext->ee_block > nearex->ee_block) {
+
+		if (nearex != EXT_LAST_EXTENT(eh)) {
+			len = EXT_MAX_EXTENT(eh) - nearex;
+			len = (len - 1) * sizeof(struct ext3_extent);
+			len = len < 0 ? 0 : len;
+			memmove(nearex + 2, nearex + 1, len);
+		}
+		path[depth].p_ext = nearex + 1;
+	} else {
+		len = (EXT_MAX_EXTENT(eh) - nearex)*sizeof(struct ext3_extent);
+		len = len < 0 ? 0 : len;
+		memmove(nearex + 1, nearex, len);
+		path[depth].p_ext = nearex;
+	}
+
+	eh->eh_entries = eh->eh_entries+1;
+	nearex = path[depth].p_ext;
+	nearex->ee_block = newext->ee_block;
+	nearex->ee_start = newext->ee_start;
+	nearex->ee_start_hi = newext->ee_start_hi;
+	nearex->ee_len = newext->ee_len;
+
+merge:
+	/* try to merge extents to the right */
+	while (nearex < EXT_LAST_EXTENT(eh)) {
+		if (!ext3_can_extents_be_merged(nearex, nearex + 1))
+			break;
+		/* merge with next extent! */
+		nearex->ee_len = nearex->ee_len + nearex[1].ee_len;
+
+		if (nearex + 1 < EXT_LAST_EXTENT(eh)) {
+			len = (EXT_LAST_EXTENT(eh) - nearex - 1)
+					* sizeof(struct ext3_extent);
+			memmove(nearex + 1, nearex + 2, len);
+		}
+		eh->eh_entries = eh->eh_entries-1;
+	}
+
+	/* try to merge extents to the left */
+
+	/* time to correct all indexes above */
+	ext3_ext_correct_indexes(save_eh, path);
+
+cleanup:
+	if (npath) {
+		free(npath);
+	}
+
+	return err;
+}
+
ext4migrate/extents.h new file +10 -0
new file mode 100644
index 0000000..38114dd
@@ -0,0 +1,10 @@
+
+struct ext4_ext_path * ext3_ext_find_extent(struct ext3_extent_header *,
+						blk_t , struct ext4_ext_path *);
+struct blk_cache * get_block(ext4_fsblk_t);
+int ext3_ext_space_root(void);
+ext4_fsblk_t ext_pblock(struct ext3_extent *);
+ext4_fsblk_t idx_pblock(struct ext3_extent_idx *);
+void ext3_ext_store_pblock(struct ext3_extent *, ext4_fsblk_t );
+int ext3_ext_insert_extent(struct ext3_extent_header *,
+				struct ext4_ext_path *, struct ext3_extent *);
Signature · 1 line
1.5.2.rc0.71.g4342-dirty
#3 lore raw

PATCH 3/4 e2fsprogs: Add ext4migrate

From: Aneesh Kumar K.V <[email protected]>

Add ext4migrate utility that helps in migrating a ext3 block mapped inode to ext4 extent mapped inode.

ext4migrate command takes the below syntax ext4migrate --display | --migrate <image_name> [<filename>]

The --display option helps in displaying the block map details for an ext3 inode and extent map details for an ext4 inode. The --migrate option convert a block mapped ext3 inode to extent mapped ext4 inode.

This needs to be run on an unmounted file system (offline migration).

The inode modification is done only at the last stage. This is to make sure that if we fail at any intermediate stage, we exit without touching the disk.

---
 Makefile.in             |    3 +-
 configure.in            |   21 ++-
 ext4migrate/Makefile.in |   66 ++++
 ext4migrate/migrate.c   |  791 +++++++++++++++++++++++++++++++++++++++++++++++
 ext4migrate/migrate.h   |   30 ++
 5 files changed, 909 insertions(+), 2 deletions(-)
 create mode 100644 ext4migrate/Makefile.in
 create mode 100644 ext4migrate/migrate.c
 create mode 100644 ext4migrate/migrate.h
Makefile.in +2 -1
index 0d31caa..9d8d291 100644
@@ -9,9 +9,10 @@ INSTALL = @INSTALL@
 
 @RESIZER_CMT@RESIZE_DIR= resize
 @DEBUGFS_CMT@DEBUGFS_DIR= debugfs
+@EXT4MIGRATE_CMT@EXT4MIGRATE_DIR= ext4migrate
 
 LIB_SUBDIRS=lib/et lib/ss lib/e2p lib/ext2fs lib/uuid lib/blkid intl
-PROG_SUBDIRS=e2fsck $(DEBUGFS_DIR) misc $(RESIZE_DIR) tests/progs po
+PROG_SUBDIRS=e2fsck $(DEBUGFS_DIR) misc $(RESIZE_DIR) tests/progs po $(EXT4MIGRATE_DIR)
 SUBDIRS=util $(LIB_SUBDIRS) $(PROG_SUBDIRS) tests
 
 SUBS= lib/ext2fs/ext2_types.h lib/blkid/blkid_types.h lib/uuid/uuid_types.h
configure.in +20 -1
index 44b718d..5ae5e0b 100644
@@ -436,6 +436,24 @@ echo "Building e2fsck statically by default"
 )
 AC_SUBST(E2FSCK_TYPE)
 dnl
+dnl handle --enable-ext4migrate
+dnl
+AC_ARG_ENABLE([ext4migrate],
+[  --disable-ext4migrate	  disable support of ext4migrate program],
+if test "$enableval" = "no"
+then
+	echo "Disabling ext4migrate support"
+	EXT4MIGRATE_CMT="#"
+else
+	EXT4MIGRATE_CMT=
+	echo "Enabling ext4migrate support"
+fi
+,
+echo "Enabling ext4migrate support by default"
+EXT4MIGRATE_CMT=
+)
+AC_SUBST(EXT4MIGRATE_CMT)
+dnl
 dnl See whether to install the `fsck' wrapper program (that calls e2fsck)
 dnl
 AC_ARG_ENABLE([fsck],
@@ -862,7 +880,8 @@ for i in MCONFIG Makefile e2fsprogs.spec \
 	lib/e2p/e2p.pc lib/blkid/blkid.pc lib/ext2fs/ext2fs.pc \
 	misc/Makefile ext2ed/Makefile e2fsck/Makefile \
 	debugfs/Makefile tests/Makefile tests/progs/Makefile \
-	resize/Makefile doc/Makefile intl/Makefile po/Makefile.in ; do
+	resize/Makefile doc/Makefile intl/Makefile po/Makefile.in \
+	ext4migrate/Makefile ; do
 	if test -d `dirname ${srcdir}/$i` ; then
 		outlist="$outlist $i"
 	fi
ext4migrate/Makefile.in new file +66 -0

66 lines not on this page · raw message

ext4migrate/migrate.c new file +791 -0

791 lines not on this page · raw message

ext4migrate/migrate.h new file +30 -0
new file mode 100644
index 0000000..7dbb93b
@@ -0,0 +1,30 @@
+
+#include "ext2fs/ext2_fs.h"
+#include "ext2fs/ext2fs.h"
+#include "blkid/list.h"
+
+typedef unsigned long long ext4_fsblk_t;
+
+extern ext2_filsys current_fs;
+
+#define EXT4_EXT_MAGIC		0xf30a
+#define EXT2_SET_INCOMPAT_FEATURE(sb,mask) \
+	( EXT2_SB(sb)->s_feature_incompat |= (mask) )
+
+
+struct list_blocks_struct {
+	blk_t		first_block, last_block;
+	e2_blkcnt_t	first_bcnt, last_bcnt;
+};
+
+struct  blk_stat {
+	struct list_head  head;
+	ext4_fsblk_t pblock;
+	int dirty;
+};
+
+struct blk_cache {
+	struct list_head  head;
+	ext4_fsblk_t pblock;
+	void * block_mem_addr;
+};
Signature · 1 line
1.5.2.rc0.71.g4342-dirty
#4 lore raw

PATCH 4/4 Add ext4replay tool.

From: Aneesh Kumar K.V <[email protected]>

This tool can be used to replay the transactions stored in the database during ext4migrate.

---
 ext4migrate/Makefile.in |    7 ++++++-
 ext4migrate/replay.c    |   45 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+), 1 deletions(-)
 create mode 100644 ext4migrate/replay.c
ext4migrate/Makefile.in +6 -1
index b000ae7..8fe6291 100644
@@ -21,7 +21,7 @@ DEPLIBS= $(LIBEXT2FS)  $(LIBCOM_ERR)
 	@#cc -g -I../lib/  -Wunreachable-code -Wunused -Wunused-function
 	@#-Wunused-label  -Wunused-parameter -Wunused-value  -Wunused-variable  -c migrate.c
 
-PROGS= ext4migrate
+PROGS= ext4migrate ext4replay
 
 all:: $(PROGS)
 
@@ -29,6 +29,10 @@ ext4migrate: migrate.o  extents.o $(DEPLIBS)
 	@echo "	LD $@"
 	@$(CC) $(ALL_LDFLAGS) -o ext4migrate migrate.o extents.o $(LIBS)
 
+ext4replay: replay.o
+	@echo "	LD $@"
+	@$(CC) $(ALL_LDFLAGS) -o ext4replay replay.o  -ltdb
+
 installdirs:
 	@echo "	MKINSTALLDIRS $(root_sbindir)"
 	@$(MKINSTALLDIRS) $(DESTDIR)$(root_sbindir)
@@ -64,3 +68,4 @@ distclean: clean
 #
 migrate.o: $(srcdir)/migrate.c
 extents.o: $(srcdir)/extents.c
+replay.o:  $(srcdir)/replay.c
ext4migrate/replay.c new file +45 -0

45 lines not on this page · raw message

Signature · 1 line
1.5.2.rc0.71.g4342-dirty

A message that is not in the archive (sent off-list or not yet received): eb16ba5f2c6f399ea747169097434ae84443728f.1178262585.git.aneesh.kumar@linux.vnet.ibm.com

#6 lore raw

Re: [PATCH 1/4] Add unix COW io manager

Quoted 2 lines · Aneesh Kumar K.V

+ * unix_cow_io.c --- This is the Unix (well, really POSIX) implementation
+ * of the COW I/O manager.

I really wish you had based this not on unix_io.c, but rather on test_io.c, for two reasons (a) this reduces code duplication, (b) it means that we can chain/stack I/O managers. i.e., test_io -> cow_manager -> unix_io.

The one change I would recommend is that instead of assigning to a globally visible external variable (as I did in test_io, and which I now regret as a not necessarily being portable --- different shared library implementations have different restrictions on whether you can access global variables defined in shared libraries from an application or vice versa --- with AIX having some of the most peverse shared library restritions, at least 7-8 years ago when I had the misfortune of trying to make Kerberos v5 shared libraries work on AIX :-). Instead, what I would recommend is creating a function which an application could call to set the backing I/O manager for the cow_manager, as well as the filenames to use for the tdb journal.

The second change I would recommend is to NOT call it "cow_manager" but rather an "undo_manager", since that makes it clear that what is being backed up is not the work that we are about to do, but rather the original contents of the disk. I would also rename the ext4_replay tool something like e2undofs, since "replay" has the connotation that you are applying changes that were done in a log, when in fact what you are really doing is backing out changes.

We want to make sure we avoid this confusion, since Xen and UML and most other virtualization engines, for good reasons, do the exact opposite --- the COW file contains the changes, and the base file/device remains unchanged.

lib/ext2fs/unix_cow_io.c line 295

Quoted 6 lines
lib/ext2fs/unix_cow_io.c new file +6 -0
+	retval = tdb_store(data->tdb, tdb_key, tdb_data, TDB_INSERT);
+	if (retval == -1) {
+		/* FIXME!! Not sure what should be the error */
+		retval = EXT2_ET_SHORT_WRITE;
+		goto error_out;
+	}

You need to write the tdb_store() around a tdb_transaction_start() and tdb_transaction_commit() pair, so that we can be sure the original data is actually synced to disk. Otherwise, on a crash, we could have a situation where the original data on disk has been overwritten, but the backup of the original block was never forced to disk!

- Ted

#7 lore raw
In reply to Aneesh Kumar K.V
Quoted 6 lines · Aneesh Kumar K.V

Changes from from my previous patches are as follows:
a) support for files with holes
b) use the block iterator present in libext2fs
c) don't mark the indirect blocks as unused early. If we mark the the blocks unused, they can be resued for extent index. If we later fail to migrate, the indirect blocks content would be overwritten.

If this code could also (or optionally just) increase the size of inodes it would be very useful. AFAIK right now it will only change the inodes from block-mapped to extent-mapped?

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

#8 lore raw
Quoted 8 lines · Andreas Dilger
Quoted 6 lines · Aneesh Kumar K.V

Changes from from my previous patches are as follows:
a) support for files with holes
b) use the block iterator present in libext2fs
c) don't mark the indirect blocks as unused early. If we mark the the blocks unused, they can be resued for extent index. If we later fail to migrate, the indirect blocks content would be overwritten.

Quoted 5 lines

If this code could also (or optionally just) increase the size of inodes it would be very useful. AFAIK right now it will only change the inodes from block-mapped to extent-mapped?

That is correct. My next step is to enhance the tool to support migration to large inodes.

-aneesh

#9 lore raw
Quoted 4 lines · Aneesh Kumar K.V
Quoted 3 lines · Andreas Dilger

If this code could also (or optionally just) increase the size of inodes it would be very useful. AFAIK right now it will only change the inodes from block-mapped to extent-mapped?

Actually, changing the inode size is actually going to be more useful to a larger number of people, since they can use it today to support in-inode EA's. In addition, changing the inode size must be done off-line, whereas it's not so clear that off-line conversion to extents is the best way to go (more on that below).

One generic concern I have about simply migrating existing files to use extents is that unless we actually also combine the tool with defragmentation support, the file ends up being fairly badly fragmented simply because of the "holes" left in the file from the indirect blocks. Therefore the tree ends up being far larger than it needs to be, and future attempts allocate blocks in that block group will fill in the holes, further degrading the performance of the filesystem and whatever file is being written at the time.

The other reason why increasing the inode size would actually be more valuable is that right now, if we have the disk space, the user can do on-line migration of a file simply by copying it and then moving it over the original file --- and if we have delayed allocation support and sufficient memory, we can avoid the fragmentation problems currently with the off-line ext4migration approach.

So we may need to think a bit about what's the best way to handle ext4migrate. It may be that correct approach is to combine it with a defragmentation tool, either on-line or off-line.

Quoted 2 lines

That is correct. My next step is to enhance the tool to support migration to large inodes.

I would actually suggest that the place to add that functionality would be via "tune2fs -I <inode_size>". Since mke2fs -I <inode_size> is used to set the inode size, it makes sense that tune2fs -I would change the inode size. Obviously this would have to be done when the filesystem is not mounted, and tune2fs should abort gracefully if determines that the filesystem is mounted.

- Ted

#10 lore raw

Hi,

Quoted 6 lines · Theodore Tso
Quoted 4 lines · Aneesh Kumar K.V
Quoted 3 lines · Andreas Dilger

If this code could also (or optionally just) increase the size of inodes it would be very useful. AFAIK right now it will only change the inodes from block-mapped to extent-mapped?

Quoted 6 lines

Actually, changing the inode size is actually going to be more useful to a larger number of people, since they can use it today to support in-inode EA's. In addition, changing the inode size must be done off-line, whereas it's not so clear that off-line conversion to extents is the best way to go (more on that below).

One of the option i was thinking was to use this tool to migrate to extent map and then towards the end use the online defrag ioctl to defrag the resulting ext4 inode.

That way we don't need to add kernel code that convert ext3 inode to ext4 inode while defragmentation. Last time i looked at the online defrag code, i was not able to figure out a easy way to take indirect map inode as input, then defrag the same and write the result in extent map.

-aneesh

#11 lore raw
Quoted 4 lines · Aneesh Kumar K.V

One of the option i was thinking was to use this tool to migrate to extent map and then towards the end use the online defrag ioctl to defrag the resulting ext4 inode.

As we discussed on the phone, my recommendation would be to take your existing code and move it into the kernel so that triggered off of an ioctl, your code could migrate an inode from using indirect blocks to extents while a filesystem is mounted. The main things you will need to watch for is to make sure the inode is locked so that another process doesn't try to extend or truncate it, and to use the JBD layer to provide appropriate journalling support.

Given that ext4migrated imported the kernel extent functions, it should be relatively straightforward to simply make them use the kernel extent functions while in kernel space.

Once the the inode has been converted on-line then it can be defragmented on-line. That would be much more convenient than having to unmount the filesystem to do the off-line migration, followed by mounting it to do the defragmentation.

Regards,

- Ted

#12 lore raw
In reply to Aneesh Kumar K.V
Quoted 7 lines · Aneesh Kumar K.V
Quoted 5 lines · Theodore Tso

Actually, changing the inode size is actually going to be more useful to a larger number of people, since they can use it today to support in-inode EA's. In addition, changing the inode size must be done off-line, whereas it's not so clear that off-line conversion to extents is the best way to go (more on that below).

Quoted 5 lines

That way we don't need to add kernel code that convert ext3 inode to ext4 inode while defragmentation. Last time i looked at the online defrag code, i was not able to figure out a easy way to take indirect map inode as input, then defrag the same and write the result in extent map.

I think the online defrag code _should_ be fairly easily made to handle defrag of block-mapped files, so long as it is always creating extent- mapped files in the end. All that should be necessary is to have the kernel read data from the block-mapped file and write it into the newly mapped blocks. That functionality needs to exist for a long time anyways to support upgrading the filesystem so it shouldn't be a huge burden.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

#13 lore raw
In reply to Theodore Tso
Quoted 9 lines · Theodore Tso
Quoted 2 lines

That is correct. My next step is to enhance the tool to support migration to large inodes.

I would actually suggest that the place to add that functionality would be via "tune2fs -I <inode_size>". Since mke2fs -I <inode_size> is used to set the inode size, it makes sense that tune2fs -I would change the inode size. Obviously this would have to be done when the filesystem is not mounted, and tune2fs should abort gracefully if determines that the filesystem is mounted.

That is nice !!. I will add the large inode migration support to tune2fs.

-aneesh