e2fsprogs: Undo I/O manager

This is a very early prototype of what i working on. I have done only minimal testing of this. To use one need to use the below steps.

       io_manager io_mng = stack_io_manager_init();
       stack_push_io_manager(io_mng, unix_io_manager);
       stack_push_io_manager(io_mng, undo_io_manager);
       flags = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_RW;
       retval = ext2fs_open2(dev_name,"tdbfilename=/tmp/test.tdb&",
                                       flags, superblock, block_size,
                                       io_mng, &current_fs);

I am sending this across to find out if this is the approach that we want to follow.

-aneesh

From: Aneesh Kumar K.V <[email protected]>

This I/O manager helps in stacking different I/O managers. For example one can stack the undo I/O manager on top of Unix I/O manager to achieve the undo functionality.

---
 lib/ext2fs/Makefile.in |   10 +-
 lib/ext2fs/ext2_io.h   |    4 +
 lib/ext2fs/stack_io.c  |  419 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 431 insertions(+), 2 deletions(-)
 create mode 100644 lib/ext2fs/stack_io.c

index 799659e..f5d141a 100644
@@ -66,7 +66,8 @@ OBJS= $(DEBUGFS_LIB_OBJS) $(RESIZE_LIB_OBJS) $(E2IMAGE_LIB_OBJS) \
 	unix_io.o \
 	unlink.o \
 	valid_blk.o \
-	version.o
+	version.o \
+	stack_io.o
 
 SRCS= ext2_err.c \
 	$(srcdir)/alloc.c \

@@ -134,7 +135,8 @@ SRCS= ext2_err.c \
 	$(srcdir)/tst_bitops.c \
 	$(srcdir)/tst_byteswap.c \
 	$(srcdir)/tst_getsize.c \
-	$(srcdir)/tst_iscan.c
+	$(srcdir)/tst_iscan.c \
+	$(srcdir)/stack_io.c
 
 HFILES= bitops.h ext2fs.h ext2_io.h ext2_fs.h ext2_ext_attr.h ext3_extents.h
 HFILES_IN=  ext2_err.h ext2_types.h

@@ -550,3 +552,7 @@ tst_iscan.o: $(srcdir)/tst_iscan.c $(srcdir)/ext2_fs.h \
  $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \
  $(srcdir)/ext2_fs.h $(top_srcdir)/lib/et/com_err.h $(srcdir)/ext2_io.h \
  $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/bitops.h
+stack_io.o: $(srcdir)/stack_io.c $(srcdir)/ext2_fs.h \
+ $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \
+ $(srcdir)/ext2_fs.h $(top_srcdir)/lib/et/com_err.h $(srcdir)/ext2_io.h \
+ $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/bitops.h

index eada278..65a294e 100644
@@ -96,6 +96,10 @@ extern errcode_t io_channel_write_byte(io_channel channel,
 /* unix_io.c */
 extern io_manager unix_io_manager;
 
+/* stack_io.c */
+extern struct struct_io_manager *stack_io_manager_init(void);
+extern errcode_t stack_push_io_manager(struct struct_io_manager *stack,
+						struct  struct_io_manager *io);
 /* test_io.c */
 extern io_manager test_io_manager, test_io_backing_manager;
 extern void (*test_io_cb_read_blk)

1.5.2.rc0.71.g4342-dirty

From: Aneesh Kumar K.V <[email protected]>

This I/O manager saves the contents of the location being overwritten to a tdb database. This helps in undoing the changes done to the file system.

This can be stacked over unix I/O manager to get the undo functionality.

---
 lib/ext2fs/Makefile.in |   10 +-
 lib/ext2fs/ext2_io.h   |    3 +
 lib/ext2fs/undo_io.c   |  443 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 454 insertions(+), 2 deletions(-)
 create mode 100644 lib/ext2fs/undo_io.c

index f5d141a..669d4e7 100644
@@ -67,7 +67,8 @@ OBJS= $(DEBUGFS_LIB_OBJS) $(RESIZE_LIB_OBJS) $(E2IMAGE_LIB_OBJS) \
 	unlink.o \
 	valid_blk.o \
 	version.o \
-	stack_io.o
+	stack_io.o \
+	undo_io.o
 
 SRCS= ext2_err.c \
 	$(srcdir)/alloc.c \

@@ -136,7 +137,8 @@ SRCS= ext2_err.c \
 	$(srcdir)/tst_byteswap.c \
 	$(srcdir)/tst_getsize.c \
 	$(srcdir)/tst_iscan.c \
-	$(srcdir)/stack_io.c
+	$(srcdir)/stack_io.c  \
+	$(srcdir)/undo_io.c
 
 HFILES= bitops.h ext2fs.h ext2_io.h ext2_fs.h ext2_ext_attr.h ext3_extents.h
 HFILES_IN=  ext2_err.h ext2_types.h

@@ -556,3 +558,7 @@ stack_io.o: $(srcdir)/stack_io.c $(srcdir)/ext2_fs.h \
  $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \
  $(srcdir)/ext2_fs.h $(top_srcdir)/lib/et/com_err.h $(srcdir)/ext2_io.h \
  $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/bitops.h
+undo_io.o: $(srcdir)/undo_io.c $(srcdir)/ext2_fs.h \
+ $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \
+ $(srcdir)/ext2_fs.h $(top_srcdir)/lib/et/com_err.h $(srcdir)/ext2_io.h \
+ $(top_builddir)/lib/ext2fs/ext2_err.h $(srcdir)/bitops.h

index 65a294e..78ea27c 100644
@@ -100,6 +100,9 @@ extern io_manager unix_io_manager;
 extern struct struct_io_manager *stack_io_manager_init(void);
 extern errcode_t stack_push_io_manager(struct struct_io_manager *stack,
 						struct  struct_io_manager *io);
+/* undo_io.c */
+extern io_manager undo_io_manager;
+
 /* test_io.c */
 extern io_manager test_io_manager, test_io_backing_manager;
 extern void (*test_io_cb_read_blk)

1.5.2.rc0.71.g4342-dirty

This is probably more generality than is strictly necessary; and the place where the excess generality gets messy is the fact that you make the stacking layer responsible for calling all of the io manager's open routines (which is still a FIXME). So I would flush the stack_io layer entirely.

What I would recommend as the fast and dirty approach. Basically, ape the approach used by test_io layer _exactly_, except instead of using a global variable test_io_backing_manager, you provide a function which sets the static variable, undo_io_backing_manager. This variable is used only by the subsequent call to the ->open method, which just like test_io simply passes the name down to the backing manager specified in the static variable. Then just make the undo_io manager work the way test_io does, where does its thing, and then it calls the appropriate function in its private->real io_channel. Basically, make undo_io responsible for calling the next io_manager down in the chain, This is workable because we don't need to initialize the tdb file until we first try to write to the io_channel, and ext2fs_open() only needs to do read operations, so we can set the tdb filename via an optoin after ext2fs_open() returns.

I could imagine a more complicated approach where we still wouldn't have a stacking layer, but have an integrated parsing layer which would have registered names for each of the io_managers. It would then take the part of the name before the colon, and treat it as the type (i.e., "file"), and the part after the colon and pass it to the relevant's io_manager (i.e., "/dev/sda4"). So you could pass in a name like "file:/dev/sda4".

For the test_io manager, since it doesn't need any arguments, it could just take the part after the colon, and use it to pass back to the parsing layer. So names to test_io manager would look like this:
"test:file:/dev/sda4".

For managers that need two names, we would have to do something a bit more complicated, like this: "undo:/var/undo/tdbfile#file:/dev/sda4". But basically, we still make the stacking up to the individual io_manager. The main advantage with this approach is that all of the stacking is done via the ASCII string provided by the user, and there's no need for any special ad-hoc options.

On the other hand, arguably, you want the program decided where to put the undo file, so that it can be found automatically by e2fsck. So the fast-and-dirty approach might be the better one anyway --- it certainly is easier!

Regards,

- Ted

Can't we address the FIXME by calling the respective close routine in the failure case.

One thing i didn't like with the stack_io was, we will be opening the device at each stacked I/O manager layer; which i also think is okey provided we expect to use these I/O managers independently

One thing i was confused about was the usage of read_blk, write_blk etc pointers in test_private_data. With respect to undo I/O manager do i need to provide them ?. If we really need them, then i was thinking a generic stacking layer as i send in the patches would be better. That means any pluggable functionality that we achieve right now by setting test_io_cb_read_blk etc will be implemented as a I/O manager that does the particular task. Later we stack all these I/O manager to get the full functionality.

-aneesh

You can but it doesn't address the biggest problem, which is if you're going to add all of this extra complexity, we might as well deal with the name parsing issue. In your stacking I/O layer, you pass the same name to all of the stacked modules. That's not necessarily the right thing to do. It works for test_io because it doesn't need a name parameter (it just passes the name straight down to the its lower-layer module). It mostly works for the undo_io layer because while the tdb pathname could be the passed in open argument, it's ext2fs_openfs() is openfs, and you don't have to give it its tdb name until after the ext2fs_openfs() succeeds, and you can pass it down as an option.

In general, though, you need to worry about what sub-parts of the name you need to pass down to each of the stacked modules. So the stack_io layer adds a lot of complexity, but it's not a fully general solution. So if it's not fully general, maybe something that's simpler is sufficient, since in truth we probably don't need unlimited levels of stacking.

The read_blk, write_blk, etc. pointers in test_private_data() are specific to the test_io layer. That's part of the value-add which the test_io module provides, and no, the undo I/O manager doesn't need them. It's just there as part of what the user of the test i/o manager might want to use, by allowing some arbitrary callback function to be called for each test i/o. The main place I've used it is when I want to set effectively a watchpoint on e2fsck, because I'm trying to figure out which part of e2fsck is reading or writing a particular block, and I want to dump out the contents of that block when it reads/writes it. The fastest way to do that is to add a callback function that tests for the block number, and when it is triggered, I can have the debugging function print out some or all of the contents the block. So that's strictly a test i/o manager thing; the undo i/o manager wouldn't need this at all!

Regards,

- Ted