CheckFS: Checkpoints and Block Level Incremental Backup (BLIB)
Ted,
Thanks for your suggestions and help.
We started it from 2.6.7 last year and then it was sitting idle for several months for lack of resources. We'll go back to that version and generate a diff that's easier to read.
Yes, changing the name has made the task of rebasing wrt. changing kernels lot difficult. Our original intention was to make testing easier by keeping ext3 and checkfs filesystems in the same kernel. Had we continued it at that point, we would have posted differences wrt. ext3 sources themselves. There was compelling reason to change the name.
Regards.
-Amit
Quoted 24 lines
This looks like very interesting technology, but out of curiosity, why did you develop this as separate filesystem with a new filesystem name, and doing a global search-and-replace of "ext3" with "checkfs" in the source files, instead of simply just modifying ext3 and posting diffs? Especially since that the apparent intention is to keep ext3 compatibility using the same ext3 magic numbers, data formats, and user-mode utilities.
I'll reserve the superblock fields and compatibility bitmap fields used by your code in e2fsprogs to help avoid the possibility of other folks working on ext3 extensions from colliding with the codepoints which you "borrowed", but in the future, it would be good if people contact me in advance so I ensure that there are no collisions with other development groups.
What version of the source base did you originally fork checkfs from? That way we can do a "s/checkfs/ext3/g" search and replace and then generate some diffs to see what you changed, or alternatively, it would be even better if you minimized differences between your version and mainline and generated the diffs yourself.
Thanks, regards,
- Ted
Quoted 10 lines · Amit S. Kale
We started it from 2.6.7 last year and then it was sitting idle for several months for lack of resources. We'll go back to that version and generate a diff that's easier to read.
Yes, changing the name has made the task of rebasing wrt. changing kernels lot difficult. Our original intention was to make testing easier by keeping ext3 and checkfs filesystems in the same kernel. Had we continued it at that point, we would have posted differences wrt. ext3 sources themselves. There was compelling reason to change the name.
One easier way of doing development, particularly for people doing filesystem work, is to compile the kernel with the test filesystem code using user-mode linux (UML) architecture. This significantly shortens your edit-compile-debug cycle time, and it makes it easier to run your filesystem code under a debugger. That way you also don't have to worry about your filesystem changes toasting your system, either. This technique doesn't work all that well for people doing architecture-specific code or for device drivers, but for filesystems, it's ideal.
Regards,
- Ted
Hi!
Quoted 11 lines
Thanks for your suggestions and help.
We started it from 2.6.7 last year and then it was sitting idle for several months for lack of resources. We'll go back to that version and generate a diff that's easier to read.
Yes, changing the name has made the task of rebasing wrt. changing kernels lot difficult. Our original intention was to make testing easier by keeping ext3 and checkfs filesystems in the same kernel. Had we continued it at that point, we would have posted differences wrt. ext3 sources themselves. There was compelling reason to change the name.
Maybe you want to put your development machines on ext*2* while doing this ;-). Or perhaps reiserfs/xfs/something.
Pavel
Signature · 1 line
teflon -- maybe it is a trademark, but it should not be.
Quoted 3 lines
Maybe you want to put your development machines on ext*2* while doing this ;-). Or perhaps reiserfs/xfs/something.
Or perhaps into at the VFS level, so any fs can benefit from it.
Jan Engelhardt
Quoted 4 lines · Jan Engelhardt
Quoted 2 lines
Maybe you want to put your development machines on ext*2* while doing this ;-). Or perhaps reiserfs/xfs/something.
Or perhaps into at the VFS level, so any fs can benefit from it.
We thought about that. While it's possible to do that, it would need hooks into all filesystems etc. Definitely worth trying once we get some more basic stuff working for ext3
After all the things that need to be saved at the time of taking a checkpoint for any filesystem would be the superblock and inode table (or their equivalents). Everything else is automatically taken care of.
-Amit
Quoted 6 lines · Amit S. Kale
Quoted 4 lines · Jan Engelhardt
Quoted 2 lines
Maybe you want to put your development machines on ext*2* while doing this ;-). Or perhaps reiserfs/xfs/something.
Or perhaps into at the VFS level, so any fs can benefit from it.
Quoted 8 lines
We thought about that. While it's possible to do that, it would need hooks into all filesystems etc. Definitely worth trying once we get some more basic stuff working for ext3
After all the things that need to be saved at the time of taking a checkpoint for any filesystem would be the superblock and inode table (or their equivalents). Everything else is automatically taken care of.
You are aware of the block-device-level snapshot solutions, right? They can be more painful to use, although that's mostly a UI issue, and they have the added advantage that you can safely run e2fsck on the snapshot image if you want to check the consistency of the filesystem while it is mounted. (If it is clean, you can then use tune2fs to reset the max-mount-count and last-checked-time on the original filesystem image; if it is not clean, you can send e-mail to the system administrator suggesting that she schedule downtime ASAP and do a e2fsck on the filesystem image. This is a good thing that a paranoid sysadmin might schedule via cron every Saturday morning at 3am, for example.)
- Ted
Quoted 16 lines · Theodore Ts'o
Quoted 6 lines · Amit S. Kale
Quoted 4 lines · Jan Engelhardt
Quoted 2 lines
Maybe you want to put your development machines on ext*2* while doing this ;-). Or perhaps reiserfs/xfs/something.
Or perhaps into at the VFS level, so any fs can benefit from it.
Quoted 8 lines
We thought about that. While it's possible to do that, it would need hooks into all filesystems etc. Definitely worth trying once we get some more basic stuff working for ext3
After all the things that need to be saved at the time of taking a checkpoint for any filesystem would be the superblock and inode table (or their equivalents). Everything else is automatically taken care of.
Quoted 5 lines
You are aware of the block-device-level snapshot solutions, right? They can be more painful to use, although that's mostly a UI issue, and they have the added advantage that you can safely run e2fsck on the snapshot image if you want to check the consistency of the filesystem while it is mounted.
That would be a lesser advantage IMHO compared to the advantage of being able to manage disks in a better way.
Block-device level snapshots are simple to implement, though making them intelligent is quite difficult. Block device level snapshots can't detect which block are allocated and which are not (unless they look into a filesystem's block allocation map, which requires a device->fs interface). A snapshot may hence require an exhorbitant amount of space when it's not really needed.
The BLIB procedure to be used with checkfs is to create a checkpoint. Transfer it to disk as the first full backup. This operation uses the free space available within the filesystem till the time the data is transfered to a tape. A similar procedure used with block-level backup will require an auxilliary device.
Generally filesystem level snapshots/checkpoints can be more intelligent hence easier to use and requires fewer resources.
-Amit
Quoted 7 lines
(If it is clean, you can then use
tune2fs to reset the max-mount-count and last-checked-time on the original filesystem image; if it is not clean, you can send e-mail to the system administrator suggesting that she schedule downtime ASAP and do a e2fsck on the filesystem image. This is a good thing that a paranoid sysadmin might schedule via cron every Saturday morning at 3am, for example.)