e2defrag 0.81 released

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

It's back!

Back in the days when dinosaurs walked the earth ( the '90s ), there was a defragger for ext2 written by Stephen Tweedie with contributions from others including Ted Ts'o. After many years of abandonment and bit rot, I have decided to take over maintainership of the package. I now feel that it is in good enough of a state for a wider audience ( including working with extents and other ext4 features ), so I am announcing it here and asking for testing. The project page is
http://launchpad.net/e2defrag.

The program opens an unmounted block device and parses the filesystem itself, assigns new locations for all blocks, packing them to the left within their native block group if possible, then moves all the blocks around quickly and efficiently. The process it fast, but unsafe: should anything go wrong or the process be interrupted, your fs will be toast.
Therefore:

DO NOT USE ON A FS YOU CARE ABOUT AND/OR HAVE NOT BACKED UP FIRST

Should a bug cause it to trash your fs, a raw e2image ( e2image -r /dev/sda1 - | bzip2 -c > sda1.e2i.bz2 ) made before the defrag, and obviously saved on another fs, would be most helpful in debugging.

One of the more interesting features is the ability to pass a list of inodes to be given priority over others. This can be used to pack a set of files together at the start of the disk to allow for faster booting.
I cobbled together a simple python script, dump2inodes, that can obtain
the list of files that ureadahead ( Ubuntu ) loads during boot and generates the inode priority listing you can pass to e2defrag, and this gives some nice boot time improvements.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJP9243AAoJEJrBOlT6nu75GPAIAJuZ7IiBH1YinJXFTil+R6z9 EtLI4lZkHeWUGbVr/CnhV9l1iYIVo/aqPCBo4yH0efFGPSxF9z/KN8zIEvqs3Q30 nDe1dNACyvkeIfzyP7wkFxuqK7hX7wkF+cIYT24MftTGTfaXssnoJ5bGd9TyJ0+s 8fNM/Erq4saRBnq4aty/Q3FsAF9sc4ke5042FHNSM+xCFfHjZcsciTdPZP5svAcH 8crQFmR9ieDIbn1M54mWWp3hYTd368ScJSoiM+K+d8ofbrXRSapJ9bSvuhLjvoxE Tx7QXFdHoJb+Id4yf6xI/OZjg9egPW6otkPWwUJfAIE2Wh2AAfnAsGNg4QA86Ws= =PK87
-----END PGP SIGNATURE-----

I hate to rain on your parade, but, are you aware of e4defrag?

It is already in e2fsprogs, and can be used on a mounted ext4 filesystem... It also needs some lovin' to make it really robust, but would definitely be a better starting point than the ancient e2defrag code...

That's why it is always a good idea to post to the mailing list _before_ you start on a project, to see what else is going on... Definitely there is a need for such a tool, but I hate to see effort being spent in two different directions to make two so-so tools, when it could be going in the same direction to make one excellent tool.

Cheers, Andreas

Cheers, Andreas

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Of course. I started working on e2fsprogs two years ago though, when e4defrag was still on the drawing board, and just haven't gotten around to polishing and announcing it until now.

My understanding of e4defrag is that it works essentially like tools such as shake: find a fragmented file and copy it, hoping the copy will be less fragmented. Thus, it does nothing for free space fragmentation, file locality, etc. Plus, it doesn't have cool ansi graphics ;)

I'm hoping that eventually e4defrag will be able to do everything e2defrag can, and not too much slower, but for the moment, e2defrag already works, and I haven't seen any indication that e4defrag is planned to do free space defragging and file packing. Perhaps having a little friendly competition will change that.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJP+E1MAAoJEJrBOlT6nu75bkYH/2rXRYqMYAA+D/EUwxfvq18p ltrb3nOWXA7ZA6Gy1Lg+zcoG203F44XOKg8aNADS/1sYKkyiF5NhFcGRgXrp/gM/ 0586WqpkAc0rhuk0vGcDinr8KOdaeTH3Ye5zN/6jju/bEls3Q5wpSCU71LfYYj62 Q+YOJK+TRq6r5NudIe6rIbNArVPMedPDw17L6yP4opi2iPrylW/WndStFFzjDAgK Fa0/YRLl+SLwfKvCSorBtuSpuOToC1bfdq2ZmqIX7VCe7WThTVAzE9avDoaDhzIj LK3ErsyztJZWPcfr34A1BEYZn2ZH5+Hsf+/Pmh4M8grMRgGfUhy1UU1/t7Zhyq0= =Bxln
-----END PGP SIGNATURE-----

e4defrag has been around more than 2 years. It's been at least 2 years since I did a code review of how it works.

More or less true except iirc the process is more like:

find a fragmented file.
fallocate a new donor file with uninitialized data blocks.. (ie. data blocks get assigned, but not written to.)
if new donor file is less fragmented than orig file:
copy data from orig_blocks to donor_blocks
swap donor blocks with original blocks at the inode level.
end if
delete donor

At a minimum I think e4defrag should work better with sparse files and defrag each block group separately.

Also it used to hold the file lock for the entire time the copy data / swap donor blocks operation took place. For a large file that could be a very long time. I don't know if that was ever made less intrusive or not. ie. defragging a 100 GB file used to require putting a i/o lock in place for the entire time it took to copy that 100GB of blocks to the donor blocks.

Of course, even that is better than having to unmount the filesystem like you new tool apparently requires.

I think you're right e4defrag userspace code ignores consolidating freespace to make bigger extents possible and I don't think userspace has any file packing specific knowledge.

Does fallocate itself assist with either?

ie. since e4defrag uses fallocate to allocate the donor file blocks, logically it would make sense to add the intelligence you are looking for into fallocate. And hopefully it is already there.

Greg

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Old tool... very old tool ;)

I remember using the thing with one of the first releases of slackware when I first started playing with Linux, and iirc, DOS hadn't yet included its own defragger.

You need a method of requesting specific blocks instead of just asking for the right number. Then you need to analyze all of the files on the disk to figure out where their blocks are, and decide where you want them to be instead. Then you could use the donor inode method to move things around.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJP+yfPAAoJEJrBOlT6nu75Y8AH/09qDUt0jdhPjNyXMUUFRVaV U7Ih5j6r0srRHH9VmS7ppq6ingqpbrFEExbalg8ynTRX1nYDGL0nLiBU79rfK+KK XOTK0ZYm+CUMElLdO5bYnaHti9vsSHL5xRyGmI5OR0CY7LwfnyJeDw6wS7+WsgyQ o6ziTPcBEfaJzn9BBS+EP/1WI06lQ/cJbWUqUAJhefu2my3IXJUrjUYjnH46HKGM 2ptPbIJM2fT4QubBNnuPpdTNI5yc56+auzoB3hrK2Nf/Yh17w5R0MomHk/VitzPt AD7zKLgpk6R9G/IbnajsMVzbLsjBbn8Tcyrhher2uyw3xE0XtJgj2fmPq32ZrL0= =hMeu
-----END PGP SIGNATURE-----