Do we need dump for ext4?

I was talking to Ric about dump benchmarks, and he was of the impression that dump may not be used that often anymore, at least in the enterprise. (Ric, hope I'm paraphrasing correctly)

Undaunted :) I ran off and tested an artificial backup scenario:

* Untar a kernel tree into 128 different top level dirs
* Make a level 0 backup
* Untar a kernel tree into 128 MORE different top level dirs
* Make a level 1 backup

128 kernel trees uses about 6.5M inodes, and about 80G of space.

I tested ext3 with dump; ext4 with tar, and xfs with xfsdump.

for ext3:
dump -1 -u -f $DUMPDIR/dump1 $DATADIR

for ext4:
tar --atime-preserve --xattr --after-date=$DUMPDIR/dump0.tar -cf $DUMPDIR/dump1.tar $DATADIR

for xfs:
xfsdump -F -l 1 -f $DUMPDIR/dump0 $DATADIR

DUMPDIR and DATADIR were 2 partitions on the same (fast hardware) lun.

Results:

	level0	level1
        ------	------

ext3 38m52s 42m21s
ext4 57m55s 69m35s
xfs 25m18s 37m44s

Clearly tar on ext4, at least for my incantation, lags. Is dump for ext4 anywhere on the todo list, or should it be? Or am I just running tar wrong? :)

-Eric

Many people don't use the dump/restore any more program any more, that's definitely true. Whether people use backups (as opposed to large amounts of RAID) in the enterprise is a different question. I'm not so sure about the second question.

So a couple of comments. First, it's probably not fair to use different backup programs for the different filesystems. We probably want to do one set of comparisons where we use tar for all three. (Note: not all backup/dump programs are doing the right things with xattr's, so we're not necessarily comparing programs with completely identical functionality.)

Secondly, it really wouldn't be hard to update dump/restore for ext4. It uses libext2fs, so the real problem is that it is explicitly checking the feature flags. Removing those checks may be all that is necessary, given that ext2_block_iterate() still works for extent-based files. I just noted BTW that the dump/restore doesn't seem to be TOTALLY abandoned. It was last updated in 2006, true, but there is support for backing up and restoring extended attributes and ACL's. I wonder if they broke format compatibility with BSD 4.4 format dump/restore backups when they did it --- and if anyone would still cares. :-)

Finally, I suspect most of the problem with using tar is the HTREE dirent sorting problem. If we modify tar to sort the directory entries before emitting the files, and then use that tar across all the filesystems, I suspect the results would be much more better for ext3 and ext4.

- Ted

I think a lot of high end customers still back up to tape (or virtual tape which is basically a tape emulation on top of RAID arrays), but they use commercial programs to do that.

I like Chris's acp program since that is heavily optimized (read files in inode sorted order) and is small enough to tweak.

We may as well just time "tar" as an easy baseline.

Like acp ;-)

ric

I think there is little benefit to fixing each program to do sorting. Either your LD_PRELOAD library should become more standard (e.g. put into glibc), or we use something like e2scan or a modified find to generate filenames in inode order.

NB: e2scan itself scans the inode table in order, but it makes no effort
to generate filenames in that order, since Lustre doesn't care about that. It likely gets it largely right by coincidence.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Yeah, but that requires dealing with Ulrich and for my own mental health I try to avoid that as much as possible. :-)

This idea is something that has been in my "if only I had time or some minions to dispatch" category for quite some time. We can actually do this in the kernel.

For small directories which could potentially get converted into htree format, we already sucking the entire directory and putting it into an rbtree. We could just do this for all directories less than or equal to 32k, but have them returned sorted by inode instead of by hash value. At least on my laptop, this accounts for 99.93% of the directories on my root filesystem.

There are some fancy things that would have to be done to make telldir/seekdir, but the basic idea is pretty simple.

- Ted

What happens if the directory is grown at that point? I thought the reason for keeping it sorted in hash order was to deal with the telldir headache? I guess if the whole thing is in memory then it can be attached to the fd and discarded once read or seeked-on (and POSIX doesn't require reporting new entries after the start of the read).

Doing this at the VFS level would also benefit _most_ filesystems, though maybe not ones like XFS or btrfs that have their own preferred order.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

It's fine, because according to POSIX it's undefined what happens to files that are created or deleted after the last opendir() or rewindir(). So basically, the b-tree is attached to the opendir, and we discard it and re-create it if we ever seek to the beginning of the directory. This logic is already in the kernel, since it's exactly the same situation if the file is grown past the point where it becomes an HTREE directory.

Possibly, but it's a lot easier inside ext3/ext4 because we have most of the machinery in place and in use already.

- Ted

... presumably only if the directory has been modified in the meantime? It seems like this is missing from the ext3_dx_readdir() code, and could be easily achieved by checking inode->i_version vs. filp->f_version or similar. It looks like this is checked later on, but by that time we've already discarded everything.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Well, if a filesystem has a dedicated, presumably optimized backup utility, why would you not benchmark that as part of the mix? :)

Yep, I'm doing that now ... also realized I tested on an inappropriate elevator (cfq) for a fancy-raid. I'll resend in a bit.

well, tar supposedly is with the options I gave it, xfsdump certainly does, and dump, I dunno offhand.

Eh, I'll test that then.

Ah, ok, so they all should be backing up acls/attrs then.

True enough, just testing what we have now. I can play with acp...

-Eric

at Ric & hch's request here is tar on the other fs's as well, re-sorted
by level 0 dump time. I put acp into the mix as well.

Oh, and this time I remembered to set the elevator to something sane (noop) for this storage, oops (was cfq last time)

Also, this time the dup/tar/acp was written to /dev/null rather than another filesystem. Interesting how routing to /dev/null alone changed the ranking quite a bit.

		level0	level1
		======	======

ext4-acp 12m22s ------
ext3-acp 14m11s ------
ext4-tar 18m24s 34m56s
ext3-dump 19m30s 35m30s
xfs-dump 20m07s 40m24s
ext3-tar 21m16s 42m41s
xfs-tar 21m19s 46m13s
xfs-acp 29m38s ------

-Eric

Just for completeness's sake, can you apply the following patch to dump, and then try doing a benchmark run using dump on ext4?

I'm curious how it would compare.

- Ted

Index: dump-0.4b41/dump/traverse.c
===================================================================
@@ -157,14 +157,6 @@ int dump_fs_open(const char *disk, ext2_
 			retval = EXT2_ET_UNSUPP_FEATURE;
 			ext2fs_close(*fs);
 		}
-		else if ((retval = es->s_feature_incompat &
-					~(EXT2_LIB_FEATURE_INCOMPAT_SUPP |
-					  EXT3_FEATURE_INCOMPAT_RECOVER))) {
-			msg("Unsupported feature(s) 0x%x in filesystem\n",
-				retval);
-			retval = EXT2_ET_UNSUPP_FEATURE;
-			ext2fs_close(*fs);
-		}
 		else {
 			if (es->s_feature_compat &
 				EXT3_FEATURE_COMPAT_HAS_JOURNAL && 

Note that tar has a (questionable) optimization when writing to /dev/null. It will NOT open the file or read the data, and just do the filename traversal to generate the file list and total file size. It does this by comparing the output file to "/dev/null":

$ strace tar cvf /dev/null /t# mp/scheduler.pdf 
:
[opening libs and stuff]
:
creat("/dev/null", 0666)                = 3
fstat(3, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
write(1, "/tmp/scheduler.pdf\n", 19/tmp/scheduler.pdf
)    = 19
lstat("/tmp/scheduler.pdf", {st_mode=S_IFREG|0664, st_size=150262, ...}) = 0
close(3)                                = 0
close(1)                                = 0
munmap(0x7f2c00dd7000, 4096)            = 0
close(2)                                = 0
exit_group(0)                           = ?

A way to get around this is to hard link to /dev/null and use that as output:

# cp -a /dev/null /tmp/foo
$ ls -l /tmp/foo
0 crw-rw-rw- 1 root root 1, 3 2008-08-18 18:15 /tmp/foo
$ strace tar cvf /dev/null /t# mp/scheduler.pdf 
:
[opening libs and stuff]
:
creat("/tmp/foo", 0666)                 = 3
fstat(3, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
stat("/dev/null", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
lstat("/tmp/scheduler.pdf", {st_mode=S_IFREG|0664, st_size=150262, ...}) = 0
open("/tmp/scheduler.pdf", O_RDONLY)    = 4
write(1, "/tmp/scheduler.pdf\n", 19/tmp/scheduler.pdf) = 19
read(4, "%PDF-1.4\n%\303\244\303\274\303\266\303\237\n2 0 obj\n<</Le"..., 9728) = 9728
write(3, "tmp/scheduler.pdf\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240
read(4, "\5\351\204\216\220N\224\304\344\331\355\366\264\216\311n\345\365\224\370\352.\205\354\341\0237\306\31>\201"..., 10240) = 10240
:
[repeats]
:
write(3, "\253\2708\264\262\275\36\335\323\212\262\33\244\301\226\246\256d^\214n\246\220BW\363\306\25b%_L"..., 10240) = 10240
read(4, "i\264\364\243\264\220\327+u\3B\1\21\210\376\23\2035@D2\316t\356\271\v\37\210S\307]["..., 7414) = 7414
fstat(4, {st_mode=S_IFREG|0664, st_size=150262, ...}) = 0
close(4)                                = 0
write(3, "i\264\364\243\264\220\327+u\3B\1\21\210\376\23\2035@D2\316t\356\271\v\37\210S\307]["..., 10240) = 10240
close(3)                                = 0
close(1)                                = 0
munmap(0x7fb194cea000, 4096)            = 0
close(2)                                = 0
exit_group(0)                           = ?

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Ah, crud.

-Eric