xfsdump -s unacceptable performances

Hi all,
I'm new to the list. I have a problem with xfsdump using the -s option. I'm aware that the latest version (2.2.38) skip the scan and prune of the entire filesystem (see the test below), but there are other places where xfsdump performs an entire scan of the filesystem, slowing down the backup process.

For example in:
* dump/inomap.c after "phase 3" there is a function call to bigstat_iter that scan the entire filesystem
* dump/content.c the function dump_dirs again scan the entire filesystem

Are all there scan really necessary?
Could we expect a performance fix?
Is there a workaround?

I have done some test, and with quite large filesystem the performances where unacceptable. Dumping one directory with 4 file using 4KB of space takes hours (or days, it hasn't finished yet) if the underlying filesystem contains around 10.000.000 inodes.

On request I could provide more information and the output of some test.

Let's start with a simple comparison between 2.2.27 and 2.2.38 using a small filesystem (40.000 inodes).

time xfsdump -p 60 -v drive=debug,media=debug \
-s etc/20060401-0030 - /media/xfs-test | dd of=/dev/null
xfsdump: using file dump (drive_simple) strategy
xfsdump: version 2.2.27 (dump format 3.0) - Running single-threaded [...]
xfsdump: media file size 17780576 bytes
xfsdump: dump size (non-dir files) : 16477472 bytes
xfsdump: dump complete: 219459 seconds elapsed
xfsdump: Dump Status: SUCCESS
34727+1 records in
34727+1 records out
17780576 bytes transferred in 219459.047053 seconds (81 bytes/sec)

real    3657m39.120s
user    0m2.361s
sys     0m9.238s

time /usr/lib/bin/xfsdump -p 60 -v drive=debug,media=debug \
-s etc/20060401-0030 - /media/xfs-test | dd of=/dev/null /usr/lib/bin/xfsdump: using file dump (drive_simple) strategy /usr/lib/bin/xfsdump: version 2.2.38 (dump format 3.0) - Running single-threaded [...]
/usr/lib/bin/xfsdump: media file size 17780576 bytes /usr/lib/bin/xfsdump: dump size (non-dir files) : 16477472 bytes /usr/lib/bin/xfsdump: dump complete: 18 seconds elapsed /usr/lib/bin/xfsdump: Dump Status: SUCCESS
34727+1 records in
34727+1 records out
17780576 bytes transferred in 17.451939 seconds (1018831 bytes/sec)

real    0m17.575s
user    0m0.132s
sys     0m1.035s

Thanks in advance for your suggestions.

Daniele P.

Hi Daniele,

hum, so the 2.2.27 was running 3657m39.120s and the 2.2.38 is running the same (?) in 0m17.575s. Where is your performance problem running now 10000 times faster than the old 2.2.27 version???
Or did i miss something?

Ciao
Klaus

Mit freundlichen Grüssen / best regards

Klaus Strebel, Dipl.-Inform. (FH), mailto:[email protected]

/"\
\ /     ASCII RIBBON CAMPAIGN
 X        AGAINST HTML MAIL
/ \

Hi Klaus,
Yes, there was a huge performance improvement for /small/ filesystem, (40.000 inodes, in my test case).
But xfsdump still doesn't scale down well with a small subtree on a large filesystem.
Now the test on the /large/ (11.000.000 inodes) filesystem has ended.

ori:~# find /media/xfs-large/backup/pin/ -type f | wc -l
2
ori:~# find /media/xfs-large/backup/pin/ -type d | wc -l
2
ori:~# find /media/xfs-large/backup/pin/ | wc -l
4
ori:~# du -s /media/xfs-large/backup/pin/
8       /media/xfs-large/backup/pin/

I stopped the test with 2.2.27 after 5 days of "phase 3". Here is the test with 2.2.38, which is still too slow:

time /usr/local/bin/xfsdump -p 60 -v drive=debug,media=debug \
-s backup/pin - /media/xfs-large | dd of=/dev/null /usr/local/bin/xfsdump: using file dump (drive_simple) strategy /usr/local/bin/xfsdump: version 2.2.38 (dump format 3.0) - Running single-threaded
/usr/local/bin/xfsdump: level 0 dump of ori:/media/xfs-large /usr/local/bin/xfsdump: dump date: Wed Aug 16 12:56:57 2006 /usr/local/bin/xfsdump: session id: a293b79a-a5ac-4e49-be3e-55fbc5f8fba2 /usr/local/bin/xfsdump: session label: ""
/usr/local/bin/xfsdump: ino map phase 1: constructing initial dump list /usr/local/bin/xfsdump: status at 12:58:12: 75 seconds elapsed /usr/local/bin/xfsdump: ino map phase 2: skipping (no pruning necessary) /usr/local/bin/xfsdump: ino map phase 3: skipping (only one dump stream) /usr/local/bin/xfsdump: status at 12:58:57: inomap phase 3 79568/11017767 inos scanned, 120 seconds elaps
ed
[...]
/usr/local/bin/xfsdump: status at 13:10:58: inomap phase 3 1890564/11017767 inos scanned, 841 seconds elapsed
/usr/local/bin/xfsdump: ino map construction complete /usr/local/bin/xfsdump: estimated dump size: 5617536 bytes /usr/local/bin/xfsdump: Media op: begin media file /usr/local/bin/xfsdump: creating dump session media file 0 (media 0, file 0) /usr/local/bin/xfsdump: dumping ino map
/usr/local/bin/xfsdump: flushing write buf addr 0xb7e58000 size 0x40000 [...]
/usr/local/bin/xfsdump: flushing write buf addr 0xb7e58000 size 0x40000 /usr/local/bin/xfsdump: dumping directories /usr/local/bin/xfsdump: dumping non-directory files /usr/local/bin/xfsdump: status at 13:44:48: 1/2 files dumped, 31.6% data dumped, 2871 seconds elapsed
[...]
/usr/local/bin/xfsdump: status at 14:02:57: 2/2 files dumped, 44.5% data dumped, 3960 seconds elapsed
/usr/local/bin/xfsdump: ending media file
/usr/local/bin/xfsdump: Media op: end media file /usr/local/bin/xfsdump: flushing write buf addr 0xb7e58000 size 0x1a6b8 /usr/local/bin/xfsdump: media file size 5613240 bytes /usr/local/bin/xfsdump: dump size (non-dir files) : 3648 bytes /usr/local/bin/xfsdump: dump complete: 3999 seconds elapsed /usr/local/bin/xfsdump: Dump Status: SUCCESS
10963+1 records in
10963+1 records out
5613240 bytes transferred in 3998.904421 seconds (1404 bytes/sec)

real    66m38.933s
user    0m2.882s
sys     0m36.289s

Regards,
Daniele

That is very true.
It is really designed for dumping whole filesystems (or at least, large parts of them).
For dumping small subtrees, I'd be looking at using something else. That was one of the 1st things I noticed when looking at the xfsdump code on IRIX a while back, was all the scans it does and how for a subtree it will do another scan and prune the data - i.e. implemented like an afterthought :)
It is built around bulkstat which walks all the inodes which is great for dumping them all out.
But if you want a small directory walk then you need something else IMHO.

Cheers,
--Tim

Hi Timothy,
Yes, you are right, but there is another problem on my side. The /small/ subtree of the filesystem usually contains a lot of hard links (our backup software uses hard links to save disk space, so expect one hard link per file per day) and using a generic tool like tar/star or rsync that uses "stat" to scan the filesysem should be significant slower (no test done) than a native tool like xfsdump, as Bill in a previous email pointed out.

It seems that there isn't a right tool for this job.

Regards,
Daniele

[ ... ]

daniele> Hi Timothy, Yes, you are right, but there is another daniele> problem on my side. The /small/ subtree of the daniele> filesystem usually contains a lot of hard links (our daniele> backup software

Given the context, I would imagine that this backup filesystem is stored on a RAID5 device... Is it? It can be an important part of the strategy.

daniele> uses hard links to save disk space, so expect one hard daniele> link per file per day) and using a generic tool like daniele> tar/star or rsync that uses "stat" to scan the daniele> filesysem should be significant slower (no test done) daniele> than a native tool like xfsdump, as Bill in a previous daniele> email pointed out.

Depends a lot, for example on whether the system has enough RAM to cache the inodes affected, and anyhow on the ratio between inodes in the subtree and inodes in the whole filesystem.

As to this case:

deniale> Dumping one directory with 4 file using 4KB of deniale> space takes hours (or days, it hasn't finished yet) deniale> if the underlying filesystem contains around deniale> 10.000.000 inodes.

probably using 'tar'/'star' may be a bit faster...

daniele> It seems that there isn't a right tool for this job.

Or perhaps it is not the right job :-).

Anyhow sequential scans of large filesystems are not an awesome idea in general. I wonder how long and how much RAM your 10m inode filesystem will take to 'fsck' for example :-), perhaps you don't want to read this older entry in this mailing list:

http://OSS.SGI.com/archives/linux-xfs/2005-08/msg00045.html

The basic problem is that the bottleneck is the ''pickup'' and its speed has not grown as fast as disc capacity, so one has to do RAID to work around that, but RAID delivers only for parallel, not sequential, scans of the filesystem.

A significant issue that nobody seems in a hurry to address. As

:-)

This scan stops after determining start points for all streams. On Linux there's always one dump stream, so it returns after reading one buffer full of inodes.

And of course there's another scan for dumping the non-dir inodes. Keep in mind these are inode scans, which are substantially faster than recursing through the directories doing individual stat(2) calls.

Nonetheless, these scans could be optimized by seeking the scan to the next inode of interest, which could be found using xfsdump's inomap (created in the first scan). This would be beneficial to -s and incremental dumps.

A lot of this stuff could be done in a single scan in a disk-to-disk backup approach. But in the current scheme, they are necessary.

Nothing is planned, but patches are always welcome.

Regards,
Bill

On Wednesday 16 August 2006 18:38, Bill Kendall wrote:

Hi Bill,
Thanks for the explanations. I only have had a glimpse into the sources.

Yes, but scanning 10.000.000 inodes when you really need to scan only 4 it's really a waste of resource.
This is a corner case. But usually I want to backup only 10% of the entire filesystem.
At this point I have to redesign my backup strategy taking care of xfsdump strength and weakness.

That's interesting.

This too, but for me the xfs filesystem it's a big black box.

Thanks again,
Daniele