Skip to contentSkip to conversation

is xfs good if I have millions of files and thousands of hardlinks?

DISCUSSION Tomasz Chmielewski 3 participants 5 messages Feb 19, 2008 – Feb 20, 2008

lore
Share & tools
lore
#0 lore raw

I have a ext3 filesystem with almost 200 million files (1.2 TB fs, ~65% full); most of the files are hardlinked multiple times, some of them are hardlinked thousands of times.

I described my problem yesterday on linux-fsdev list:

http://marc.info/?t=120333985100003

In general, because new files and hardlinks are being added all the time and the old ones are being removed, this leads to a very, very poor performance.

When I want to remove a lot of directories/files (which will be hardlinks, mostly), I see disk write speed is down to 50 kB/s - 200 kB/s (fifty - two hundred kilobytes/s) - this is the "bandwidth" used during the deletion.

Also, the filesystem is very fragmented ("dd if=/dev/zero of=some_file bs=64k" writes only about 1 MB/s).

Will xfs handle a large number of files, including lots of hardlinks, any better than ext3?

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#1 lore raw
Quoted 5 lines · Tomasz Chmielewski

I have a ext3 filesystem with almost 200 million files (1.2 TB fs, ~65% full); most of the files are hardlinked multiple times, some of them are hardlinked thousands of times.

I described my problem yesterday on linux-fsdev list:

http://marc.info/?t=120333985100003

quite a long discussion there .. I haven't read it .. but some comments below anyway ..

Quoted 9 lines

In general, because new files and hardlinks are being added all the time and the old ones are being removed, this leads to a very, very poor performance.

When I want to remove a lot of directories/files (which will be hardlinks, mostly), I see disk write speed is down to 50 kB/s - 200 kB/s (fifty - two hundred kilobytes/s) - this is the "bandwidth" used during the deletion.

Also, the filesystem is very fragmented ("dd if=/dev/zero of=some_file bs=64k" writes only about 1 MB/s).

Will xfs handle a large number of files, including lots of hardlinks, any better than ext3?

defragmenting by copying from the ext3 filesystem to a new filesystem should help, for a while at least. Whether xfs would have an on-going performance problem compared to ext3 depends on your usage patterns .. does "all the time" mean you are continuously adding new files and links and removing files at a high rate/second? Are multiple threads doing this? Are all the files the same size? Block-size been tuned?

-- Mark

#2 lore raw
Quoted 5 lines · Mark Goodwin

defragmenting by copying from the ext3 filesystem to a new filesystem should help, for a while at least. Whether xfs would have an on-going performance problem compared to ext3 depends on your usage patterns .. does "all the time" mean you are continuously adding new files and links and removing files at a high rate/second? Are multiple threads doing this?

Yes. Multiple threads adding new files (or hardlinks, if there are such files already) all the time (24h/day).
Normally, there is only one thread removing the files. Because of this performance problem I described, it also does its job 24h/day - it just can't finish removing the unneeded files in a couple of hours, not to say one day.

Quoted 1 line

Are all the files the same size? Block-size been tuned?

No, file sizes are mostly random stuff you will normally find on any rootfs, home, etc. directory. It's a backup system which uses hardlinks so that files which are already in backup do not take additional place.

I didn't do any block-size tuning, as I don't really know where to bite.

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#3 lore raw
In reply to Mark Goodwin
Quoted 6 lines · Peter Grandi

mangoo> In general, because new files and hardlinks are being mangoo> added all the time and the old ones are being removed, mangoo> this leads to a very, very poor performance.

That is not the cause of the poor performance. The ultimate cause is rather different.

Well, adding new files and hardlinks all the time leads to that that the inodes are scattered all over the disk.

Quoted 8 lines

mangoo> When I want to remove a lot of directories/files (which mangoo> will be hardlinks, mostly), I see disk write speed is mangoo> down to 50 kB/s - 200 kB/s (fifty - two hundred mangoo> kilobytes/s) - this is the "bandwidth" used during the mangoo> deletion.

How is bandwidth relevant for that? OK that there are quotes, but it seems very very stranget regardless.

The filesystem is available via iSCSI, so it's easy to measure the current performance. But iSCSI is not a problem here - performance is very good on an empty filesystem on that very same iSCSI/SAN device.

What I mean, is that when I remove large amount of files, the bandwidth used for writing to the disk is only down to 50-200 kB/s. Down from what, one might ask? Let me paste here yet another quotation from linux-fsdevel list, it may shed some more light:

Recently I began removing some of unneeded files (or hardlinks) and to my surprise, it takes longer than I initially expected.

After cache is emptied (echo 3 > /proc/sys/vm/drop_caches) I can usually remove about 50000-200000 files with moderate performance. I see up to 5000 kB read/write from/to the disk, wa reported by top is usually 20-70%.

After that, waiting for IO grows to 99%, and disk write speed is down to 50 kB/s - 200 kB/s (fifty - two hundred kilobytes/s).

Quoted 5 lines

mangoo> Also, the filesystem is very fragmented ("dd mangoo> if=/dev/zero of=some_file bs=64k" writes only about 1 mangoo> MB/s).

Then more the merrier.

Umm, no.
Usually, one is merrier when these numbers are high, not low ;)

Quoted 6 lines

mangoo> Will xfs handle a large number of files, including lots mangoo> of hardlinks, any better than ext3?

It shows consideration to consult the archives of a mailing list before aking a question. It may be a good idea to do it even after posting a question :-).

Oh, I did consult the archive. There are not many posts about hardlinks here on this xfs list (or, at least I didn't find many).

There was even a similar subject last year: someone had a 17 TB array used for backup, which was getting full, and asked if xfs is or will be capable of transparent compression. As xfs will not have transparent compression in a foreseeable future, it was suggested to him that he should use hardlinks - that alone could save him lots of space.

I wonder if the guy uses hardlinks now, and if yes, how does it behave on this 17 TB array (my filesystem is just 1.2 TB, but soon, I'm about to create a bigger one on another device - and hence my questions).

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#4 lore raw
In reply to Tomasz Chmielewski
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

On Tue, 19 Feb 2008 14:53:57 +0100, Tomasz Chmielewski <[email protected]> said:

mangoo> I have a ext3 filesystem with almost 200 million files mangoo> (1.2 TB fs, ~65% full); most of the files are hardlinked mangoo> multiple times, some of them are hardlinked thousands of mangoo> times.

Lucky numbers! :-)

mangoo> In general, because new files and hardlinks are being mangoo> added all the time and the old ones are being removed, mangoo> this leads to a very, very poor performance.

That is not the cause of the poor performance. The ultimate cause is rather different.

mangoo> When I want to remove a lot of directories/files (which mangoo> will be hardlinks, mostly), I see disk write speed is mangoo> down to 50 kB/s - 200 kB/s (fifty - two hundred mangoo> kilobytes/s) - this is the "bandwidth" used during the mangoo> deletion.

How is bandwidth relevant for that? OK that there are quotes, but it seems very very stranget regardless.

mangoo> Also, the filesystem is very fragmented ("dd mangoo> if=/dev/zero of=some_file bs=64k" writes only about 1 mangoo> MB/s).

Then more the merrier.

mangoo> Will xfs handle a large number of files, including lots mangoo> of hardlinks, any better than ext3?

It shows consideration to consult the archives of a mailing list before aking a question. It may be a good idea to do it even after posting a question :-).