XFS unlink still slow on 3.1.9 kernel ?

This is a backup system running dirvish, so most files in the dirs I am removing are hard links. Almost all of the files do have ACLs set.

Quoted 2 lines · Richard Ems

This is a backup system running dirvish, so most files in the dirs I am removing are hard links. Almost all of the files do have ACLs set.

How many ACLs do you usually have set? If they aren't stored inline but need to go out of the inode unlinks will be extremly slow for kernels before v3.2.

Quoted 8 lines · Christoph Hellwig

Quoted 2 lines · Richard Ems

This is a backup system running dirvish, so most files in the dirs I am removing are hard links. Almost all of the files do have ACLs set.

How many ACLs do you usually have set? If they aren't stored inline but need to go out of the inode unlinks will be extremly slow for kernels before v3.2.

Almost all dirs and files there do have ACLs set. Each of them do have about 10 user ACLs and 10 default ACls. Is that too many?
Is this then the reason for being that slow?

Quoted 10 lines · Richard Ems

Quoted 8 lines · Christoph Hellwig

Quoted 2 lines · Richard Ems

This is a backup system running dirvish, so most files in the dirs I am removing are hard links. Almost all of the files do have ACLs set.

How many ACLs do you usually have set? If they aren't stored inline but need to go out of the inode unlinks will be extremly slow for kernels before v3.2.

Quoted 4 lines

Almost all dirs and files there do have ACLs set. Each of them do have about 10 user ACLs and 10 default ACls. Is that too many?
Is this then the reason for being that slow?

That doesn't sound like a lot to me, but instead of guessing around, let's just check the actual facts.

Does "xfs_bmap -a" for the kind of files you are deleting show any extents? If it doesn't the output will look like:

# xfs_bmap -a internal
internal: no extents

if it has any it will look like:

# xfs_bmap -a external
external:
	0: [0..7]: 8557712..8557719

YES. All files (and dirs) that I checked do show something as

0: [0..7]: 18531216..18531223

So, what improvements can I expect from a kernel > 3.2 ? Can I read somewhere about the changes/patches introduced?

Is there another way to mount/create/mkfs the XFS to improve the unlink time for this case?

Quoted 6 lines · Richard Ems

YES. All files (and dirs) that I checked do show something as

0: [0..7]: 18531216..18531223

So, what improvements can I expect from a kernel > 3.2 ? Can I read somewhere about the changes/patches introduced?

On some crazy workloads I've seen speedups up to a factor of 10.000 (5 orders or magnitude). You probably won't get that much of a speedup, but it will still be significant.

The patch in mainline for this is:

commit 859f57ca00805e6c482eef1a7ab073097d02c8ca

Author: Christoph Hellwig <[email protected]>
Date:   Sat Aug 27 14:45:11 2011 +0000

xfs: avoid synchronous transactions when deleting attr blocks

Quoted 2 lines

Is there another way to mount/create/mkfs the XFS to improve the unlink time for this case?

Try increasing the inode size during filesystem creating using the "-i size=512" option or even "-i size=1024" if you still have out of line attributes. The should give you even bigger speedups for this workload than the patch above.

I will try to install a > 3.2 kernel and will create new XFS partitions with "-i size=1024", since we use ACLs a lot for user access. Is there a chance to change existing XFS partitions to "-i size=1024" ?

Quoted 3 lines · Richard Ems

I will try to install a > 3.2 kernel and will create new XFS partitions with "-i size=1024", since we use ACLs a lot for user access. Is there a chance to change existing XFS partitions to "-i size=1024" ?

Unfortunately not. Note that the speedups in 3.2 only matter for out of line attributes - once you store the ACLs inside the inode the code that makes it dog slow in old kernels is never used.

So

1. use kernels > 3.2, if XFS partition was not created using "-i size=1024"

2. if XFS partition was created using "-i size=1024", kernels < 3.2 will also be fast while unlinking files with out of line attributes

Quoted 3 lines · Richard Ems

So

1. use kernels > 3.2, if XFS partition was not created using "-i size=1024"

Yes.

Quoted 2 lines

2. if XFS partition was created using "-i size=1024", kernels < 3.2 will also be fast while unlinking files with out of line attributes

Exactly.

For workloads like yours creating the large inodes (probably only with 512 byte inodes, though) will be preferably if you have a choice, as it should be even faster.

Quoted 3 lines · Richard Ems

So

1. use kernels > 3.2, if XFS partition was not created using "-i size=1024"

Yes.

Quoted 2 lines

2. if XFS partition was created using "-i size=1024", kernels < 3.2 will also be fast while unlinking files with out of line attributes

Exactly.

For workloads like yours creating the large inodes (probably only with 512 byte inodes, though) will be preferably if you have a choice, as it should be even faster.

I already updated to 3.2.4 and started the same "find dir" command again that previously took 100 min to run. It has been running now for over 30 min ...

Should this "find" run time also improve ?

Or will only unlink run time improve ?

Do I have to mount the XFS partition with some new/old/special option?

I'd have to look into it in more detail. IIRC you said you're using RAID6 which can be fairly nasty for small reads. Did you use the inode64 mount option on the filesystem?

# mount | grep xfs
/dev/sda1 on /backup/IFT type xfs

20 TB XFS
partition is 100% full

The case is connected through SCSI.

Quoted 5 lines · Richard Ems

I already updated to 3.2.4 and started the same "find dir" command again that previously took 100 min to run. It has been running now for over 30 min ...

Should this "find" run time also improve ?

No, not by that change anyway.

Quoted 1 line

Or will only unlink run time improve ?

Yes.

Quoted 1 line

Do I have to mount the XFS partition with some new/old/special option?

I'd have to look into it in more detail. IIRC you said you're using RAID6 which can be fairly nasty for small reads. Did you use the inode64 mount option on the filesystem?

It didn't improve, 100 min again.

Quoted 3 lines

Quoted 1 line

Or will only unlink run time improve ?

Yes.

rm took about 110 min.

Quoted 4 lines · Richard Ems

I already updated to 3.2.4 and started the same "find dir" command again that previously took 100 min to run. It has been running now for over 30 min ...

Quoted 1 line

Should this "find" run time also improve ?

No, not by that change anyway.

Quoted 1 line

Or will only unlink run time improve ?

Yes.

Quoted 4 lines · Richard Ems

I already updated to 3.2.4 and started the same "find dir" command again that previously took 100 min to run. It has been running now for over 30 min ...

Quoted 1 line

Should this "find" run time also improve ?

No, not by that change anyway.

Quoted 1 line

Or will only unlink run time improve ?

Yes.

Quoted 4 lines · Richard Ems

I already updated to 3.2.4 and started the same "find dir" command again that previously took 100 min to run. It has been running now for over 30 min ...

Quoted 1 line

Should this "find" run time also improve ?

No, not by that change anyway.

Quoted 1 line

Or will only unlink run time improve ?

Yes.

Hello list !

I ran a "find dir" on one directory with 11 million files and dirs in it and it took 100 minutes. Is this a "normal" run time to be expected?

I am running openSUSE 12.1, kernel 3.1.9-1.4-default. The 20 TB XFS partition is 100% full

and is on an external InforTrend RAID system with
24 x 1 TB SATA HDDs on RAID 6 with one hot-spare HDD, so 21 data discs plus 2 parity discs plus 1 hot-spare disc. The case is connected through SCSI.

The system was not running anything else on that discs and the load on the server was around 1 because of only this one find command running.

I am asking because I am seeing very long times while removing big directory trees. I thought on kernels above 3.0 removing dirs and files had improved a lot, but I don't see that improvement.

This is a backup system running dirvish, so most files in the dirs I am removing are hard links. Almost all of the files do have ACLs set.

Quoted 2 lines · Richard Ems

I am running openSUSE 12.1, kernel 3.1.9-1.4-default. The 20 TB XFS partition is 100% full

Running filesystems to 100% full is always a bad idea - it causes significant increases in fragementation of both data and metadata compared to a filesystem that doesn't get past ~90% full.

Quoted 3 lines

I am asking because I am seeing very long times while removing big directory trees. I thought on kernels above 3.0 removing dirs and files had improved a lot, but I don't see that improvement.

You won't if the directory traversal is seek bound and that is the limiting factor for performance.

Quoted 2 lines

This is a backup system running dirvish, so most files in the dirs I am removing are hard links. Almost all of the files do have ACLs set.

The unlink will have an extra IO to read per inode - the out-of-line attribute block, so you've just added 11 million IOs to the 800,000 the traversal already takes to the unlink overhead. So it's going to take roughly ten hours because the unlink is gong to be read IO seek bound....

Christophs suggestions to use larger inodes to keep the attribute data inline is a very good one - whenever you have a workload that is attribute heavy you should use larger inodes to try to keep the attributes in-line if possible. The down side is that increasing the inode size increases the amount of IO required to read/write inodes, though this typically isn't a huge penalty compared to the penalty of out-of-line attributes.

Also, for large directories like this (millions of entries) you should also consider using a larger directory block size (mkfs -n size=xxxx option) as that can be scaled independently to the filesystem block size. This will significantly decrease the amount of IO and fragmentation large directories cause. Peak modification performance of small directories will be reduced because larger block size directories consume more CPU to process, but for large directories performance will be significantly better as they will spend much less time waiting for IO.

Quoted 2 lines

You won't if the directory traversal is seek bound and that is the limiting factor for performance.

*Seek bound*? *When* is the directory traversal *seek bound*?

Quoted 3 lines · Richard Ems

Quoted 2 lines

You won't if the directory traversal is seek bound and that is the limiting factor for performance.

Quoted 1 line

*Seek bound*? *When* is the directory traversal *seek bound*?

You read the inode for the directory first, then the external attribute block for the ACLs, then if the directory isn't tiny you'll start reading directory blocks, the more the larger the directory is, and if the filesystem is close to beeing full they often will be non-contiguous. Then you read the inode for each file/directory in it, then the external attribute block, then the extent list, and so on.

Quoted 6 lines · Dave Chinner

Quoted 3 lines

I am asking because I am seeing very long times while removing big directory trees. I thought on kernels above 3.0 removing dirs and files had improved a lot, but I don't see that improvement.

You won't if the directory traversal is seek bound and that is the limiting factor for performance.

*Seek bound*? *When* is the directory traversal *seek bound*?

Quoted 8 lines

Quoted 2 lines

This is a backup system running dirvish, so most files in the dirs I am removing are hard links. Almost all of the files do have ACLs set.

The unlink will have an extra IO to read per inode - the out-of-line attribute block, so you've just added 11 million IOs to the 800,000 the traversal already takes to the unlink overhead. So it's going to take roughly ten hours because the unlink is gong to be read IO seek bound....

It took 110 minutes and not 10 hours. All files and dirs there had ACLs set.

Quoted 9 lines

Also, for large directories like this (millions of entries) you should also consider using a larger directory block size (mkfs -n size=xxxx option) as that can be scaled independently to the filesystem block size. This will significantly decrease the amount of IO and fragmentation large directories cause. Peak modification performance of small directories will be reduced because larger block size directories consume more CPU to process, but for large directories performance will be significantly better as they will spend much less time waiting for IO.

This was not ONE directory with that many files, but a directory containing 834591 subdirectories (deeply nested, not all in the same dir!) and 10539154 files.

Quoted 5 lines · Richard Ems

Quoted 4 lines · Dave Chinner

Quoted 3 lines

I am asking because I am seeing very long times while removing big directory trees. I thought on kernels above 3.0 removing dirs and files had improved a lot, but I don't see that improvement.

Quoted 4 lines

Quoted 2 lines

You won't if the directory traversal is seek bound and that is the limiting factor for performance.

*Seek bound*? *When* is the directory traversal *seek bound*?

Whenever you are traversing a directory structure that is not alrady hot in the cache. IOWS, almost always.

Quoted 11 lines

Quoted 9 lines

Quoted 3 lines

This is a backup system running dirvish, so most files in the dirs I am removing are hard links. Almost all of the files do have ACLs set.

The unlink will have an extra IO to read per inode - the out-of-line attribute block, so you've just added 11 million IOs to the 800,000 the traversal already takes to the unlink overhead. So it's going to take roughly ten hours because the unlink is gong to be read IO seek bound....

It took 110 minutes and not 10 hours. All files and dirs there had ACLs set.

I was basing that on you "find dir" time of 100 minutes, which was the only number you gave, and making the assumption it didn't read the attribute blocks and that it was seeing worse case seek times (i.e. avg seek times) for every IO.

Given the way locality works in XFS, I'd suggest that the typical seek time will be much less (a few blocks, not half the disk platter) and not necessarily on the same disk (due to RAID) so the average seek time for your workload is likely to be much lower. If it's at 1ms (closer to track-to-track seek times) instead of the 5ms, then that 10hrs becomes 2hrs for that many IOs....

Quoted 10 lines

Quoted 9 lines

Also, for large directories like this (millions of entries) you should also consider using a larger directory block size (mkfs -n size=xxxx option) as that can be scaled independently to the filesystem block size. This will significantly decrease the amount of IO and fragmentation large directories cause. Peak modification performance of small directories will be reduced because larger block size directories consume more CPU to process, but for large directories performance will be significantly better as they will spend much less time waiting for IO.

Quoted 3 lines

This was not ONE directory with that many files, but a directory containing 834591 subdirectories (deeply nested, not all in the same dir!) and 10539154 files.

So you've got a directory *tree* that indexes 11 million inodes, not "one directory with 11 million files and dirs in it" as you originally described. Both Christoph and I have interpreted your original description as "one large directory", but there's no need to shout at us because it's difficult to understand any given configuration from just a few lines of text. IOWs, details like "one directory" vs "one directory tree" might seem insignificant to you, but they mean an awful lot us developers and can easily lead us down the wrong path.

FWIW, directory tree traversal is even more read IO latency sensitive than a single large directory traversal because we can't do readahead across directory boundaries to hide seek latencies as much as possible and the locality on individual directories can be very different depending on the allocaiton policy the filesystem is using. As it is, large directory blocks can also reduce the amount of IO needed in this sort of situation and speed up traversals....

Cheers,

Dave.